Skip to content

DataWareHouse system for crypto data analysis using hive, cassandra, trino, kafka, spark, etc.

Notifications You must be signed in to change notification settings

kentarokamiyajp/crypto-prediction-etl

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ETL for Crypto Price Prediction

image

System Overview

Related Repositories

Source of Crypto Price and Other Useful Data

Realtime Data Streaming

  • Data from Poloniex via WebSocket
    • Order book
    • Market trade
    • Crypto price for every minute
  • Event streaming platform
    • Kafka
  • Streaming Data Processing Tools
    • Kafka Consumer
    • Flink
    • Spark Streaming

Source Database

  • Cassandra
  • MongoDB

Data Warehouse System

  • Main DB
    • Hive built on HDFS
  • ETL tools
    • Spark
    • Trino
    • DBT(?)

Data Analyzation

  • Druid for real-time analysis
  • Trino for ad-hoc analysis

Job Scheduling Tool

  • Airflow

About

DataWareHouse system for crypto data analysis using hive, cassandra, trino, kafka, spark, etc.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published