Skip to content

Devops for DWH which is for Crypto data analysis (hadoop, hive, spark, kafka, cassandra, trino, etc.)

Notifications You must be signed in to change notification settings

kentarokamiyajp/crypto-prediction-infra

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 

Repository files navigation

Infra for Crypto Price Prediction

image

System Overview

Related Repositories

Source of Crypto Price and Other Useful Data

Realtime Data Streaming

  • Data from Poloniex via WebSocket
    • Order book
    • Market trade
    • Crypto price for every minute
  • Event streaming platform
    • Kafka
  • Streaming Data Processing Tools
    • Kafka Consumer
    • Flink
    • Spark Streaming

Source Database

  • Cassandra
  • MongoDB

Data Warehouse System

  • Main DB
    • Hive built on HDFS
  • ETL tools
    • Spark
    • Trino
    • DBT(?)

Data Analyzation

  • Druid for real-time analysis
  • Trino for ad-hoc analysis

Job Scheduling Tool

  • Airflow

About

Devops for DWH which is for Crypto data analysis (hadoop, hive, spark, kafka, cassandra, trino, etc.)

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published