seilylook / Forex-Data-Pipeline Public

Notifications You must be signed in to change notification settings
Fork 0
Star 0

This project involved the integration of several cutting-edge technologies to ensure efficient data processing and management for forex trading analytics.

Apache-2.0 license

0 stars 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 9 Commits
docker		docker
docs		docs
mnt		mnt
.Python		.Python
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml
pyvenv.cfg		pyvenv.cfg
reset.sh		reset.sh
restart.sh		restart.sh
start.sh		start.sh
stop.sh		stop.sh

Repository files navigation

Flow chart of the Forex Data Pipeline

check availability of forex rates
check availability of the file having currencies to watch
download forex rates with python
save the forex rates in HDFS
create a hive table to store forex rates from the HDFS
process forex rates with Spark
send an email notificatoin
send a slack notification

About

This project involved the integration of several cutting-edge technologies to ensure efficient data processing and management for forex trading analytics.

docker airflow spark hive postgresql hdfs hue

Apache-2.0 license

Report repository

Releases

No releases published

Packages

No packages published

Languages