Data Engineering Portfolio

This is a private repository to set up and showcase a data pipeline and architecture design.

Services to Include

• CI/CD process
  • CI with GitHub Actions
  • Coupled with unit testing
    • With pytest and pylint
• SQL dump into mariadb
  • Hosting of mariadb-server on local linux env
• File upload to AWS s3
  • File upload from local linux env to cloud storage
• Airflow schedule
  • Hosting of airflow-server on local linux env
• Error logging