Weather_Difference_Between_SFO/SEA

Some useful stuff for Data Engineers

A simple etl pipeline orchestrated with dagster, to run both locally and/or on k8s. Uses minIO as both IOManager and destination when on k8s

Automatic Trading System for OANDA. Currency pairs can be analysed by customising the list in the regression algorithms.

Orchestrate OpenSearch operations with Apache Airflow and the OpenSearch Airflow provider

This repository contains code and configuration files for an Extract, Transform, Load (ETL) project using Google Cloud Data Fusion for data extraction, Apache Airflow/Composer for orchestration, and Google BigQuery for data loading.

Programa para guardar cambios de forma automática mientras trabajas con Power BI

GUI based data transformation web-app

Data Engineering | ETL process

ETL workflow for Udacity's Data Engineering Nanodegree project 3 on Redshift data warehousing

Crosslend DE Assignment

Mean Variance Standard Deviation Calculator project for FreeCodeCamp Data Analyst with Python course.

Tea Price Scraper: A Python script to scrape and save weekly tea prices from a website. This repository is for the first round of the selection process for the role of Intern - Data Engineering at BIPP, ISB.

Data Science Demonstrations in different specialties such as Data Engineering and Data Scientist.

Projeto de construção de um ambiente analitico

A generated tech community dataset for use at DataEngHack

Data engineering with spark, SQL, and Kafka

Com este software podemos solicitar comandos shell complexos em linguagem natural e executá-los em nosso sistema, tornando a automação e a interação com o shell mais acessíveis e intuitivas para usuários menos experientes, além de ser um legítimo assistente virtual inteligente para automação de tarefas.

Stream Processing of website click data using Kafka and monitored and visualised using Prometheus and Grafana