Automating pipelines using airflow operators
-
Updated
Apr 6, 2024 - Python
Automating pipelines using airflow operators
Udacity - Data Engineering Nanodegree (Project 1)
Prefect - Data orchestration tool practice & learning
TribesAI Internship challenge task. I got this as an assignment for a Data Engineering Internship
Image Classification Model, You can upload the following images: TRANSPORTS: Car, Boat, Airplane, Rocket, Helicopter, CARNIVORES: Raccoon, Otter, Dog, Lion, Tiger, Red_panda, Lynx, Jaguar, Bear, Fox, Cat FRUITS: Apple, Grape, Common_fig, Pear, Strawberry, Tomato, Lemon, Banana, Orange, Peach, Mango, Pineapple, Grapefruit, Pomegranate, Watermelon…
Data Processing with PySpark: Parsing Data from MongoDB
A Python library for extracting information via XPaths
Example project implementing best practices and testing for PySpark data pipelines.
Repo d'un cas d'usage spécifique avec l'API Times Wire
FegTec é uma empresa fictícia que quer transferir arquivos parquet contendo dados dos clientes da nuvem AWS para a Google Cloud
This repository contains the flow for extract data from sensors at the WindFarm
This project Synthetic data generator plus (SDGP) is a python script that generates mock data based on given configurations. It can also edit and scale existing data to create high volume data. It is useful for testing, learning data domine and prototyping purposes.
Full code for UDACITY's Data Engineer Nano Degree project. Build a Data Warehouse in AWS with Amazon Redshift.
Repository for Replication of Professor Teo Calvo's Projects
In this project, we apply Data Modeling with Postgres and build an ETL pipeline using Python.
An ETL pipeline that extracts, transforms, and loads data from various sources related to electric vehicle (EV) stocks.
Data Engineer (Udacity): Project 4 Data Lakes with Spark on Amazon Web Service (AWS)
Web Scraping: 0-14 Years Old Data - All Countries
Data Engineering Project
Add a description, image, and links to the data-engineer topic page so that developers can more easily learn about it.
To associate your repository with the data-engineer topic, visit your repo's landing page and select "manage topics."