Automating pipelines using airflow operators
-
Updated
Apr 6, 2024 - Python
Automating pipelines using airflow operators
Repositório dedicado ao desafio do hackathon de engenharia de dados A3 Data Challenge Woman
Prefect - Data orchestration tool practice & learning
Nexus is a distributed workflow system designed for use by data engineers to move data around their organisations.
TribesAI Internship challenge task. I got this as an assignment for a Data Engineering Internship
Image Classification Model, You can upload the following images: TRANSPORTS: Car, Boat, Airplane, Rocket, Helicopter, CARNIVORES: Raccoon, Otter, Dog, Lion, Tiger, Red_panda, Lynx, Jaguar, Bear, Fox, Cat FRUITS: Apple, Grape, Common_fig, Pear, Strawberry, Tomato, Lemon, Banana, Orange, Peach, Mango, Pineapple, Grapefruit, Pomegranate, Watermelon…
Data Processing with PySpark: Parsing Data from MongoDB
A Python library for extracting information via XPaths
Repo d'un cas d'usage spécifique avec l'API Times Wire
Python example using Pandas to load CSV into a local SQLite DB.
Web Scraping: 0-14 Years Old Data - All Countries
Data Engineering Project
Realiza a coleta dos dados do TSE com o Scrapy e insere em um index do Elasticsearch.
Repository for Replication of Professor Teo Calvo's Projects
This repository contains the flow for extract data from sensors at the WindFarm
Continuation of GW2-SRS project focused on migrating the ETL to the cloud and making optimizations with Docker and Airflow.
This project Synthetic data generator plus (SDGP) is a python script that generates mock data based on given configurations. It can also edit and scale existing data to create high volume data. It is useful for testing, learning data domine and prototyping purposes.
Cleaning the dataframe and adding columns with the average grade and whether the student was approved based on the average grade
Add a description, image, and links to the data-engineer topic page so that developers can more easily learn about it.
To associate your repository with the data-engineer topic, visit your repo's landing page and select "manage topics."