State of Data Science Nevada Conference: Multi-track tutorial to create, provision, and version control AWS infrastructure to manage data pipelines effectively
-
Updated
Jan 23, 2021 - Python
State of Data Science Nevada Conference: Multi-track tutorial to create, provision, and version control AWS infrastructure to manage data pipelines effectively
Tutorials for DevOps tools such as Google Codelabs, Apache Airflow, Streamlit, FastAPI, Great Expectations, etc.
Kafka-Spark jobs orchestrated with Airflow
Personal Data Engineering project witch the objective is create the Data Lakehouse for a B2B e-commerce that must store the transactional and analytical data of the business. The final system delivers structured and clean data with the purpose of generate reports and find opportunities.
A pipeline to forecast the direction stock prices from data from eodhistoricaldata.com
Create data pipeline using Lambda architecture with Spark, Kafka, Airflow and Snowflake
Using Great Expectations and Notion's API, this repo aims to provide data quality for our databases in Notion.
An ML pipeline to flip nfts that makes use of the cloud and containers.
This library is inspired by the Great Expectations library. The library has made the various expectations found in Great Expectations available when using the inbuilt python unittest assertions.
Run greatexpectations.io on ANY SQL Engine using REST API. Supported by FastAPI, Pydantic and SQLAlchemy as best data quality tool
Code to demonstrate data engineering metadata & logging best practices
This repository serves as a comprehensive guide to effective data modeling and robust data quality assurance using popular open-source tools
A project for exploring how Great Expectations can be used to ensure data quality and validate batches within a data pipeline defined in Airflow.
A Covid-19 data pipeline on AWS featuring PySpark/Glue, Docker, Great Expectations, Airflow, and Redshift, templated in CloudFormation and CDK, deployable via Github Actions.
Prefect integrations for interacting with Great Expectations
Nyc_Taxi_Data_Pipeline - DE Project
Data Quality Gate based on AWS
Add a description, image, and links to the great-expectations topic page so that developers can more easily learn about it.
To associate your repository with the great-expectations topic, visit your repo's landing page and select "manage topics."