Data Engineering

This repository contains a collection of resources for data engineers. The resources include code, scripts, and documentation.

Code

The code in this repository is organized by topic. The topics include:

Data extraction
Data cleaning
Data transformation
Data modeling
Data analysis
Data visualization The code is written in a variety of languages, including Python, and Scala. Infrastructure as Code is also written in multiple languages.

The scripts in this repository are used to automate tasks related to data engineering. The tasks include:

Data loading
Data validation
Data backup
Data deployment The scripts are written in a variety of languages, including Python, Bash, and PowerShell.

The documentation in this repository provides instructions on how to use the code and scripts in this repository.

source venv/bin/activate

Name		Name	Last commit message	Last commit date
Latest commit History 43 Commits
.ipynb_checkpoints		.ipynb_checkpoints
agent		agent
aws-data-engineer-training		aws-data-engineer-training
aws-data-introduction		aws-data-introduction
boto3		boto3
data-pipelines		data-pipelines
databricks_training		databricks_training
goodinvestory		goodinvestory
iexpertify_adjustments		iexpertify_adjustments
nginx-infra		nginx-infra
pulumi_nginx		pulumi_nginx
pyspark-exmaples		pyspark-exmaples
redshift_project		redshift_project
s3_tables		s3_tables
weekly_options_predictor/data_ingestion		weekly_options_predictor/data_ingestion
.gitignore		.gitignore
.python-version		.python-version
111-06182023.csv		111-06182023.csv
Financial-06182023.csv		Financial-06182023.csv
LOAD00000001.csv		LOAD00000001.csv
Linux-commands.md		Linux-commands.md
Overview-06182023.csv		Overview-06182023.csv
Ownership-06182023.csv		Ownership-06182023.csv
Performance-06182023.csv		Performance-06182023.csv
Readme.md		Readme.md
Technical-06182023.csv		Technical-06182023.csv
Untitled Diagram.drawio		Untitled Diagram.drawio
Valuation-06182023.csv		Valuation-06182023.csv
data-engineer-toolbox.md		data-engineer-toolbox.md
input.csv		input.csv
nse_stock_prices.py		nse_stock_prices.py
pandas_test.py		pandas_test.py
reconcile.py		reconcile.py
serverless validation3.ipynb		serverless validation3.ipynb
task1_output.txt		task1_output.txt