End-to-End Data Project

Instructions

• Clone this repository
• Create a python virtual environment python -m venv env
• Activate the environment source .\env\Scripts\activate
• Install packages pip install -r requirements.txt

Overview

Part 1

Step 1: Setup an FTP Server using WSL
Step 2: Download .csv files from the web using Python
Step 3: Upload downloaded .csv files to the FTP Server either manually or based on a schedule

Running the app:

• Run wsl
• Manually — python web_to_ftp_app.py manual
• Scheduled — python web_to_ftp_app.py scheduled
  
  

Part 2

Step 1: Create a pipeline in SSIS to:

• Download files from FTP Server,
• Load it into PostgreSQL, and
• Merge the tables in PostgreSQL

Entire Pipeline

Merge OFAC Table Data Flow

Step 2: Deploy the Pipeline in SQL Server Agent and create an automated job