Python for Data Engineering Course

01: Python Basics

• Introduction

• First Python program

• Basic Data types (int, float, str, bool)

• Variables, Constants and operators

• Control flow (if statements, loops)

  • if Statement
  • for Loop
  • while Loop
  • Control flow examples

• Data structures (lists, tuples, dictionaries, sets)

  • Lists
  • Tuples
  • Dictionaries
  • Sets

• String manipulation

• Modular Programming (introduction)

  • Functions
  • Modules

• Error handling

• Assignments: Python basics assignments

  02: Data Manipulation, Analysis and Visualization

• Introduction to Data Analysis

• Setting Up Jupyter Notebook and Pandas

• Data Manipulation with Pandas

  • Introduction to Pandas
  • Basics statistics in data analysis (descriptive statistics, shape of distribution, inferential statistics)
  • Pandas Series
  • Pandas DataFrame
  • Reading and writing data with Pandas
  • Accessing and selecting data
  • Data manipulation (filtering, sorting, grouping)
  • Data Cleaning
  • Merge, Join and Concatenating

• Numerical computations with NumPy

  • NumPy array operations (slicing, indexing)
  • NumPy array manipulation
  • Working with random numbers in NumPy
  • Analyzing sales data with NumPy

• Data visualization Basics

  • Line plot, Scatter plot, and Bar plot
  • Histogram, Box plot, and Violin plot

• Assignment 1 : Analyzing E-commerce Orders using Pandas

• Assignment 2 : Exploratory Data Analysis (EDA) using Pandas, NumPy, and Matplotlib

04: Web Scraping and APIs

• Introduction to Web Scraping and APIs
• Basics of HTML, CSS, and JavaScript
  • Introduction to HTML
  • Introduction to CSS
  • Introduction to JavaScript
• Getting Started with Web Scraping Using Python
  • HTTP Requests
  • HTTP Response
  • Fetch Web Pages Using Requests Library
  • Parsing HTML with BeautifulSoup
• Dealing with Pagination and Infinite Scrolling
• Working with APIs
  • Handling API Response
  • Error Handling and Status Codes
• Assignment 1: Scraping Jobinja website, fetch last year jobs information and save into a CSV file.

05: Working with Data Sources, Storages, and Serialization

• Introduction to I/O (Input/Output)
  • Common File Formats
• Importance of File I/O (Input/Output)
• Basic Concepts of File I/O
  • File Paths and Directories
  • Opening and Closing Files
• Text Files
• CSV Files
• JSON (JavaScript Object Notation) Files
• XML Files
• Excel Files
• Binary Files
• Database File Formats (SQLite)
• Parquet and pyArrow

06: Object-Oriented Programming

Object-Oriented Programming (OOP)

• Object-oriented programming concepts (classes, objects, inheritance, polymorphism)
• Design patterns and best practices in OOP

07: SQL & NoSQL Databases with Python

Working with SQL Databases

• Introduction to SQL and relational databases
• SQL basics (SELECT, FROM, WHERE, JOIN)
• Creating and managing databases, tables, and indexes
• CRUD operations (Create, Read, Update, Delete)
• Connecting to databases
• Executing SQL queries
• Fetching and manipulating data with SQL
• Using SQLAlchemy for database interaction

Working with NoSQL Databases

• Understanding NoSQL databases (e.g., MongoDB, Redis)
• Connecting to NoSQL databases
• Querying and manipulating data in NoSQL databases
• Handling document-based and key-value data models

Assignment: Ass7

08: Building Data Pipelines

Data Pipelines

• ETL (Extract, Transform, Load)
• Understanding data pipelines and their components
• Designing and architecting data pipelines
• Implementing data ingestion, transformation, and loading (ETL)

Assignment: Ass8

Capstone Project

Project Development

• Apply all the concepts learned in a real-world data engineering project
• Work with various data sources including web data and APIs
• Implement ETL pipelines, data processing, and analysis using Python libraries and tools