Skip to content

edyoda/machine-learning-using-pyspark

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

31 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

machine-learning-using-pyspark

1. Understanding PySpark Ecosystem

  • Big Data
  • Hadoop
  • Spark
  • PySpark
  • Machine Learning using PySpark

2. Foundations of Machine Learning

  • Introduction to Machine Learning
  • Supervised vs Unsupervised
  • Classification vs Regression
  • Data Ingestion
  • Data Wrangling
  • Data Preprocessing
  • Model Training
  • Model Validation
  • Deployment

3. Internal Details of Spark

  • Driver
  • Executors
  • Partitions
  • Jobs
  • Stages
  • Tasks
  • Resilient Distributed Datastructure
  • DataFrames as a High Level Datastructure

4. Low level Understanding using RDD

  • Creation of RDD
  • Transformation methods
  • Aggregation methods
  • Actions
  • Caching
  • Debugging

5. Data Ingestion

  • Loading CSV, JSON & parquet
  • Connecting to databases
  • Getting data from streaming server

5. Data Wrangling using DataFrames

  • Descriptive Statistics
  • Accessing subsets of data - Rows, Columns, Filters
  • Handling Missing Data
  • Dropping rows & columns
  • Handling Duplicates
  • Aggregate functions
  • Merge, Join & Concatenate

6. Data Preprocessing

  • Why Preprocessing ?
  • Scaling Techniques
  • Encoding Techniques
  • Text Processing
  • Dimensionality Reduction
  • Vectorization of Data

7. Regression Learning Models

  • Linear Regression
  • Decision Tree Regressor
  • Random Forest Regressor
  • GBT Regressor
  • Evaluation of Regression Models

8. Classification Learning Models

  • LogisticRegression
  • DecisionTreeClassifier
  • GBT Classifier
  • RandomForestClassifier
  • NaiveBayes
  • MultiLayerPerceptronClassifier
  • Evaluation of Classification Models

9. Clustering Learning Models

  • Motivation behind clustering
  • KMeans
  • GaussianMixtureModel
  • Latent Dirichlet Allocation

10. Recommandation Engine

11. Pipeline & Hyper-parameter Tuning

  • Composite Estimators using Pipelines
  • Model Selection
  • Hyper-parameter Tuning
  • Persisting trained models
  • Deployment

About

Learn Machine Learning using PySpark from scratch

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published