machine-learning-using-pyspark

1. Understanding PySpark Ecosystem

Big Data
Hadoop
Spark
PySpark
Machine Learning using PySpark

2. Foundations of Machine Learning

Introduction to Machine Learning
Supervised vs Unsupervised
Classification vs Regression
Data Ingestion
Data Wrangling
Data Preprocessing
Model Training
Model Validation
Deployment

3. Internal Details of Spark

Driver
Executors
Partitions
Jobs
Stages
Tasks
Resilient Distributed Datastructure
DataFrames as a High Level Datastructure

4. Low level Understanding using RDD

Creation of RDD
Transformation methods
Aggregation methods
Actions
Caching
Debugging

5. Data Ingestion

Loading CSV, JSON & parquet
Connecting to databases
Getting data from streaming server

5. Data Wrangling using DataFrames

Descriptive Statistics
Accessing subsets of data - Rows, Columns, Filters
Handling Missing Data
Dropping rows & columns
Handling Duplicates
Aggregate functions
Merge, Join & Concatenate

6. Data Preprocessing

Why Preprocessing ?
Scaling Techniques
Encoding Techniques
Text Processing
Dimensionality Reduction
Vectorization of Data

7. Regression Learning Models

Linear Regression
Decision Tree Regressor
Random Forest Regressor
GBT Regressor
Evaluation of Regression Models

8. Classification Learning Models

LogisticRegression
DecisionTreeClassifier
GBT Classifier
RandomForestClassifier
NaiveBayes
MultiLayerPerceptronClassifier
Evaluation of Classification Models

9. Clustering Learning Models

Motivation behind clustering
KMeans
GaussianMixtureModel
Latent Dirichlet Allocation

10. Recommandation Engine

11. Pipeline & Hyper-parameter Tuning

Composite Estimators using Pipelines
Model Selection
Hyper-parameter Tuning
Persisting trained models
Deployment

Name		Name	Last commit message	Last commit date
Latest commit History 9 Commits
BinaryClassification_adult_income.ipynb		BinaryClassification_adult_income.ipynb
Churn_log_reg.ipynb		Churn_log_reg.ipynb
CruiseShip_LinearRegression.ipynb		CruiseShip_LinearRegression.ipynb
Data Ingestion.ipynb		Data Ingestion.ipynb
Data Preprocessing using PySpark.ipynb		Data Preprocessing using PySpark.ipynb
Data Wrangling using DataFrames.ipynb		Data Wrangling using DataFrames.ipynb
Feature Importance.ipynb		Feature Importance.ipynb
Foundations of Machine Learning.ipynb		Foundations of Machine Learning.ipynb
Internal Details of Spark.ipynb		Internal Details of Spark.ipynb
K-Means.ipynb		K-Means.ipynb
KMeans-Project.ipynb		KMeans-Project.ipynb
Linear_regression_ecommerce.ipynb		Linear_regression_ecommerce.ipynb
Linear_regression_house.ipynb		Linear_regression_house.ipynb
Logistic_Regression_Titanic.ipynb		Logistic_Regression_Titanic.ipynb
Low level Understanding using RDD.ipynb		Low level Understanding using RDD.ipynb
Machine Learning PySpark and MLlib.ipynb		Machine Learning PySpark and MLlib.ipynb
NLP_basics.ipynb		NLP_basics.ipynb
NLP_project.ipynb		NLP_project.ipynb
Pipeline & Hyper-parameter Tuning.ipynb		Pipeline & Hyper-parameter Tuning.ipynb
PySpark_Basic_DataFrame_Operations.ipynb		PySpark_Basic_DataFrame_Operations.ipynb
PySpark_safaribooksonline.ipynb		PySpark_safaribooksonline.ipynb
README.md		README.md
Recommender system.ipynb		Recommender system.ipynb
Regression Models in PySpark.ipynb		Regression Models in PySpark.ipynb
Spark DataFrames Project Exercise_Udemy.ipynb		Spark DataFrames Project Exercise_Udemy.ipynb
Spark Streaming_basics.ipynb		Spark Streaming_basics.ipynb
UDF, UDAF using pandasUDF.ipynb		UDF, UDAF using pandasUDF.ipynb
Understanding PySpark Ecosystem.ipynb		Understanding PySpark Ecosystem.ipynb
decision tree and random forest.ipynb		decision tree and random forest.ipynb
dt_PySpark.ipynb		dt_PySpark.ipynb
learning_spark_MLlib.ipynb		learning_spark_MLlib.ipynb
linear_regression_auto.ipynb		linear_regression_auto.ipynb
linear_regression_wine.ipynb		linear_regression_wine.ipynb
safaribooks_online.ipynb		safaribooks_online.ipynb
spark_lynda_1.ipynb		spark_lynda_1.ipynb

eswarchandt/Machine-Learning-Algorithms-with-Pyspark

Folders and files

Latest commit

History

Repository files navigation

machine-learning-using-pyspark

1. Understanding PySpark Ecosystem

2. Foundations of Machine Learning

3. Internal Details of Spark

4. Low level Understanding using RDD

5. Data Ingestion

5. Data Wrangling using DataFrames

6. Data Preprocessing

7. Regression Learning Models

8. Classification Learning Models

9. Clustering Learning Models

10. Recommandation Engine

11. Pipeline & Hyper-parameter Tuning

About

Resources

Stars

Watchers

Forks

Languages