2110531 Data Science and Data Engineering Tools @Chula 2022

Short links for exercises:

Week1: Intro to Numpy, Pandas

1. Numpy:

2. Pandas:

3. Pandas with Youtube stat data:

4. (Advanced) Pandas with Youtube stat data:

Assignment (Pandas with Youtube stat data):

Week2: Data Preparation

1. EDA:

2. Impute Missing Value:

3. Split Train/Test:

4. Outliers with Log:

5. Outliers with Log (Titanic DataSet):

Assignment:

Week3-4: Traditional ML

1. Decision Trees:

2. Linear Regression:

3. Logistic Regression:

4. Neural Network:

5. K Nearest Neighbors:

6. SVM:

7. Save and Load Model:

8. K-Means:

9. Market-Basket Analysis:

Assignment for Week3 (Safe to eat or deadly poison?):

Week5-6: Intro to Deep Learning

1. Image classification (basic): flower classification

2. Image classification (advanced): flower classification

3. Semantic Segmentation (UNET): The Oxford-IIIT pet dataset

4. LSTM: Stock price prediction

5. SARIMAX: PM2.5 forecasting

Assignment (Fashion MNIST):

Week8: Data Storage with Redis

Redis Example using local data

Assignment (connect to redis server)

Week9: Data Storage with Redis

1. Basic Webpage Scarping

2. Wikipeia Data Extraction

3. Settrade Rest API

4. Twitter Data Extraction

5. Selenium

Assignment (Counting วันพระ)

Week10: Data Ingestion with Kafka

1. Several simple examples including both produxer and consumer in simple folder

2. Complex example in complex folder

3. AVRO Producer and Consumer

4. Group example in group folder

Assignment (Transaction Verifier)

Note: Do not forget to upload the following schema files to your Colab

• transaction.avsc
• submit.avsc
• result.avsc

Week11: Big Data Processing with Spark

1. Basic Spark

Note: Do not forget to upload the following data file to your Colab

• (star-wars.txt)[https://github.com/kaopanboonyuen/2110531_DataScience_2022s1/blob/main/code/week11_spark/star-wars.txt]

2. Spark SQL

Note: Do not forget to upload the following data file to your Colab

• (bank-additional-full.csv)[https://github.com/kaopanboonyuen/2110531_DataScience_2022s1/blob/main/code/week11_spark/bank-additional-full.csv]

3. Spark ML

Note: Do not forget to upload the following data file to your Colab

• (bank-additional-full.csv)[https://github.com/kaopanboonyuen/2110531_DataScience_2022s1/blob/main/code/week11_spark/bank-additional-full.csv]

Assignment (Analyze IMDB)

Note: Do not forget to upload the following data file to your Colab

• (netflix-rotten-tomatoes-metacritic-imdb.csv)[https://github.com/kaopanboonyuen/2110531_DataScience_2022s1/blob/main/code/week11_spark/assignment/netflix-rotten-tomatoes-metacritic-imdb.csv]

Week12: Ops Stars

1. Several airflow examples in (airflow folder)[https://github.com/kaopanboonyuen/2110531_DataScience_2022s1/tree/main/code/week12_orchestration/airflow]

2. Several fastapi examples in (fastapi folder)[https://github.com/kaopanboonyuen/2110531_DataScience_2022s1/tree/main/code/week12_orchestration/fastapi]

Reference:

1. https://www.kaggle.com/code
2. https://www.tensorflow.org/tutorials
3. https://github.com/topics/machine-learning
4. https://archive.ics.uci.edu/ml/datasets.php