Repository navigation
Releases: CS131-BigDataProcessing/mini-assignments-LilaNguyen
Releases · CS131-BigDataProcessing/mini-assignments-LilaNguyen
Release list
V.A14
V.A12
Using Crime dataset, write unit tests and package the unit tests into a module that can be published.
Objectives:
- Write modules and unit test
- Build the package
- Publish package on GitHub
V.A8
Using Airbnb NY datasetLinks to an external site., practice cleaning and preprocessing data
Objective: Perform the following tasks to clean the data set
- Identify missing values
Note that there are trailing spaces/new lines. How to handle? - Remove rows with missing values
Or is it better to replace the missing values with NA or another value? Try writing script for both method depending on the field of interest. - Remove duplicate entries
- Identifying and handling outliers
How to identify outliers? Calculate the mean/median/or mode and use that as the condition - After validating each set, write a shell script to automate. Assume the NY dataset with be merged with other datasets from AirBnb in other regions
V.A11
Practice using shell script to call python script, writing and importing a module, and unit testing.
Objectives:
- Write a python script to define a power function (raising a number to a power) and divide function.
- Create a unit tests for the power and divide functions that cover edge test such as zero powers, zero as a divisor, and negative exponents. Include at least 3 edge cases for power function and 3 edge cases for divide function.
- Run unit test using a shell script
V.A9
Practice task automation
Objective:
- Understand how to use
cronfor scheduling recurring tasks. - Learn how to use
atfor scheduling one-time tasks. - Practice setting up, running, and troubleshooting automated jobs.
V.A6
Extracting data from airline data set
Objective:
- Working with csv files
- Applying awk and grep to extract data
V.A5
Working with structured command; if statements
Objectives:
- Create nested if scripts using conditional statements
- Use positional arguments and pass them to script
- Test script
V.A4
Basic shell scripting
Objectives:
- Display messages
- Work with positional arguments
- Math
- Work with built-in bash calculator (math with floating-point solution)
V.A2
Good luck babe! practice using vi
Objectives:
-Learn to use grep, wc, wildcards
-Learn how to use the parameters of these commands
-Learn stream redirection
-Practice using editors, like vi