SF DAT15 Course Repository

Course materials for General Assembly's Data Science course in San Francisco, DC (6/15/15 - 8/26/15).

Instructors: Sinan Ozdemir (who is awesome)

Teaching Assistants: Liam Foley, Patrick Foley, and Ramesh Sampath (who are all way more awesome)

Office hours: All will be held in the student center at GA, 225 Bush Street

• Monday 5:15-6:15pm
• Tuesday 6:30-8:30pm
• Wednesday 5:15-6:15pm
• Friday 12:30-2:30pm
• Saturday 10:00am-12:00pm

Course Project information

Monday	Wednesday
6/15: Introduction / Expectations / Git Intro	6/17: Python
6/22: Data Science Workflow / Pandas	6/24: More Pandas
6/29: Intro to ML / Numpy / KNN	7/1: Scikit-learn / Model Evaluation Project Milestone: Question and Data Set HW Homework 1 Due
7/6: Linear Regression	7/8: Logistic Regression
7/13: Working on a Data Problem	7/15: Clustering
7/20: Natural Language Processing	7/22: Naive Bayes Milestone: First Draft Due
7/27: Decision Trees	7/29:Ensembling Techniques
8/3: Recommendation Engines Milestone: Peer Review Due	8/5: Databases / MapReduce
8/10: Dimension Reduction	8/12: Ensemble Techniques
8/17: Web Development with Flask	8/17: Neural Networks
8/24: Projects	8/26: Projects

Installation and Setup

• Install the Anaconda distribution of Python 2.7x.
• Install Git and create a GitHub account.
• Once you receive an email invitation from Slack, join our "SF_DAT_15 team" and add your photo!

Resources

• PEP 8 - Style Guide for Python

Class 1: Introduction / Expectations / Git Intro

• Introduction to General Assembly
• Course overview: our philosophy and expectations (slides)
• Git overview: (slides)
• Tools: check for proper setup of Git, Anaconda, overview of Slack

Homework:

• Resolve any installation issues before next class.
• Make sure you have a github profile and created a repo called "SF_DAT_15"
• Clone the class repo (this one!)
• Review this code for a recap of some Python basics.

Optional:

• Read Analyzing the Analyzers for a useful look at the different types of data scientists.
• Look over Data science overview (slides)
• Read about some Markdown Techniques

Class 2: Python

• Brief overview of Python environments: Python interpreter, IPython interpreter, Spyder, Rodeo
• Python quiz (code)
• Check out some iPython Notebooks!
• Working with data in Python in Spyder
  • Today's data is brought to you by Nate Silver's blog, FiveThirtyEight
  • and also from the Open Weather Data Project They have a great API!!
  • Reading, writing, and graphing oh my! in Python (code)
• Lab on files and API usage

Homework:

• Finish the Python exercise
• Read through the project page in detail.
• Review a few projects from past Data Science courses to get a sense of the variety and scope of student projects.

Optional:

• If you need more practice with Python, review the "Python Overview" section of A Crash Course in Python, work through some of Codecademy's Python course, or work through Google's Python Class and its exercises.
• For more project inspiration, browse the student projects from Andrew Ng's Machine Learning course at Stanford.

Resources:

• Online Python Tutor is useful for visualizing (and debugging) your code.

Class 3: Data Science Workflow / Pandas

Agenda

• Slides on the Data Science workflow here
  • Data Science Workflow
• Intro to Pandas walkthrough here
  • I will give you semi-cleaned data allowing us to work on step 3 of the data science workflow
  • Pandas is an excellent tool for exploratory data analysis
  • It allows us to easily manipulate, graph, and visualize basic statistics and elements of our data
  • Pandas Lab!

Homework

• Begin thinking about potential projects that you'd want to work on. Consider the problems discussed in class today (we will see more next time and next Monday as well)
  • Do you want a predictive model?
  • Do you want to cluster similar objects (like words or other)?

Resources:

• Pandas
  • Split-Apply-Combine pattern
  • Simple examples of joins in Pandas
  • Check out this excellent example of data wrangling and exploration in Pandas
    • For an extra challenge, try copying over the code into your own .py file
  • To learn more Pandas, review this three-part tutorial
  • For more on Pandas plotting, read the visualization page from the official Pandas documentation.

Class 4 - More Pandas

Agenda

• Class code on Pandas here
• We will work with 3 different data sets today:
  • the UFO dataset (as scraped from the reporting website
  • Fisher's Iris Dataset (as cleaned from a machine learning repository
  • A dataset of (nearly) every FIFA goal ever scored (as scraped from the website)
• Pandas Lab! here

####Homework

• Please review the readme for the first homework. It is due NEXT Wednesday (7/1/2015)
• The one-pager for your project is also due. Please see project guidelines

Class 5 - Intro to ML / Numpy / KNN

####Agenda

• Intro to numpy code
  • Numerical Python, code adapted from tutorial here
  • Special attention to the idea of the np.array
• Intro to Machine Learning and KNN slides
  • Supervised vs Unsupervised Learning
  • Regression vs. Classification
• Iris pre-work code and code solutions
  • Using numpy to investigate the iris dataset further
  • Understanding how humans learn so that we can teach the machine!
• Lab to create our own KNN model

####Homework

• The one page project milestone as well as the pandas homework!
• Read this excellent article, Understanding the Bias-Variance Tradeoff, and be prepared to discuss it in class on Wednesday. (You can ignore sections 4.2 and 4.3.) Here are some questions to think about while you read:
  • In the Party Registration example, what are the features? What is the response? Is this a regression or classification problem?
  • In the interactive visualization, try using different values for K across different sets of training data. What value of K do you think is "best"? How do you define "best"?
  • In the visualization, what do the lighter colors versus the darker colors mean? How is the darkness calculated?
  • How does the choice of K affect model bias? How about variance?
  • As you experiment with K and generate new training data, how can you "see" high versus low variance? How can you "see" high versus low bias?
  • Why should we care about variance at all? Shouldn't we just minimize bias and ignore variance?
  • Does a high value for K cause over-fitting or under-fitting?

Resources:

• For a more in-depth look at machine learning, read section 2.1 (14 pages) of Hastie and Tibshirani's excellent book, An Introduction to Statistical Learning. (It's a free PDF download!)

  Class 6: scikit-learn, Model Evaluation Procedures

• Introduction to scikit-learn with iris data (code)

• Exploring the scikit-learn documentation: user guide, module reference, class documentation

• Discuss the article on the bias-variance tradeoff

• Look as some code on the bias variace tradeoff

  • To run this, I use a module called "seaborn"
  • To install to anywhere in your terminal (git bash) and type in sudo pip install seaborn

• Model evaluation procedures (slides, code)

Homework:

• Keep working on your project. Your data exploration and analysis plan is due in three weeks!

Optional:

• Practice what we learned in class today!
  • If you have gathered your project data already: Try using KNN for classification, and then evaluate your model. Don't worry about using all of your features, just focus on getting the end-to-end process working in scikit-learn. (Even if your project is regression instead of classification, you can easily convert a regression problem into a classification problem by converting numerical ranges into categories.)
  • If you don't yet have your project data: Pick a suitable dataset from the UCI Machine Learning Repository, try using KNN for classification, and evaluate your model. The Glass Identification Data Set is a good one to start with.
  • Either way, you can submit your commented code to your SF_DAT_15_WORK, and we'll give you feedback.

Resources:

• Here's a great 30-second explanation of overfitting.
• For more on today's topics, these videos from Hastie and Tibshirani are useful: overfitting and train/test split (14 minutes), cross-validation (14 minutes). (Note that they use the terminology "validation set" instead of "test set".)
  • Alternatively, read section 5.1 (12 pages) of An Introduction to Statistical Learning, which covers the same content as the videos.
• This video from Caltech's machine learning course presents an excellent, simple example of the bias-variance tradeoff (15 minutes) that may help you to visualize bias and variance.

Class 7: Linear Regression

• Linear regression (notebook, notebook code)

Homework:

• (Optional) questions here with the Yelp reviews data. It is not required but your next homework will involve this dataset so it would be helpful to take a look now!
• Watch these videos on probability and odds (8 minutes) if you're not familiar with either of those terms.
• Read these excellent articles from BetterExplained: An Intuitive Guide To Exponential Functions & e and Demystifying the Natural Logarithm (ln).

Resources:

• Setosa has an excellent interactive visualization of linear regression.
• To go much more in-depth on linear regression, read Chapter 3 of An Introduction to Statistical Learning, from which this lesson was adapted. Alternatively, watch the related videos or read my quick reference guide to the key points in that chapter.
• To learn more about Statsmodels and how to interpret the output, DataRobot has some decent posts on simple linear regression and multiple linear regression.
• This introduction to linear regression is much more detailed and mathematically thorough, and includes lots of good advice.
• This is a relatively quick post on the assumptions of linear regression.
• John Rauser's talk on Statistics Without the Agonizing Pain (12 minutes) gives a great explanation of how the null hypothesis is rejected.
• A major scientific journal recently banned the use of p-values:
  • Scientific American has a nice summary of the ban.
  • This response to the ban in Nature argues that "decisions that are made earlier in data analysis have a much greater impact on results".
  • Andrew Gelman has a readable paper in which he argues that "it's easy to find a p < .05 comparison even if nothing is going on, if you look hard enough".

Class 8: Logistic Regression

• Logistic regression (notebook, notebook code)
• Confusion matrix (slides)
• Exercise with Titanic data (instructions, solution)

Homework:

• If you aren't yet comfortable with all of the confusion matrix terminology, watch Rahul Patwari's videos on Intuitive sensitivity and specificity (9 minutes) and The tradeoff between sensitivity and specificity (13 minutes).

Resources:

• To go deeper into logistic regression, read the first three sections of Chapter 4 of An Introduction to Statistical Learning, or watch the first three videos (30 minutes) from that chapter.
• For a math-ier explanation of logistic regression, watch the first seven videos (71 minutes) from week 3 of Andrew Ng's machine learning course, or read the related lecture notes compiled by a student.
• For more on interpreting logistic regression coefficients, read this excellent guide by UCLA's IDRE and these lecture notes from the University of New Mexico.
• The scikit-learn documentation has a nice explanation of what it means for a predicted probability to be calibrated.
• Supervised learning superstitions cheat sheet is a very nice comparison of four classifiers we cover in the course (logistic regression, decision trees, KNN, Naive Bayes) and one classifier we do not cover (Support Vector Machines).
• This simple guide to confusion matrix terminology may be useful to you as a reference.

Class 9: Working a Data Problem

• Today we will work on a real world data problem! Our data is stock data over 7 months of a fictional company ZYX including twitter sentiment, volume and stock price. Our goal is to create a predictive model that predicts forward returns.

• Project overview (slides)

  • Be sure to read documentation thoroughly and ask questions! We may not have included all of the information you need...

Class 10: Clustering and Visualization

• The slides today will focus on our first look at unsupervised learning, K-Means Clustering!
• The code for today focuses on two main examples:
  • We will investigate simple clustering using the iris data set.
  • We will take a look at a harder example, using Pandora songs as data. See data. See code here
  • Checking out some of the limitations of K-Means Clutering here

Homework:

• HW2 and Project Milestone 2 are due in one week!
• Download all of the NLTK collections.
  • In Python, use the following commands to bring up the download menu.
  • import nltk
  • nltk.download()
  • Choose "all".
  • Alternatively, just type nltk.download('all')
• Install two new packages: textblob and lda.
  • Open a terminal or command prompt.
  • Type pip install textblob and pip install lda.

Resources:

• Introduction to Data Mining has a nice chapter on cluster analysis.
• The scikit-learn user guide has a nice section on clustering.

##Class 11: Natural Language Processing

Agenda

• Naural Language Processing is the science of turning words and sentences into data and numbers. Today we will be exploring techniques into this field
• code showing topics in NLP
• lab analyzing tweets about the stock market

Homework:

• Read Paul Graham's A Plan for Spam and be prepared to discuss it in class on Wednesday. Here are some questions to think about while you read:
  • Should a spam filter optimize for sensitivity or specificity, in Paul's opinion?
  • Before he tried the "statistical approach" to spam filtering, what was his approach?
  • How exactly does his statistical filtering system work?
  • What did Paul say were some of the benefits of the statistical approach?
  • How good was his prediction of the "spam of the future"?
• Below are the foundational topics upon which Wednesday's class will depend. Please review these materials before class:
  • Confusion matrix: a good guide roughly mirrors the lecture from class 10.
  • Sensitivity and specificity: Rahul Patwari has an excellent video (9 minutes).
  • Basics of probability: These introductory slides (from the OpenIntro Statistics textbook) are quite good and include integrated quizzes. Pay specific attention to these terms: probability, sample space, mutually exclusive, independent.
• You should definitely be working on your project! First draft and HW2 are both due Wednesday!!

##Class 12: Naive Bayes Classifier

Today we are going over advanced metrics for classification models and learning a brand new classification model called naive bayes!

Agenda

• Learn about ROC/AUC curves
  • Slides here
  • Code here
• Learn the Naive Bayes Classifier
  • Slides here
  • Code here
  • In the code file above we will create our own spam classifier!

Resources

• Video on ROC Curves (12 minutes).
• My buddy's blog post about the ROC video includes the complete transcript and screenshots, in case you learn better by reading instead of watching.

##Class 13: Decision Trees

We will look into a slightly more complex model today, the Decision Tree.

Agenda

• Slides here

• Code here

Homework

• Project reviews due August 3rd!

Resources

• Chapter 8.1 of An Introduction to Statistical Learning also covers the basics of Classification and Regression Trees
• The scikit-learn documentation has a nice summary of the strengths and weaknesses of Trees.
• For those of you with background in javascript, d3.js has a nice tree layout that would make more presentable tree diagrams:
  • Here is a link to a static version, as well as a link to a dynamic version with collapsable nodes.
  • If this is something you are interested in, Gary Sieling wrote a nice function in python to take the output of a scikit-learn tree and convert into json format.
  • If you are intersted in learning d3.js, this a good tutorial for understanding the building blocks of a decision tree. Here is another tutorial focusing on building a tree diagram in d3.js.
• Dr. Justin Esarey from Rice University has a nice video lecture on CART that also includes an R code walkthrough

Class 15: Recommenders

• Recommendation Engines slides
• Recommendation Engine Example code

Resources:

• The Netflix Prize
• Why Netflix never implemented the winning solution
• Visualization of the Music Genome Project
• The People Inside Your Machine (23 minutes) is a Planet Money podcast episode about how Amazon Mechanical Turks can assist with recommendation engines (and machine learning in general).

Class 16: Databases and Mapreduce

• Slides slides
• Lab code

Class 17: Dimension Reduction

• Fun with indeed here
• PCA
  • Slides
  • Code: PCA and SVD
  • Code: image compression with PCA (original source)

Resources

• Some hardcore math in python here
• PCA using the iris data set here and with 2 components here
• PCA step by step here
• Check out Pyxley for our guest speaker's (Nick Kridler) talk on Wednesday

Class 18: Ensembling

• Talk from Nick Kridler
• Ensembling (IPython notebook)
• BONUS: Regularization (IPython notebook)
  • Bonus advanced sklearn code here

Resources:

• scikit-learn documentation: Ensemble Methods
• Quora: How do random forests work in layman's terms?

Class 19: Web Development

• slides here
• We will be working with the flask app found here

Resources:

• MVC Architecture blog post
• More on using Flask and Heroku here (Note you can ignore the virtual environment stuff, unless you want a challenge!)

Homework

• Try to deploy your own ML model to Heroku!
• Read an intro to Neural Networks here
• And this intro to SVM

Class 20: Neural Networks and SVM

Agenda

• slides here

• code here

• Lab here

Resources

• SVM Margins Example here
• SVM digits was adapted from here
• Google Deep Dream: why does it always see dogs?!
• Deep Dream Generator
• Most used non-sklearn ANN PyBrain
• Step by Step back propagation here
• Project time signups here

##Project Info

• Everyone will have a maximum of 15 minutes to present including Q&A
• Please sign up for a slot if you haven't done so here
  • If you don't want to be crunched for time, try going on Monday :)
  • If you don't sign up by end of class today (Wednesday 8/19) we will assign you a slot
• Final Projects are mandatory if you want a certification of completion from General Assembly
• Remember you must submit both a presentation as well as a write up (What a write up you never mentioned that!) I did and also it is in the project requirements :)