CSCI181: Deep Learning

About the Instructor

Name	Mike Izbicki (call me Mike)
Email	mizbicki@cmc.edu
Office	Adams 216
Office Hours	Monday 9:00-10:00AM, Tuesday/Thursday 2:30-3:30PM, or by appointment (see my schedule); if my door is open, feel free to come in
Webpage	izbicki.me
Research	Machine Learning (see izbicki.me/research.html for some past projects)
Fun Facts	grew up in San Clemente, 7 years in the navy, phd/postdoc at UC Riverside, taught in DPRK

About the Course

This is a course on deep learning (not big data).

Course Objectives:

Learning objectives:

1. Write basic PyTorch applications
2. Understand the "classic" deep network architectures
3. Use existing models in a "reasonable" way
4. Understand the limitations of deep learning
5. Read research papers published in deep learning
6. Understand what graduate school in machine learning is like
7. (Joke) Understand that Schmidhuber invented machine learning

My personal goal:

1. Find students to conduct research with me

Expected Background:

Necessary:

1. Basic Python programming
2. Linear algebra
3. Calc III
4. Statistics

Good to have:

1. Machine learning / data mining
2. Lots of math
3. Familiarity with Unix and github

Resources:

Textbook:

1. The Deep Learning Book, by Ian Goodfellow and Yoshua Bengio and Aaron Courville; I will assume that you already know all of Part I of this book (basically the equivalent of a data mining/machine learning course)
2. Various papers/webpages as listed below

Deep learning examples:

1. Images / Video

   1. Deoldify
   2. style transfer
   3. more style transfer
   4. dance coreography
   5. StyleGAN
   6. DeepPrivacy
   7. https://thispersondoesnotexist.com/
   8. https://thiscatdoesnotexist.com/
   9. Deep fakes
   10. In Event of Moon Disaster

2. Text

   1. Image captioning
   2. AI Dungeon
   3. https://www.thisstorydoesnotexist.com/
   4. https://translate.google.com

3. Games

   1. AlphaGo
   2. Dota 2
   3. StarCraft 2
   4. MarioCart
   5. Mario

4. Other

   1. iSketchNFill
   2. scrying-pen
   3. Tacotron

The good:

1. most influential research in 2019 is deep learning papers
2. /r/machinelearning
3. recent open source AI programs
4. The state of jobs in deep learning
5. The decade in review

The bad:

1. Machine learning reproducibility crisis
2. Logistic regression vs deep learning in aftershock prediction
3. Pictures of black people
4. NLP Clever Hans BERT
5. Ex-Baidu researcher denies cheating at machine learning competition

Computing resources:

1. Google Colab provides 12 hours of free GPUs in a Jupyter notebook
2. Kaggle provides 30 hours of free GPU
3. I have a 40CPU/8GPU machine that you can access for the course
4. I have another 4CPU/1GPU machine that needs someone to set it up

Videos:

1. 3blue1brown
2. 2 minute papers
3. arxiv insights

Schedule

Week	Date	Topic
1	Tues, 21 Jan	Intro: Examples of Deep Learning
1	Thur, 23 Jan	Automatic differentiation pytorch tutorial part 1 pytorch tutorial part 2 automatic differentiation tutorial einstein summation tutorial NeurIPS paper JMLR paper pytoch: forward mode ad tensorflow: forward mode ad
2	Tues, 28 Jan	Machine Learning Basics (Deep Learning Book Part 1, especially chapters 5.2-5.4)
2	Thur, 30 Jan	Optimization why momentum really works Leon Bottou's SGD paper pytorch loss functions reflections on random kitchen sinks Ali Rahimi's NIPS/NeurIPS 2017 keynote OpenAI switches to PyTorch
3	Tues, 04 Feb	Image: CNNs Stanford lecture slides
3	Thur, 06 Feb	Image: CNNs II An intuitive explanation of CNNs (compare with eigenfaces) The history of neural networks Summer Research
4	Tues, 11 Feb	Regularization
4	Thur, 13 Feb	Image: ResNet Deep Residual Learning for Image Recognition An Overview of ResNet and its Variants CVPR2016 Video More links: Schmidhuber on ResNet I Schmidhuber on ResNet II Baidu scandal at ILSVRC15 What I learned from competing against a convnet on ImageNet MSCOCO
5	Tues, 18 Feb	ResNet continued
5	Thur, 20 Feb	ResNet continued DenseNet Visualizing the Landscape of Neural Network (OpenReview, NIPS/NeurIPS)
6	Tues, 25 Feb	YOLO YOLOv3: An Incremental Improvement YOLO9000: Better, Faster, Stronger and reviews You Only Look Once: Unified Real-Time Object Detection and reviews YOLO video example YOLO video presentation Joseph Redmon's CV Ethical concerns and YOLO The MvMF loss for geolocation: ECML-PKDD paper
6	Thur, 27 Feb	Text: Basic text models bag of words tf-idf n-grams zipf's law hashing trick Python text processing libraries: spacy neuralcoref NLTK TextBlob textstat
7	Tues, 03 Mar	Text: CNNs character-level convolutional networks for text classification very deep convolutional networks for text classification (uses resnets internally) Text: RNNs RNN vs GRU vs LSTM
7	Thur, 05 Mar	Text: Lab exercise
8	Tues, 10 Mar	Text: Seq2seq the unreasonable effectiveness of RNNs what is temperature? sampling strategies in pictures automatic image captioning
8	Thur, 12 Mar	Text: Attention
9	Tues, 17 Mar	NO CLASS: Spring Break
9	Thur, 19 Mar	NO CLASS: Spring Break
10	Tues, 24 Mar	Text: Transformers (paper, blog post)
10	Thur, 26 Mar	TBD
11	Tues, 31 Mar	TBD
11	Thur, 02 Apr	TBD
12	Tues, 07 Apr	TBD
12	Thur, 09 Apr	TBD
13	Tues, 14 Apr	TBD
13	Thur, 16 Apr	TBD
14	Tues, 21 Apr	TBD
14	Thur, 23 Apr	TBD
15	Tues, 28 Apr	TBD
15	Thur, 30 Apr	Project Presentations
16	Thur, 05 May	Project Presentations
16	Thur, 07 May	NO CLASS: Reading Day

Assignments

Week	Weight	Topic
2	10	Rosenbrock Function
3	10	Crossentropy Loss
4	10	CNN
6	10	Image Transfer Learning
7	10	RNN
10	10	Text Transfer Learning
--	10	Reading
15	30	Project

There are no exams in this course.

Late Work Policy:

You lose 10% on the assignment for each day late. If you have extenuating circumstances, contact me in advance of the due date and I may extend the due date for you.

Collaboration Policy:

You are encouraged to work together with other students on all assignments and use any online resources. Learning the course material is your responsibility, and so do whatever collaboration will help you learn the material.

Accommodations for Disabilities

I want you to succeed and I'll make every effort to ensure that you can. If you need any accommodations, please ask.

If you have already established accommodations with Disability Services at CMC, please communicate your approved accommodations to me at your earliest convenience so we can discuss your needs in this course. You can start this conversation by forwarding me your accommodation letter. If you have not yet established accommodations through Disability Services, but have a temporary health condition or permanent disability (conditions include but are not limited to: mental health, attention-related, learning, vision, hearing, physical or health), you are encouraged to contact Assistant Dean for Disability Services & Academic Success, Kari Rood, at disabilityservices@cmc.edu to ask questions and/or begin the process. General information and the Request for Accommodations form can be found at the CMC DOS Disability Service’s website. Please note that arrangements must be made with advance notice in order to access the reasonable accommodations. You are able to request accommodations from CMC Disability Services at any point in the semester. Be mindful that this process may take some time to complete and accommodations are not retroactive. It is important to Claremont McKenna College to create inclusive and accessible learning environments consistent with federal and state law. If you are not a CMC student, please connect with the Disability Services Coordinator on your campus regarding a similar process.