Modern Methods of Applied Statistics (Spring 2023) STAT 34800

Instructor: Aaron Schein
TAs: Wei Kuang, Sounak Paul
Term: Spring 2023
The University of Chicago

---

Logistics:

• Time: Tuesday and Thursday, 3:30am-4:50pm
• Place: Eckhart room 133
• Office hours:
  • Wei: Monday 2-3 pm (Jones 304)
  • Sounak: Friday 3-4 pm (Jones 308)

Assignments

• Assignment 1: Bayesian decision theory. Due Monday April 3 at 11:59pm on GradeScope.
• Assignment 2: Logistic regression and beta-binomial updating. Due Monday April 10 at 11:59pm on GradeScope.
• Assignment 3: Exponential families, conjugacy, and entropy. Due Monday April 17 at 11:59pm on GradeScope.
• Assignment 4: HMMs, belief propagation, variable elimination. Due Tuesday May 2 at 11:59pm on GradeScope.
• Assignment 5: Bayesian mixture models, Gibbs sampling, EM. Due Sunday May 21 at 6:00pm on GradeScope.

Schedule

Lecture 1 (March 23): Introduction & the hunt for the USS Scorpion

• Suggested readings:
  • Freedman (1994): "Some issues in the foundations of statistics"
  • Freedman "Notes on the Dutch Book argument"
  • Chap 15 "The Navy Searches" of The Theory That Would Not Die (see Canvas for e-copy)
• Materials:
  • Slides

Lecture 2 (March 25): Bayesian and frequentist decision theory

• Suggested readings:
  • Chap 1 of Berger (1985) Statistical Decision Theory and Bayesian Analysis (see Canvas for e-copy)

Lecture 3 (March 28): Supervised learning, posterior updating, empirical Bayes

• Suggested readings:
  • Chap 2 of Hastie et al. (2009) Elements of Statistical Learning
• Materials:
  • Slides

Lecture 4 (March 30): Conjugacy and the exponential families

• Suggested readings:
  • David Blei's lecture notes on exponential families

Lecture 5 (April 4): Prior & predictive distributions, exponential families, information theory

• Suggested readings:
  • Fong & Holmes (2020) On the marginal likelihood and cross-validation
  • Chap 9.1-9.2 of Murphy (2012) Machine Learning: A Probabilistic Perspective
  • Chap 14 of John Duchi's lecture notes on info theory

Lecture 6 (April 6): Conditional entropy, mutual information, symbol codes

• Suggested readings:
  • Chap 1-2, 4.1-4.3, 5.1-5.3, 8 of MacKay (2005) Information Theory, Inference, and Learning Algorithms
  • Wikipedia article on Z-channels

Lecture 7 (April 11): Probabilistic graphical models (PGMs)

• Suggested readings:
  • Chap 2 of Jordan (2003) An Introduction to Probabilistic Graphical Models
  • David Blei's lecture notes on PGMs

Lecture 8 (April 13): Inference in PGMS: variable elimination, belief propagation

• Suggested readings:
  • Chap 3 and chap 4 of Jordan (2003) An Introduction to Probabilistic Graphical Models
  • David Blei's lecture notes on inference in PGMs

Lecture 9 (April 18): HMMs and the forwards-backwards algorithm

• Suggested readings:
  • David Blei's lecture notes on inference in PGMs
  • Scott Linderman's slides on HMMs
  • Ramesh Sridharan's lecture notes on HMMs
  • Chap 13 (specifically 13.2.2-13.2.4) of Bishop (2006) Pattern Recognition and Machine Learning

Midterm (April 20)

Lecture 10 (April 25): Baum-Welch, EM, mixture models

• Suggested readings:
  • Chap 9 of Bishop (2006) Pattern Recognition and Machine Learning
  • Scott Linderman's slides on EM
  • Scott Linderman's slides on HMMs

Lecture 11 (April 27): Bayesian mixture models, conditional conjugacy, EM

• Suggested readings:
  • Scott Linderman's slides on Bayesian mixtures
  • David Blei's lecture notes on Bayesian mixtures
  • Last lecture's readings

Lecture 12 (May 2): Gibbs sampling and MCMC

• Suggested readings:
  • Chap 11.1.6-11.3 of Bishop (2006) Pattern Recognition and Machine Learning
  • David Blei's lecture notes on Bayesian mixtures and Gibbs sampling
  • Matthew Stephen's lecture notes on Gibbs sampling
  • Scott Linderman's slides on MCMC
  • Gregory Gunderson's blogpost on ergodic Markov chains

Lecture 13 (May 4): Gibbs sampling (cont.): missing data, collapsed Gibbs, Geweke testing

• Suggested readings:
  • Chap 24.2 of Murphy (2012) Machine Learning: A Probabilistic Perspective
  • David Blei's lecture notes on Bayesian mixtures and Gibbs sampling
  • Stephens (2000): "Dealing with label switching in mixture models"
  • Geweke (2004): "Getting it Right: Joint Distribution Tests of Posterior Simulators"
  • Roger Grosse's blogpost on Geweke testing

Lecture 14 (May 9): Variational inference

• Suggested readings:
  • Blei, Kucukelbir & McAuliffe (2017) "Variational inference: A review for statisticians"
  • Chap 10.1-10.2, 10.4 of Bishop (2006) Pattern Recognition and Machine Learning
  • Scott Linderman's slides on VI

Lecture 15 (May 11): Admixture models and LDA

• Suggested readings:
  • Chap 27.3 of Murphy (2012) Machine Learning: A Probabilistic Perspective
  • Scott Linderman's slides on CAVI for LDA
  • Jeffrey Miller's slides on CAVI for LDA
  • Matt Gormley's slides on LDA
  • Blei, Ng, Jordan (2003) "Latent Dirichlet Allocation"
  • Pritchard, Stephens, Donnelly (2000) "Inference of Population Structure Using Multilocus Genotype Data"

Lecture 16 (May 16): Non-negative matrix factorization (NMF) and Poisson factorization

• Suggested readings:
  • Lee and Seung (1999) "Learning the parts of objects by non-negative matrix factorization"
  • Gillis (2014) "The How and Why of Nonnegative Matrix Factorization"
  • Gopalan et al. (2014) "Scalable Recommendation with Poisson Factorization"

Lecture 17 (May 18): Wrap-up, review, future topics

• Materials:
  • Slides