diff --git a/policies.md b/policies.md new file mode 100644 index 0000000..875a033 --- /dev/null +++ b/policies.md @@ -0,0 +1,165 @@ +# Class-specific policies + +Advanced Convex Optimization, APPM 5630. University of Colorado Boulder, Spring 2021. + +[Canvas link](https://canvas.colorado.edu/) (via SSO) + +## Details in a nutshell +This class is expected to sometimes meet remotely, sometimes in person, depending on CU's current COVID-19 status. + +When we meet **remotely**, this is a flipped class. Students are responsible for watching pre-recorded videos made by the instructor (and get attendance points by filling out PlayPosit quizzes). At the usual classtime, we meet via zoom to have a live session which is either for **demos** or **Q&A**, e.g., more like office hours. During remote weeks, we'll only have **one hour of office hours** since the main lecture is like office hours. The zoom link is listed in [Canvas](https://canvas.colorado.edu/). + +When we meet **in person**, this is a traditional class. The instructor gives a lecture in the classroom. We will have **three hours of office hours** on these weeks; the office hours will be via zoom since we are not allowed to actually meet in an office due to COVID-19 restrictions (same zoom link as used for remote sessions). If a student does not want to attend class in person, they may request that we setup a live-stream via zoom, which can be recorded. + + +## Instructor and contact Information + +- The instructor is Stephen Becker, Assistant Professor of Applied Mathematics +- Contact him at +- There are no teaching assistants (TAs) for this course + +## Meeting times +Meeting time: MWF 10:20 -- 11:10 AM. When we meet in person, the location is ECCR 1B40. When we meet remotely, use the zoom link (to be found in [Canvas](https://canvas.colorado.edu/)). + +## Office Hours +Prof. Becker's office hours (all held via zoom): + +**In-person weeks** +- 3-5 PM Thursdays +- 1-2 PM Fridays + +**Remote weeks** +- 4-5 PM Thursdays + + +## Course Topic Information and Syllabus +See the separate [syllabus document](syllabus.md). + +## Estimated Workload +This is intended to have a normal workload for an elective course, lower than the workload for a core course + +## Prereqs +APPM 3310 “Matrix Methods” or similar; but APPM 4440 “Real Analysis” or equivalent (e.g., APPM 5440) is suggested. A first-course on optimization like CSCI 5254 “Convex Optimization” is helpful but not necessary. + +## Textbook +See the separate [syllabus document](syllabus.md). + +## Recitations +There are no recitations + +## Exams +There are no exams + +## Projects + +There is one project, worth 20% of your grade. The last few weeks, there will be no/less homework and instead there is a final project, which is open-ended. Students are encouraged to form groups (up to three people). It can be either theory or computation. Topics should be discussed with the instructor. The final week, groups will give brief (about 10 min) presentations on their topic. + +## Homeworks + +There will be bi-weekly double-homeworks, consistent of two parts (theory and programming), so we count this as two homeworks. You are allowed to drop two single homeworks (this will be done automatically), meaning either one double-homework or mix-and-match. + +## Grading +- 10% for participation + - When we meet **in-person**, this is either an **attendance** score or a **reading** score (via Perusall) + - If you miss a class for a legitimate reason (sickness, important travel), you can be excused. Doctor's notes are not needed to document illness (traveling to the student health center when only mildly sick can in fact be dangerous, as you spread disease to very sick people). + - When we meet **remotely**, this is a PlayPosit score: fill out the PlayPosit quizzes that are wrapped around some of the posted videos. Access these links via Canvas. +- 70% Homework. + - Late homework is not accepted, but you are allowed two free "dropped" homeworks. Your lowest-scoring homeworks will be automatically dropped. See below for caveat +- 20% Project + +The overall grade may be **curved** as appropriate (not guaranteed), but note that there is no set "quota" of A's, B's, etc., so you are not directly competing with your classmates. + +**Note**: To get a grade of "A", you **cannot drop any homeworks**. If you have to drop a homework, then the highest grade you can get is "A-". (There are no "A+" grades at CU) + +## Late assignment and cheating policy + +In general, **late homework** assignments are not accepted; instead, you can use your two dropped homeworks. Under exceptional circumstances (such as serious illness, including COVID-19, or serious family issues), homework can be turned in late. If the reason is foreseeable (e.g., planned travel), you must contact the instructor in advance. + +Examples: +- Your sister is getting married and you have to travel out-of-state. That's great, but this is when you use the one dropped homework. This is foreseeable, and not an "emergency", so it does not count as an exceptional circumstance. +- A close family member becomes infected with COVID-19 and you have to return to your home country to take care of family. This *does* count as an exceptional circumstance. Please email the instructor to discuss arrangements. + +**Cheating** is not acceptable. Take-home exams and homeworks are easy to cheat on if you really want to, but as this is an upper-division course, I am relying on the fact that students are here to learn (and paying the university to do so), and thus cheating does not make sense. Cheating does not hurt the instructor, it hurts the student (and hurts the grades of honest classmates). + +If a student is caught cheating, on the first occurrence, the penalty ranges from losing points on the item in question (like one test problem; this is for very minor infractions) to losing all points for the assignment (i.e., the entire homework or entire exam). Students may be referred to the honor council. On the second occurrence of cheating, similar penalties may apply, and additionally the student may fail the class, at the instructor's discretion. + +"Minor infractions" include not following the instructions during an exam (in person or remote). For example, if the instructions on a remote test are to keep your microphone on and your hands in sight of your webcam, then failing to follow these instructions construes a minor infraction, and (even though cheating may not be proven) you are subject to losing points. + +On homeworks, you are free to **collaborate** with other students, and to use resources like the internet appropriately. However, you must do your own work. There is a gray area between collaboration and cheating, and we rely on the students' and instructors discretion. Copying code verbatim is never permissible. You should be writing up your own work, and explaining answers in your own words. Snippets of code are allowed to be similar (sometimes there is only one good way to do it), but longer chunks of code should never be identical. If not expressly forbidden by the assignment, you may use the internet, but you may never post for help on online forums. (Regarding forums, please use our Piazza website if you want a Q&A forum). + +Cheating is not usually an issue in this class, and I have faith that students will continue to act appropriately. + + + +## Course website + +We will use [github](https://github.com/stephenbeckr/convex-optimization-class) for public content (notes, demos, syllabus), and use CU's default LMT **Canvas** for private content like grades and homework solutions. Canvas will also be used to organize things, like the **pre-recorded lectures** (posted as unlisted on Youtube, with links in Canvas), comments made via **gradescope**, and Q&A forums via **piazza**. + +## Notes +When the class meets remotely, handwritten notes will be posted to the github website. When the class meets in person, notes will probably **not** be posted. + +In Fall 2018, Mitchell Krock started some [typed notes](https://github.com/stephenbeckr/convex-optimization-class/tree/master/TypedNotes) that are hosted on our github website; you are free to use these. You can also add to them; just make a pull request on github, or if you want to make a lot of changes, ask to be added as a collaborator on github. + + + +## Online behavior +On zoom, please have your webcam on if at all possible +- Valid reasons for not having the camera on: to protect the privacy of your family or roommate, if you cannot have a virtual background +- Invalid reason: you don't feel like it, or you didn't wash your hair. + +We have the same standards of behavior as we would in a classroom: appropriate attire, appropriate and not distracting virtual backgrounds, verbal and chat remarks should be respectful, etc. Real-world backgrounds should be appropriate and professional (please, no drugs or alcohol behind you). + +It's always important to have respectful remarks, and even more so in an online setting, since it is easier to get carried away with chat comments since you cannot see the effect on other people. + +If we enable private chat on zoom, remember that the zoom host can later see even "private" chats. Inappropriate or inconsiderate remarks, even on private chats, are not allowed. + + +## Dropping the Course +Advice from your department advisor is recommended before dropping any course. After 11:59 PM Mon Feb. 1, dropping a course results in a "W" on your transcript and you’ll be billed for tuition. After 11:59 PM Wed Mar. 24, dropping the course is possible only with a petition approved by the Dean’s office. + +Note that due to COVID-19, these requirements or dates may change. + +# Generic policies + +## CLASSROOM BEHAVIOR +Both students and faculty are responsible for maintaining an appropriate learning environment in all instructional settings, whether in person, remote or online. Those who fail to adhere to such behavioral standards may be subject to discipline. Professional courtesy and sensitivity are especially important with respect to individuals and topics dealing with race, color, national origin, sex, pregnancy, age, disability, creed, religion, sexual orientation, gender identity, gender expression, veteran status, political affiliation or political philosophy. For more information, see the policies on [classroom behavior](http://www.colorado.edu/policies/student-classroom-and-course-related-behavior) and the [Student Code of Conduct](https://www.colorado.edu/sccr/sites/default/files/attached-files/2020-2021_student_code_of_conduct_0.pdf). + +## REQUIREMENTS FOR COVID-19 +As a matter of public health and safety due to the pandemic, all members of the CU Boulder community and all visitors to campus must follow university, department and building requirements, and public health orders in place to reduce the risk of spreading infectious disease. Required safety measures at CU Boulder relevant to the classroom setting include: +- maintain 6-foot distancing when possible, +- wear a face covering in public indoor spaces and outdoors while on campus consistent with state and county health orders, +- clean local work area, +- practice hand hygiene, +- follow public health orders, and +- if sick and you live off campus, do not come onto campus (unless instructed by a CU Healthcare professional), or if you live on-campus, please alert [CU Boulder Medical Services](https://www.colorado.edu/healthcenter/coronavirus-updates/symptoms-and-what-do-if-you-feel-sick). + +Students who fail to adhere to these requirements will be asked to leave class, and students who do not leave class when asked or who refuse to comply with these requirements will be referred to [Student Conduct and Conflict Resolution](https://www.colorado.edu/sccr/). For more information, see the policies on [COVID-19 Health and Safety](https://www.colorado.edu/policies/covid-19-health-and-safety-policy) and [classroom behavior](http://www.colorado.edu/policies/student-classroom-and-course-related-behavior) and the [Student Code of Conduct](http://www.colorado.edu/osccr/). If you require accommodation because a disability prevents you from fulfilling these safety measures, please see the “Accommodation for Disabilities” statement on this syllabus. + +All students who are new to campus must complete the [COVID-19 Student Health and Expectations Course](https://www.colorado.edu/protect-our-herd/how#anchor1). Before coming to campus each day, all students are required to complete the [Buff Pass](https://pass.colorado.edu/login). + +Students who have tested positive for COVID-19, have symptoms of COVID-19, or have had close contact with someone who has tested positive for or had symptoms of COVID-19 must stay home. In this class, if you are sick or quarantined, please contact the instructor via email; you do not need to disclose the exact nature of your illness. + +[//]: # (Faculty: insert your procedure here for students to alert you about absence due to illness or quarantine. Because of FERPA student privacy laws, do not require students to state the nature of their illness when alerting you. For markdown comments, see https://stackoverflow.com/a/20885980) + +## ACCOMMODATION FOR DISABILITIES +If you qualify for accommodations because of a disability, please submit your accommodation letter from Disability Services to your faculty member in a timely manner so that your needs can be addressed. Disability Services determines accommodations based on documented disabilities in the academic environment. Information on requesting accommodations is located on the [Disability Services website](https://www.colorado.edu/disabilityservices/). Contact Disability Services at 303-492-8671 or for further assistance. If you have a temporary medical condition, see [Temporary Medical Conditions](http://www.colorado.edu/disabilityservices/students/temporary-medical-conditions) on the Disability Services website. + +## PREFERRED STUDENT NAMES AND PRONOUNS +CU Boulder recognizes that students' legal information doesn't always align with how they identify. Students may update their preferred names and pronouns via the student portal; those preferred names and pronouns are listed on instructors' class rosters. In the absence of such updates, the name that appears on the class roster is the student's legal name. + +## HONOR CODE +All students enrolled in a University of Colorado Boulder course are responsible for knowing and adhering to the Honor Code. Violations of the policy may include: plagiarism, cheating, fabrication, lying, bribery, threat, unauthorized access to academic materials, clicker fraud, submitting the same or similar work in more than one course without permission from all course instructors involved, and aiding academic dishonesty. All incidents of academic misconduct will be reported to the Honor Code (); 303-492-5550). Students found responsible for violating the academic integrity policy will be subject to nonacademic sanctions from the Honor Code as well as academic sanctions from the faculty member. Additional information regarding the Honor Code academic integrity policy can be found at the [Honor Code Office website](https://www.colorado.edu/osccr/honor-code). + +## SEXUAL MISCONDUCT, DISCRIMINATION, HARASSMENT AND/OR RELATED RETALIATION +The University of Colorado Boulder (CU Boulder) is committed to fostering an inclusive and welcoming learning, working, and living environment. CU Boulder will not tolerate acts of sexual misconduct (harassment, exploitation, and assault), intimate partner violence (dating or domestic violence), stalking, or protected-class discrimination or harassment by members of our community. Individuals who believe they have been subject to misconduct or retaliatory actions for reporting a concern should contact the Office of Institutional Equity and Compliance (OIEC) at 303-492-2127 or . Information about the OIEC, university policies, [anonymous reporting](https://cuboulder.qualtrics.com/jfe/form/SV_0PnqVK4kkIJIZnf), and the campus resources can be found on the [OIEC website](http://www.colorado.edu/institutionalequity/). + +Please know that faculty and instructors have a responsibility to inform OIEC when made aware of incidents of sexual misconduct, dating and domestic violence, stalking, discrimination, harassment and/or related retaliation, to ensure that individuals impacted receive information about options for reporting and support resources. + +## RELIGIOUS HOLIDAYS +Campus policy regarding religious observances requires that faculty make every effort to deal reasonably and fairly with all students who, because of religious obligations, have conflicts with scheduled exams, assignments or required attendance. + +In this class, if the homework is due on the date of a religious holiday, you are expected to turn the homework in **early**. If you have a religious holiday that lasts longer than one week, and so therefore you would not receive the homework with enough time to turn it in early, talk to the instructor at the beginning of the semester. For participation grades, talk to the instructor the **week before** the holiday. + +[//]: # ( In this class, {Faculty: insert your procedures here}. ) + +See the [campus policy regarding religious observances](http://www.colorado.edu/policies/observance-religious-holidays-and-absences-classes-andor-exams) for full details. diff --git a/syllabus.md b/syllabus.md new file mode 100644 index 0000000..f6b37a1 --- /dev/null +++ b/syllabus.md @@ -0,0 +1,249 @@ +# Syllabus for APPM 5630 Advanced Convex Optimization + +Spring 2021, Instructor: Stephen Becker + +This semester, the class is taught remotely some of the time due to COVID-19 + +Since there is no established optimization curriculum at CU Boulder, we will attempt to cover a lot of material in just one semester. We can divide topics into four broad categories: +- **Analysis**: Basic theory (convex analysis, optimality, duality) +- **Methods**: Standard methods, both classical and modern, and standard classifications. The idea is to give students a toolkit of standard approaches to solve actual problems. +- **Convergence**: Convergence theory (ellipsoid, Nesterov, etc.). This involves some very beautiful mathematics (and is also useful!) +- **Applications**: Often in signal processing and machine learning + +The course investigates landmark convex optimization algorithms and their complexity results. We aim to cover both key theory, such as polynomial time algorithms for linear programming and the optimality of Nesterov’s method, while also surveying current practical state-of-the-art methods. + +### Official course description +Investigates landmark convex optimization algorithms and their complexity results. Studies both theory while also surveying current practical state-of-the-art methods. Topics may include Fenchel-Rockafellar duality, KKT conditions, proximal methods, and Nesterov acceleration. + + +### Related courses at CU +This course is similar to the CS department's [CSCI 5254](http://spot.colorado.edu/~lich1539/cvxopt.html) but goes a bit farther. + +Here's a list of [optimization classes taught at CU](https://sites.google.com/colorado.edu/optimization) + + + +### Principal Topics +convex sets, convex analysis, Fenchel-Legendre conjugates, Fenchel-Rockafellar and Lagrangian duality, KKT and Slater’s conditions, conic optimization, (stochastic) gradient descent, calculating gradients, proximal methods, constrained optimization + +### Learning Goals/Outcomes +[//]: # ( Not testable; high-level ) +[//]: # ( Learning Objectives, i.e., quantifiable outcomes ) +[//]: # ( Something measurable ) +Students will learn the mathematics behind convex analysis, and why convexity is useful in optimization problems. Students will understand the setup and optimality of optimization problems, and how math can be used to provide explicit certificates of sub-optimality as well as implicit certificates via worst-case convergence rates. Students will know how to program core methods, applied to a few examples, and learn to construct programs in a modular, debuggable fashion. Students will synthesis their knowledge by producing a final project demonstrating application of optimization to real-world problems + + + +# Detailed list/schedule of Lectures/Topics + +| |Mon|Wed|Fri| +|-|-|-|-| +|Week 1: Jan 11-15| No class | No class | 1 | +|Week 2: Jan 18-22| No class | 2 | 3 | +|Week 3: Jan 25-29 | 4* | 5 | 6 (HW 1-2 due)| +|Week 4: Feb 1-5 | 7** | 8 | 9 | +|Week 5: Feb 8-12 | 10 | 11 | 12 (HW 3-4 due)| +|Week 6: Feb 15-19 | 13 | No class | 14 | +|Week 7: Feb 22-26 | 15 | 16 | 17 (HW 5-6 due)| +|Week 8: Mar 1-5 | 18 | 19 | 20 | +|Week 9: Mar 8-12 | 21 | 22 | 23 (HW 7-8 due)| +|Week 10: Mar 15-19 | 24 | 25 | 26 | +|Week 11: Mar 22-26 "spring pause" | 27 | 28*** | 29 | +|Week 12: Mar 29-Apr 2 | 30 | 31 | 32 (HW 9-10 due)| +|Week 13: Apr 5-9 | 33 | 34 | 35 | +|Week 14: Apr 12-16 | 36 | 37 | 38 (no HW, work on projects)| +|Week 15: Apr 19-23 | 39 | 40 | 41 | +|Week 16: Apr 26-30 | 42 | 43 | No class | + +Legend: * means last day to add a class (without instructor approval), ** is last day to drop a class (without a W), *** is last day to withdraw from a class (student receives a W) + +HW 1-2 is due early, but I don't want to push the due-date back farther because I want students to get feedback on a homework before they have to decide about withdrawing from the class (Feb 1, lecture #7). + +### Fall 2018 +Here's what we did in Fall 2018. Spring 2021 will be somewhat similar. + +1. Intro +2. Intro +3. Convex Sets [BV04 ch 2] +4. `cvx`, `cvxpy` tutorial +5. Convex sets, PSD cone, separating hyperplanes [BV04 ch2] +6. Convex functions [BV04 ch 3] +7. Convex functions, subgradients, commuting addition and subgradients [BC11] +8. subgradients, more on convex fcn not from [BV04], strong/strict convexity; see handout on Lipschitz bounds +9. Convex functions, sections 3.1--3.3 in [BV04] (Fenchel-Legendre conjugate functions) +10. 3.3 in [BV04], convex relaxation f^** [BC11], Fenchel-Young +11. convex relaxation f^** [BC11], [Luc10]; existence/uniqueness of minimizers [BC11 ch 11]; proximity operator [BC11, CP11] +12. prox, Moreau envelope [CW05] +13. prox, Moreau envelope [CW05]; Convex Optimization [BV04 ch 4]; gradient descent +14. convergence of gradient descent [Van16]; fire drill! +15. worst-case counter-example for 1st order methods [Nes04] +16. Convex Optimization [BV04 ch 4.2] +17. Convex Optimization, LP/QP/SOCP/SDP [BV04 ch 4.2-3, 4.6] +18. SDP/LMI [BV04 ch 4.6], Duality and Lagrangians [BV04 ch 5.1] +19. LP duality [Van08], sensible rules for remembering duals (SOB) [Ben95], Slater +20. Slater and strong duality [BV04 ch5.2], geometric interpretation [BV04 ch5.3] +21. Game theory [BV04 ch 5.2.5; 5.4.3], saddle point interpretation [BV04 ch 5.4]; some Fenchel-Rockafellar duality +22. Fenchel-Rockafellar duality [BC11, chs15,18-19] +23. Optimality and KKT conditions [BV04 ch5.5], perturbation and sensitivity [BV04 ch5.6] +24. perturbation and sensitivity [BV04 ch5.6], generalized inequalities (eg. SDP) [BV04 ch5.9] +25. Interlude on proximal [CW05][Van16] and accelerated (Nesterov) gradient methods [Nes04][Van16][BT10]; convergence rates +26. Fast/fancy algorithms for unconstrained optimization [NW05]: Nesterov, Newton, linear/non-linear conjugate gradient +27. Fast/fancy algorithms for unconstrained optimization [NW05]: quasi-Newton and L-BFGS +28. Fast/fancy algorithms for unconstrained optimization [NW05]: inexact/matrix-free Newon (Newton-CG), non-linear least-squares and Gauss-Newton, Levenberg-Marquardt, active-set methods +29. Constrained optimization [NW05]: penalty methods, augmented Lagrangian +30. derivatives; some thms from [BC11], [Ber99] +31. derivatives, automatic differentiation [NW05] +32. adjoint-state method +33. Newton's method, self-concordant [BV04 ch 9] +34. Newton's method, self-concordant [BV04 ch 9] +35. Newton for equality constraints [BV04 ch 10], interior-point methods [BV04 ch 11] +36. interior-point methods [BV04 ch 11] +37. First order methods: proximal pt, subgradient descent, conditional gradient aka Frank-Wolfe +38. First order methods: ADMM [Boyd10], Douglas-Rachford [BC17], [CP11] +39. Primal-dual method [Con13][CBS14][CCPV14][ChPo11] +40. Simplex method [NW05] +41. Complexity of LP and Simplex [Nem16], smoothed analysis, integer linear programming [Van08]; LP in fixed dimension [DMW97] +42. STUDENT PRESENTATIONS +43. STUDENT PRESENTATIONS +44. STUDENT PRESENTATIONS + +**References (books)** +- [BV04] S. Boyd and L. Vandenberghe, “Convex Optimization” (Cambridge U. Press, 2004). [Free electronic version](http://www.stanford.edu/~boyd/cvxbook/) +- [NW05] J. Nocedal and S. Wright, “Numerical Optimization” (Springer, 2005). We have free electronic access at [SpringerLink](https://link.springer.com/book/10.1007%2F978-0-387-40065-5) +- [Nem16] A. Nemirovski, Introduction to Linear Optimization, [lecture notes](http://www2.isye.gatech.edu/~nemirovs/OPTI_LectureNotes2016.pdf) +- [Van08] R. Vanderbei, “Linear Programming.” (2008) Free download at [Vanderbei's website](http://www.princeton.edu/~rvdb/LPbook/) and also via SpringerLink. +- [BC11] H. Bauschke and P. Combettes, “Convex Analysis and Monotone Operator Theory in Hilbert Spaces”, 1st ed, Springer 2011, available electronically via [SpringerLink](https://link.springer.com/book/10.1007%2F978-1-4419-9467-7) +- [BC17] H. Bauschke and P. Combettes, “Convex Analysis and Monotone Operator Theory in Hilbert Spaces”, 2nd ed, Springer 2017, available electronically via [SpringerLink](https://link.springer.com/book/10.1007/978-3-319-48311-5) +- [Ber99] D. Bertsekas, "Nonlinear Programming", 2nd ed, Athena Scientific, 1999. +- [Ber16] D. Bertsekas, "[Nonlinear Programming](http://www.athenasc.com/nonlinbook.html)", 3rd ed, Athena Scientific, 2016. +- [Nes04] Y. Nesterov, "Introductory Lectures on Convex Optimization" (2004); [Springer](http://www.springer.com/us/book/9781402075537) + - A new version is out, "[Lectures On Convex Optimization](https://www.springer.com/gp/book/9783319915777)" Nesterov 2018, but it has rearranged the chapters a bit, so you need to adjust the numbering quite a bit + +**References (papers)** +- [Boyd10] “[Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers](http://web.stanford.edu/~boyd/papers/pdf/admm_distr_stats.pdf)” by S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein (2010); +- [CP11] P. L. Combettes and J.-C. Pesquet, “[Proximal splitting methods in signal processing](http://www4.ncsu.edu/~pcombet/prox.pdf),” 2011; +- [Con13] Laurent Condat “[A primal-dual splitting method for convex optimization involving Lipschitzian, proximable and linear composite terms](https://www.gipsa-lab.grenoble-inp.fr/~laurent.condat/publis/Condat-optim-JOTA-2013.pdf)”, 2011 (J. Optim. Theory and Appl. 2013). +- [CBS14] "[Convex Optimization for Big Data: Scalable, randomized, and parallel algorithms for big data analytics](http://dx.doi.org/10.1109/MSP.2014.2329397)", Volkan Cevher, Stephen Becker, Mark Schmidt, IEEE Signal Processing Magazine, vol. 31, no. 5, 2014; +- [CCPV14] "[A forward-backward view of some primal-dual optimization methods in image recovery](https://arxiv.org/abs/1406.5439)", Patrick L. Combettes, Laurent Condat, Jean-Christophe Pesquet, Bang Cong Vu (2014) +- [ChPo11] "[A first-order primal-dual algorithm for convex problems with applications to imaging](https://hal.archives-ouvertes.fr/hal-00490826/document)", A Chambolle, T Pock (2011); +- [Luc10] "[What Shape Is Your Conjugate? A Survey of Computational Convex Analysis and Its Applications](https://doi.org/10.1137/100788458)", Yves Lucet, SIAM Review, 52(3) 2010 +- [CW05] "[Signal recovery by proximal forward-backward splitting](http://www4.ncsu.edu/~pcombet/mms1.pdf)", PL Combettes, VR Wajs - Multiscale Modeling & Simulation, 2005; +- [Van16] [EE236C class, lecture on gradient methods](http://www.seas.ucla.edu/~vandenbe/236C/lectures/gradient.pdf), L. Vandenberghe, 2016; +- [Ben95] A. Benjamin, [Sensible rules for remembering duals](https://www.math.hmc.edu/~benjamin/papers/sob.pdf). SIAM Review (37)1 1995. +- [BT10] A. Beck and M. Teboulle, "[Gradient-Based Algorithms with Applications in Signal Recovery Problems](http://www.math.tau.ac.il/~teboulle/papers/gradient_chapter.pdf)" (2010); +- [DMW97] M. Dyer, N. Megiddo, E. Welzl, "[Linear programming](http://www.inf.ethz.ch/personal/emo/PublFiles/LpSurvey03.pdf)" (chapter in Handbook of Discrete and Comp. Geometry, 1997) +- [Dual] "[Please explain intuition behind dual problem](https://math.stackexchange.com/q/223235/98783)" on math.stackexchange + + +# Resources + +We will not follow a single textbook. The following resources (mostly available online) will be the **primary references**: + +- [BV2004] S. Boyd and L. Vandenberghe, “Convex Optimization” (Cambridge U. Press, 2004). [Free electronic version](http://www.stanford.edu/~boyd/cvxbook/) This is a standard text that gives some convex analysis background and focuses on interior-point methods and engineering applications. The books is very useful, and I recommend you buy a copy. We will follow the book closely at the beginning of the semester. The appendices A and C are also quite useful if you do not have an applied math background. +- [V_course] L. Vandenberghe’s [course notes for ee236b](http://www.seas.ucla.edu/~vandenbe/ee236b/ee236b.html) follow the book, while his course notes for [ee236c](www.seas.ucla.edu/~vandenbe/ee236c.html) contain more details on proofs and advanced topics. +- [NW05] J. Nocedal and S. Wright, “Numerical Optimization” (Springer, 2005). We have free electronic +access at CU via [SpringerLink](http://link.springer.com/book/10.1007/978-0-387-40065-5). This is the go-to reference for implementing a standard method. +- [Beck17] Amir Beck, “First-Order Methods in Optimization” (SIAM, 2017). A more advanced text than +Beck’s 2014 text; free on campus at [SIAM eBooks](https://epubs.siam.org/doi/book/10.1137/1.9781611974997) +- [BC17] H. Bauschke and P. Combettes, “Convex Analysis and Monotone Operator Theory in Hilbert Spaces”, 2nd ed, Springer 2017, available electronically via [SpringerLink](https://link.springer.com/book/10.1007/978-3-319-48311-5). A nice collection of results, best as a reference and not as a step-by-step textbook. Sometimes we refer to the 2011 1st edition. + +Some supplemental resources: + +- Lijun Chen teaches a similar optimization course in the CS department, [CSCI 5254](http://spot.colorado.edu/~lich1539/cvxopt.html). That course uses the Boyd and Vandenberghe book. See also a [list of optimization courses at CU](https://sites.google.com/colorado.edu/optimization) + +- [Encyclopedia of Optimization](https://link.springer.com/referencework/10.1007/978-0-387-74759-0), 2009 Edition edited by C. A. Floudas and P. M. Pardalos is very comprehensive and has many articles on detailed topics. There are many other books that have similar names ("handbook of optimization", etc) but most of those are somewhat random in content (usually compiled after a conference). In contrast, this encylopedia really is what you hope it might be. + +- Ben Recht's [CS 726: Nonlinear Optimization I at Wisconsin 2012](http://pages.cs.wisc.edu/~brecht/cs726.html) +- Sebastian Bubeck's [ORF523: The complexities of optimization](https://blogs.princeton.edu/imabandit/orf523-the-complexities-of-optimization/) at Princeton (blog style) and his [monograph on the same topic, "Convex Optimization: Algorithms and Complexity"](https://arxiv.org/abs/1405.4980) (pdf, published in Foundations and Trends in Machine Learning, Vol. 8: No. 3-4, pp 231-357, 2015); another [link](http://research.microsoft.com/en-us/um/people/sebubeck/Bubeck15.pdf) +More advanced resources: + +- Ben Recht's [Convex Optimization and Approximation: Optimization for Modern Data Analysis EECS 227C/STAT 260 +Spring 2016](https://people.eecs.berkeley.edu/~brecht/eecs227c.html) and his [CS294 - The Mathematics of Information and Data](https://people.eecs.berkeley.edu/~brecht/cs294.html) +- [Arkadi Nemirovski](http://www2.isye.gatech.edu/~nemirovs/) has many lecture notes on various topics, including linear programming. + - His [Lectures on Modern Convex Optimization](http://www2.isye.gatech.edu/~nemirovs/Lect_ModConvOpt.pdf) with A. Ben-Tal may be the most relevant; the PDF notes are from 2013, and they have a 2001 [book from SIAM of the same title](http://epubs.siam.org/doi/book/10.1137/1.9780898718829) + - For complexity and linear programming, see his [Introduction to Linear Optimization](https://www2.isye.gatech.edu/~nemirovs/OPTILectureNotes2016.pdf) from 2016 +- John Duchi's [Introductory Lectures on Stochastic Optimization](https://stanford.edu/~jduchi/PCMIConvex/) +- Stephen Boyd’s [course notes for EE364a](http://www.stanford.edu/class/ee364a/), which follow his book. +- Amir Beck, “Introduction to Nonlinear Optimization: Theory, Algorithms, and Applications with MATLAB” (SIAM, 2014, $89). This covers classical results but by a modern researcher aware of current research +- Stephen Wright, “Primal-Dual Interior-Point Methods” (SIAM 1997), available electronically from [SIAM](http://epubs.siam.org/doi/book/10.1137/1.9781611971453). Good summary of LP theory, and more practical/thorough IPM discussion than in [BV2004] +(e.g., includes Mehotra’s predictor-corrector method), though restricted to LP of course. +- R. Tyrrell Rockafellar, “Convex Analysis”, Princeton 1970 (reprinted 1997). Terry Rockafellar wrote the first, and definitive, book on the subject. +- D. Bertsekas, A. Nedic, A.E. Ozdaglar, “Convex Analysis and Optimization” (Athena Scientific). +- D. Bertsekas, “Convex Optimization Theory” (Athena Scientific). [Bertsekas has a lot of other books with useful information as well] +- J. M. Borwein and A. S. Lewis, “[Convex Analysis and Nonlinear Optimization](http://link.springer.com/book/10.1007/978-0-387-31256-9/page/1)” (Springer). A bit more mathematical +- J.B. Hiriart-Urruty and C. Lemarechal, “Convex Analysis and Minimization Algorithms” (Springer). More on analysis, less on optimization. Technical and a useful reference. +- D. Luenberger and Y. Ye, “[Linear and Nonlinear Programming](http://dx.doi.org/10.1007/978-0-387-74503-9)” (Springer). A bit more mathematical than [BV2004], slightly different topics. +- R. Tyrrell Rockafellar and Roger J. B. Wets, "[Variational Analysis](https://link.springer.com/book/10.1007/978-3-642-02431-3)" (Springer; via SpringerLink or Rockafellar's website), goes more into detail on the analysis portion; a good mathematical reference for researchers. + +And a fun miscellaneous reference: +- Harvey Greenberg (UC Denver, emeritus), [Myths and counterexamples in mathematical programming](https://glossary.informs.org/myths/CurrentVersion/myths.pdf). Entertaining and informative + + + + +### Programming +Homeworks will involve by mathematical analysis and programming. + +Students are expected to already know how to program. We encourage using **Python** and **Matlab**; **Julia** is another good choice though we will not be using it explicitly. For homework assignments, usually the deliverable is the outcome of some code, so therefore the student may choose any reasonable programming language. However, we will be doing demonstrations in Python and/or Matlab (and the instructor is best at debugging Python and Matlab). Most of our demonstrations will be using [github](http://github.com) in conjunction with [python via colab](https://colab.research.google.com/). + +For Matlab, we'll use [CVX](http://cvxr.com/cvx) (though [yalmip](https://yalmip.github.io/) is also good); for Python, we'll use [cvxpy](http://cvxpy.org/); for Julia, try [Convex.jl](http://convexjl.readthedocs.io/). These are all good prototyping packages. + +Matlab and Python mainly rely on small third-party packages. Julia has more "official"-like packages. Most of them use the [JuMP](https://jump.dev/) framework, including big packages like [JuliaSmoothOptimizers / SolverTools.jl](https://github.com/JuliaSmoothOptimizers/SolverTools.jl) and [JuliaNLSolvers / Optim.jl](https://julianlsolvers.github.io/Optim.jl/stable/). One of the nice things about Julia is that it has very good automatic differentiation support (see a [discussion of Julia AD packages](https://discourse.julialang.org/t/state-of-automatic-differentiation-in-julia/43083/3)). Other nice things about Julia: very active and friendly community that are eager to help if you post in the forums; code executes very rapidly (like C and Fortran); high-level programming like Python and Matlab; lots of packages, and increasing quickly (way behind Python overall, but catching up in terms of specialized numerics packages); easy to call Python libraries from within Julia. + +We recommend use of an IDE or something like JupyterLab. + +We do **not** recommend using C, C++ or Fortran for programming in this class; these languages take a lot longer to prototype in, and the performance can be fast but only if you do it right (and usually requires calling the right kind of libraries, like [NAG](https://www.nag.com/content/nag-library) or GLPK or using suites like PETSc, Trilinos, [Eigen](http://eigen.tuxfamily.org/index.php?title=Main_Page), etc.) + + +# More details on topics +We don't present the lectures in the following order (lectures intermingle topics more, so that students can get started on interesting homeworks right away), but thematically, the following is what we roughly cover: + +Section 1: Background, duality, motivation +1. Intro +2. Convex Sets (ch 2 in [BV2004]) +3. Convex Functions (ch 3 in [BV2004]) +4. Convex Analysis, subdifferentials ([BauschkeCombettes]) +5. Convex Optim. Problems (ch 4 in [BV2004]) +6. Duality, optimality (ch 5 in [BV2004]) +7. Some applications, ERM and machine learning, sparse recovery, image denoising, low-rank matrix +recovery +8. CVX, CVXPY, convex.jl tutorials + +Section 2: Conic programs & relaxations +1. Linear programming + - standard LP theory, duality ([WrightIPM]) + - simplex method, complexity, smoothed analysis ([Vanderbei_LP], [Nemirovski_LP]) + - IPM ([WrightIPM], [BV2004]) + - ellipsoid method (TBD, [Nemirovski_LP]) +2. Integer Linear programming ([Vanderbei_LP]) +3. SDP ([BV2004], Pataki?) +4. QP, convex and non-convex (S-procedure, relaxations) ([Nemirovski_LP],[BN_Modern], [BV2004]) +5. MAXCUT relaxation ([Nemirovski_LP],[BN_Modern]) +6. Polynomial optimization, SOS ([Nemirovski_LP],[BN_Modern]) + +Section 3: Algorithms +1. Gradient Descent ([V_course]) +2. Accelerated and projected/proximal gradient descent, optimality ([V_course]) +3. Classical methods ([NW05], Bertsekas, [Nesterov2004]) + - Subgradient descent + - Newtons method, self-concordancy + - Non-linear conjugate gradient d) quasi-Newton + - Levenberg-Marquardt, Gauss-Newton f) Augmented Lagrangian + - Sequential Quadratic Programming (SQP) + - Step-size (Barzilai-Borwein), line search +3. Recent methods + - Douglas-Rachford and ADMM + - Primal-dual methods (e.g., Condat) + - Conditional Gradient and Frank-Wolfe d) (Randomized) Coordinate Descent + - Stochastic gradient descent + - State-of-the-art variants (SDCA, SAGA, etc.) + +Possible additional topics +1. Benders decomposition +2. Geometric Programming, linear-fractional +3. Online Convex Optimization +4. Mirror Descent +5. Bregman distances +6. Robust optimization +7. Randomized Kaczmarz method; Algebraic Reconstruction Technique (ART) for tomography +8. POCS (projection onto convex sets), best approximation, Dykstra’s algorithm