start the repo!

README.md

@@ -1 +1,144 @@
-18S096SciML
+# 18.S096 Special Subject in Mathematics: Applications of Scientific Machine Learning
+
+## Lecturer: Dr. Christopher Rackauckas
+
+Machine learning and scientific computing have previously lived in separate
+worlds, with one focusing on training neural networks for applications like
+image processing and the other solving partial differential equations defined
+in climate models. However, a recently emerging discipline, called scientific
+machine learning or physics-informed learning, has been bucking the trend by
+integrating elements of machine learning into scientific computing workflows.
+These recent advances enhance both the toolboxes of scientific computing and
+machine learning practitioners by accelerating previous workflows and resulting
+in data-efficient learning techniques ("machine learning with small data").
+
+This course will be a project-based dive into scientific machine learning,
+directly going to the computational tools to learn how the practical aspects of
+"doing" scientific machine learning. Students will get hands-on experience
+building programs which:
+
+- Train data-efficient physics-informed neural networks
+- Accelerate scientific models using surrogate methods like neural networks
+- Solve hundred dimensional partial differential equations using recurrent neural networks
+- Solve classical machine learning problems like image classification with neural ordinary differential equations
+- Use machine learning and data-driven techniques to automatically discover physical models from data
+
+The class will culminate with a project where students apply these techniques
+to a scientific problem of their choosing. This project may be tied to one's
+on-going research interest (this is recommended!).
+
+#### Difference from 18.337
+
+Note that the difference from the recent
+[18.337: Parallel Computing and Scientific Machine Learning](https://github.com/mitmath/18337)
+is that 18.337 focuses on the mathematical and computational underpinning of how
+software frameworks train scientific machine learning algorithms. In contrast,
+this course will focus on the applications of scientific machine learning,
+looking at the current set of methodologies from the literature and learning
+how to train these against scientific data using existing software frameworks.
+Consult [18.337 Lecture 15](https://mitmath.github.io/18337/lecture15/diffeq_machine_learning)
+as a sneak preview of the problems one will get experience solving.
+
+Syllabus
+--------
+
+**Lectures**: Monday, Tuesday, Wednesday, and Thursday 1-3pm (2-139). Jan 6 - 31.
+Note that there will be no lectures between 13-16.
+
+**Office Hours**: There will be no formal office hours, however help can be found
+at the Julia Lab 32-G785 during most working hours.
+
+**Prerequisites**: While this course will be mixing ideas from machine learning
+and numerical analysis, no one in the course is expected to have covered all of
+these topics before. Understanding of calculus, linear algebra, and programming
+is essential. While Julia will be used throughout the course, prior knowledge
+of Julia is not necessary, but the ability to program is.
+
+**Textbook & Other Reading**: There is no textbook for this course or the field
+of scientific machine learning, so most of the materials will come from
+primary literature. For a more detailed mathematical treatment of the ideas
+presented in this course, consult the [18.337 course notes](https://github.com/mitmath/18337)
+
+**Grading**: The course is based around the individual projects. 33% of the grade
+is based on the writeup due on January 14th, 34% of the grade is based on the
+project writeup, and 33% is based on the project presentation.
+
+**Collaboration policy**: Make an effort to solve the problem on your own before
+discussing with any classmates. When collaborating, write up the solution on
+your own and acknowledge your collaborators.
+
+## Individual Project
+
+The goal of this course is to help the student get familiar with scientific
+machine learning in a way that could help their future research activities. Thus
+this course is based around the development of an individual project in the area
+of scientific machine learning.
+
+Potential projects along these lines could be:
+
+- Recreation of a parameter sensitivity study in a field like biology,
+  pharmacology, or climate science
+- [Augmented Neural Ordinary Differential Equations](https://arxiv.org/abs/1904.01681)
+- [Neural Jump Stochastic Differential Equations](https://arxiv.org/pdf/1905.10403.pdf)
+- Acceleration methods for adjoints of differential equations
+- Improved methods for Physics-Informed Neural Networks
+- New applications of neural differential equations, such as optimal control
+- Parallelized implicit ODE solvers for large ODE systems
+- GPU-parallelized ODE/SDE solvers for small systems
+
+or anything else appropriate. High performance implementations of library code
+related to scientific machine learning is also an appropriate topic, so new
+differential equation solvers or GPU kernels for accelerated mapreduce is a
+viable topic.
+
+The project is split into 3 concrete steps.
+
+### Part 1: Individual Project Proposal. Due January 12th
+
+The project proposal is a 2 page introduction to a potential project. The proposal
+should go over the background of the topic and the proposed methodology to
+investigate the subject. It should include at least 3 references and discuss
+alternative methods that could be used, and what the pros and cons would be.
+
+**You should be discuss the potential project with the instructor before submission!**
+
+### Part 2: Project Presentation. TBA
+
+Depending on the size of the course, everyone will be expected to give a 20
+minute presentation on their project, introducing the class to their topic and
+their results. This will take place during the last week of the course.
+
+### Part 3: Project Report. Due February 4th
+
+The final project is a 8-20 page paper using the style
+template from the [_SIAM Journal on Numerical Analysis_](http://www.siam.org/journals/auth-info.php)
+(or similar). The final project must include the code for the analysis as a
+reproducible Julia project with an appropriate Manifest. Model your paper on
+academic review articles (e.g. read _SIAM Review_ and similar journals for
+examples).
+
+By default these projects will be shared on the course website. You may choose
+to opt out if necessary. Additionally, projects focusing on novel research may
+consider submission to Arxiv.
+
+## Schedule of Topics
+
+A brief overview of the topics is as follows:
+
+- Introduction to Scientific Machine Learning
+- Machine Learning for SciML: how to think about, choose, and train universal approximators
+- Introduction to Julia: package/project development and writing efficient code
+- Applied numerical differential equations for scientific modeling
+- Estimating model parameters from data
+- Training neural networks to solve differential equations
+- Event handling and complex phenomena in neural-embedded models
+- Physics-Informed Neural Networks (PINNs)
+- Neural differential equations and universal differential equations
+- Acceleration of partial differential equations with neural network approaches
+- Choosing neural networks, optimizers, learning rates, and diagnosing fitting
+  issues
+- Turn neural networks back into equations with SInDy
+- Tweaking differential equation solvers to accelerate fitting
+- Choices for gradient calculations: differences and understanding the appropriate
+  methods to use
+- Writing good code for GPU acceleration

lecture2/ml.jmd

@@ -0,0 +1,75 @@
+---
+title: Introduction to Julia for Scientific Machine Learning
+author: Chris Rackauckas
+date: January 6th, 2020
+---
+
+Let's start by discussing how to use Julia for machine learning from the
+context of scientific machine learning. The core of machine learning is the
+Universal Approximation Theroem (UAT) which states that any sufficiently nice
+function can be approximated by a sufficiently large nueral network. Since this
+is what we will be looking at in practice, let's get started with training
+neural networks to match functions instead of data.
+
+## Getting Started with Julia for Machine Learning
+
+To get started with Julia for machine learning, first we will need a Julia
+installation. I would recommend going to [https://julialang.org/downloads/](https://julialang.org/downloads/)
+and downloading the latest release. The generic Julia binaries use a patched
+version of the LLVM compiler which ensures all of the mathematical operations
+are correct. **Not all Linux distributions utilize the patched LLVM, so be careful
+if you are not using these binaries!**. The installation instructions describe
+how to download and set the path to the binaries, once that is done, the `julia`
+command should just work.
+
+I would next recommend getting an IDE up and running. For this I recommend the
+[Juno](https://junolab.org/) environment, whose installation instructions can
+be found at [http://docs.junolab.org/latest/man/installation/](http://docs.junolab.org/latest/man/installation/).
+An alternative is VSCode, where the [Julia VSCode plugin can be found at https://github.com/julia-vscode/julia-vscode](https://github.com/julia-vscode/julia-vscode).
+
+Once those are up and running, if you have access to a GPU you will likely want
+to download and install [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads).
+Just grab the latest version if you're using a newer graphics card (or check the
+compatibility of your graphics card). For using convolutional neural networks,
+we will want cudnn, which you install [from the cudnn website](https://developer.nvidia.com/cudnn).
+Note that part of the installation requires overwriting files from the CUDA
+installation: it looks a little hairy but that is the way to do it!
+
+Now that your installation is up and running, you'll want to grab a few Julia
+packages. For a listing of Julia packages, consult [pkg.julialang.org](https://pkg.julialang.org/docs/).
+For this course we will mainly be making use of:
+
+- DifferentialEquations.jl
+- Flux.jl
+- Zygote.jl
+- DiffEqFlux.jl
+- NeuralNetDiffEq.jl
+
+Those many others, such as ForwardDiff.jl or FiniteDiff.jl, will make an appearance.
+To add a package in Julia we will make use of the package REPL. To enter the
+package REPL mode, hit the `]` key. For help in the package REPL mode you can
+use `?` which will list commands. From this we can see there is an `add` command
+to add packages, so for example to add Flux.jl we would do:
+
+```julia;eval=false
+]add Flux
+```
+
+To then use the package we will then use the `using` command:
+
+```julia
+using Flux
+```
+
+If you prefer to namespace all commands (like is normally done in Python, i.e.
+`Flux.gradient` instead of `gradient`), you can use the command:
+
+```julia;eval=false
+import Flux
+```
+
+Note that the installation and precompilation of these packages will occur at
+the `add` and first `using` phases, so they may take awhile (subsequent uses
+will utilize the precompiled form and take a lot less time!)
+
+## Using Projects and Manifests