New training algorithms, examples, documentation, and bug fixes

New training algorithms:

• Stochastic gradient descent
• The Adam optimizer

New examples:

Training neural nets to learn basic mathematical operations

The example/learn-*.f90 programs train neural networks to learn basic math functions. Given 8 input variables x(1), … ,x(8), the training algorithm can now learn to produce the following 6 comma-separated outputs corresponding to

• Addition: [x(1)+x(2), x(2)+x(3), x(3)+x(4), x(4)+x(5), x(5)+x(6), x(6)+x(8)]
• Multiplication: [x(1)*x(2), x(2)*x(3), x(3)*x(4), x(4)*x(5), x(5)*x(6), x(6)*x(8)]
• Exponentiation: [x(1)**2, x(2)**3, x(3)**4, x(4)**4, x(5)**3, x(6)**2]
• Power series: [1 + x(1) + (x(1)**2)/2 + (x(1)**3)/6, x(2), x(3), x(4), x(5), x(6)]

Inference-Engine's first application-relevant training example

The learn-saturated-mixing-ratio.f90 function trains a network with 2 inputs (normalized temperature and pressure), 1 output (saturated mixing ratio), and 1 hidden layer containing 72 nodes. The training inputs correspond to a uniform cartesian grid laid over the 2D space of procedure argument values bounded by the minimum and maximum values for the corresponding variables from an actual run of the ICAR regional climate model. The training data outputs are the actual result of a refactored version of the ICAR's sat_mr function, wherein a test was used to verify that the refactored code gives the same answer to all significant digits across the entire grid of input values. The inputs are normalized so that the grid covers a unit square. The outputs are unnormalized.

What's Changed

Refactoring

• Breaking change: Adopt PyTorch nomenclature: replace input/output types with new tensor_t type by @rouson in #70

Features

• Add app that reads training data from ICAR output by @rouson in #73
• Train ICAR cloud microphysics by @rouson in #74
• feat(train): make cost calculation/return optional by @rouson in #75
• feat(app): rm zero-derivative points; report cost by @rouson in #76
• feat(app): allow initial parameters to vary by @rouson in #77
• Add Adam optimizer and stochastic gradient descent by @rouson in #78
• New app features & refactoring: checkpoint/restart and user-specified training time range by @rouson in #79
• feat(app): train in strided epochs by @rouson in #81
• Reshuffle training data for each epoch by @rouson in #82
• feat(app): 1st converging cloud microphysics model by @rouson in #83
• feat(example): train ICAR saturated mixing ratio by @rouson in #90

Documentation

• Add inference_engine_t class diagram by @kareem-weaver in #71

Bug fixes

• fix(setup.sh/run-fpm.sh): add -cpp flag by @rouson in #72
• fix(app): only read opened json file if 'old' by @rouson in #80
• chore(setup.sh): rm homebrew installation of cmake by @rouson in #84

Examples and Tests

• Train linear combinations of inputs by @rouson in #88
• Training examples: Learn math operations and a function from a cloud microphysics model by @rouson in #89
• Train identity network from identity with 10% maximum random perturbations by @rouson in #86
• test(train):test near radius of non-convergence by @rouson in #87

New Contributors

• @kareem-weaver made their first contribution in #71

Full Changelog: 0.7.0...0.8.0