onnx-go-by-an-ex-ops.slide

ONNX-Go
Neural network made easy
00:00 14 May 2019

Olivier Wulveryck
Gopher
olivier.wulveryck@gmail.com
https://github.com/owulveryck/onnx-go
@owulveryck

* Agenda

- *Part* *I* - _WHY_
  - Darwinism: from Sysadmin to "ML Ops"
  - A better way to get value from Machine Learning?
- *Part* *II* - _HOW_
  - Lessons learned from operating systems and developing in Go
  - Understanding what's under the hood
- *Part* *III* - _WHAT_
  - Think different: a neural net is a Software
  - ONNX is a binary representation of this Software 
  - Go can be a runtime
  - ONNX-GO!

- `Code` , Demos, FUN!

* Part I - WHY 

.background assets/why.png

* Darwinism: the (natural) evolution of a system engineer

.background assets/darwin.png

- System engineer: The kernel Whisperer
- System administrator: The `/etc` Whisperer
- System integrator: The `application` `server` Whisperer
- DevOps: The `Jenkins` Whisperer
- SRE: The `gopher and snake` Whisperer

: Je suis un (ex) ingénieur systèmes.
: L'ingénierie système est passé de la phase: "gestion de l'OS" à une phase d'intégration de logiciels
: En effet, la maitrise du socle OS permettait de gérer au mieux la plomberie entre l'application et l'environnement d'execution

: Le SRE automatise les choses, le SRE a un role a jouer dans la conception du logiciel (SLO, cf talk)

* The future: ML-Ops?

.background assets/lamp.png

: Parler de python, Bigdata Keras, Tensorflow
: replacer ça dans le contexte de l'intégration systèmes... il faut comprendre le liant
: c'est complexe

.image assets/ml-trends.png

.caption _Machine Learning_ by [[https://trends.google.com/trends/explore?date=today%205-y&geo=US&q=machine%20learning][Google Trends]]

*  

.background assets/there_is-a-better_way.png

: There is a better way... 
: Le DevOps a montré l'importance de faire communiquer les Dev et les Ops entre eux
: Si les Ops s'intéressent à l'intégration des algorithmes de machine learning, il vont apporter
: leur expérience sur la gestion de la fiabilité et de l'exploitation à l'echelle

: C'est pour cette envie et cette passion qui animent les "travaux" que je vais vous présenter aujourd'hui..

* Part II - HOW 

.background assets/idea.png

* What did I learn by managing operations?

- Managing environments is hard and value-less;
- 90% of the costs of software are related to its maintenance [Floris and Harald, 2010];
- Running a software at scale is not only a hosting concern, but has an impact on the developement;

- Think about operations and *reliability* in the early stage of the developement.

* What did Go taugh me?

_Caution_ personal opinion here!

- static binaries make deployement easier;
- stronly typed language and the compiler are your friends; 
- A little copying is better than a big dependency
- the language is highly opinionated in a way that allows developers to stay focus instead of constantly refactoring.
- a Go program do not increase the toil when running at scale. Its self-contained static binary is one of the most DevOps friendly principles so far.

* Rise of the machines

.background assets/egg.png

Let's apply those lessons to machine learning based softwares to run it in production, at scale...

: Appying those lessons to machine learning and run it at scale (well in production is a first step)

* What's under the hood.

.background assets/tools.png

*  A brief resumé of Neural Networks

* Input of a neural network - a picture example

.play -edit utils/draw.go /func main/,/^}/ 

* Expected 0utput

  [0, 0, 0, 0, 0, 0, 0, 1, 0]

* How to transform the input into an output?

By applying mathematical operators to the input \(X\) 

$$f(X) = Y$$

* Neural networks' typical expressions

A typical layer:

$$f(X) = \sigma(W \cdot X+b)$$


This equation is what we call a *model*

\(W\) and \(B\) are the _learnable_ parameters.

The goal of a machine learning algorithm is to find _by_ _itself_ the correct values of \(W\) and \(B\) to make the equation \(f(X) = Y\) true for several *known* *values* of \(X\) and \(Y\).

- \(\cdot\) and \(+\) are linear transformation and
- \(\sigma\) is a non-linear transformation

* Equations are graphs

Let's transform this equation:
$$f(X) = \sigma(W \cdot X+b)$$

into something mode "functional":
$$f(X) = \sigma(add(mul(W,X),B))$$

* Equations are graphs

This equation can easily be represented into a graph:

.image assets/graph1.png

And depending on the phase (testing or training), it is evaluated in a way or the other.

* Conlusion about neural network

Neural networks are *Graphs* and *values*. The values are multidimentional arrays called _tensors_.

* How do you "code" neural networks?

To use a neural network, you need to encode the graph into a language...

- Caffe
- Pytorch
- Keras
- ...

and to use a backend to compute it.

- Tensorflow
- Theano
- CNTK
- ...

* But, do you really want to waste your time in the installation of...
 
.background assets/squelette.png


.image assets/TensorFlowLogo.svg

* anything and in Go?

* 

.background assets/Gorgonia_Final.png

* Part III - WHAT 

.background assets/loupe.png

* Think different: a neural net is a

.image assets/software.png

* What we want from a good software:

- *Interoperability*

Facilitate the usage of a pre-trained model. Run on any platform

- *Efficiency/Quality*

Optimize the time-to-market; from the model conception to the model runtime.

Allow the (Go) developer to use and test the neural network quickly, with the tool they already know.

- *Reliability*

Master the environment to lower the probability of failure

* About ONNX

* 

.background assets/slide8-fs8.png

* ONNX briefly

A representation of a computation graph in protocol buffers.

It is therefore independent of the tools used (Tensorflow, Keras, PyTorch, tvm...).

You need to write an Encoder/Decoder in order to read the binary files and create a structure for the targeted tools.

* ONNX

So ONNX have _DSL_ that describes a neural network in term of a computation graph.
It also describes the mathematical operators and their expected behavior.

An ONNX interpreter should understand the DSL and implement the behavior of the operators.

* Model zoo

The ONNX Model Zoo is a collection of pre-trained models for state-of-the-art models in deep learning.
There are available in the ONNX format.

.link https://github.com/onnx/models

* About onnx-go

* What is it exactly?

A package that unmarshal a binary representation of a graph into a Go structure

It allows to read the binary file (encoded by another framework/language) and recreates a Graph in a Go structure.

* onnx-go's target

* Vision statement

.code ../../README.md /For the Go/,/\./

* 

.background assets/slide9-fs8.png

* onnx-go's interface

* The Model

.link https://godoc.org/github.com/owulveryck/onnx-go

The "entrypoint" of onnx-go is the `Model`; 

It's the receiver of the computation graph (the neural network).

.code ../../decoder.go /type Model/,/}/

* Model creation

The model is a "wrapper" around an actual graph receiver that should implement the gonum's Graph
interface. This structure is itself an interface called `Backend`.

To create the Model, you need to assign it a `Backend`.

.code ../../decoder.go /func NewModel/,/{/

.link https://godoc.org/gonum.org/v1/gonum/graph

* onnx-go decoding

The `Model` is compatible with the interface encoding.BinaryUnmarshaler of Go's std library:

.code ../../decoder.go /func.*UnmarshalBinary/,/{/

So to decode a `model.onnx` file, you simply need to create a Model and call this method.

* Input and output of the Graph

Once the graph has been decoded, you need to set input and get output.

onnx-go provides those convenient methods to do that:

.code ../../io.go /.*SetInput/,/{/

.code ../../io.go /.*GetOutput/,/{/

* onnx-go's backend mechanism

* Backend of the graph: do not reinvent the wheel

 onnx-go does not handle low-level operations such as tensor products, 
 convolutions and so on itself. 
 
 Instead, it relies on a specialized, well optimized tensor manipulation library to do so, 
 serving as the "backend engine" of onnx-go. 

 Rather than picking one single tensor library
 and making the implementation of onnx-go tied to that library, 
 onnx-go handles the problem in a modular way, 
 and several different backend engines can be plugged seamlessly into onnx-go.

.link https://keras.io/backend/ This is a copy/paste of the Keras documentation with a s/keras/onnx-go/.

* The backend is simply an interface

.code ../../backend.go /type Backend/,/}/ 
.code ../../backend.go /type OperationCarrier/,/}/ 

This gives the capacity to use different execution engines (Tensorflow, Gorgonia,...)

* 

.background assets/Gorgonia_Final.png

Gorgonia is "compatible" with onnx-go expected backend through the `gorgonnx` package.

.code ../../backend/x/gorgonnx/graph.go /package/,/$/
.code ../../backend/x/gorgonnx/graph.go /type Graph/,/}/
.code ../../backend/x/gorgonnx/graph.go /func/,/}/

* 

.background assets/slide10-fs8.png

* Tests and coverage

A package `testbackend` is also part of the `onnx-go` project; 
it allows to test a backend against the functional tests of the onnx-project.

A package `testbackend` is also part of the `onnx-go` project; 
it allows to test a backend against the functional tests of the onnx-project.

.code ../../backend/x/gorgonnx/onnx_test.go /package/,/\)/

* The CODE!

* Example

* Init the model

- First, you choose and instanciate a backend;
- Then you instanciate the onnx-go top structure and associate it the backend
- And then you can unmarshal your `model.onnx` file into the backend:

.code -numbers onnx-demo.go /START_MODEL OMIT/,/END_MODEL OMIT/

*Note* this reads the model but does not execute it

* Evaluation phase

.code -numbers process.go /START_PROCESS OMIT/,/END_PROCESS OMIT/

* Demo time

.background assets/demo-time-fs8.png

* MNIST

This model predicts handwritten digits using a convolutional neural network (CNN).

*Dataset*

The model has been trained on the popular MNIST dataset.

*Source*

The model is trained in CNTK following the tutorial CNTK 103D: Convolutional Neural Network with MNIST. Note that the specific architecture used is the model with alternating convolution and max pooling layers (found under the "Solution" section at the end of the tutorial).

.link https://github.com/onnx/models/tree/master/vision/classification/mnist model from the zoo

* MNIST

.background assets/demo-fs8.png
.html demos/htdocs/mnist/index.html

* Emotion fer+

The FER+ annotations provide a set of new labels for the standard Emotion FER dataset.
In FER+, each image has been labeled by 10 crowd-sourced taggers, which provide better quality ground truth for still image emotion than the original FER labels.
Having 10 taggers for each image enables researchers to estimate an emotion probability distribution per face. This allows constructing algorithms that produce statistical distributions or multi-label outputs instead of the conventional single-label output.

.link https://arxiv.org/abs/1608.01041
.link https://github.com/onnx/models/tree/master/vision/body_analysis/emotion_ferplus

.image https://raw.githubusercontent.com/Microsoft/FERPlus/master/FER+vsFER.png

* Emotion fer+

.background assets/demo-fs8.png
.html demos/htdocs/emotion/index.html

* Conclusion

- Neural Network can now be used like any other regular library.
- Go's self contained binary makes it easy to run model at scale.
- Let the data-scientist do their job and play with data, and let the developer have fun with it!


> Get involve, nobody is a nobody
> Let's make programming with neural network fun again!