MARIANA: The Cutest Deep Learning Framework
As neural nets increase in complexity they also become harder to write and harder to teach. Our hypothesis is that these difficulties stem from the absence of a language that elegantly describe neural networks. Mariana (named after the deepest place on earth, the Mariana trench) is an attempt to create such a language within python. That being said, you can also call it an Extendable Python Machine Learning Framework build on top of Theano that focuses on ease of use.
It looks like this:
ls = MS.GradientDescent(lr = 0.01) cost = MC.NegativeLogLikelihood() inp = ML.Input(28*28, name = "inputLayer") h1 = ML.Hidden(300, activation = MA.ReLU(), regularizations = [ MR.L1(0.0001) ]) h2 = ML.Hidden(300, activation = MA.ReLU(), regularizations = [ MR.L1(0.0001) ]) o = ML.SoftmaxClassifier(9, learningScenario = ls, costObject = cost) MLP = inp > h1 > h2 > o
ph'nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn
- The full documentation.
- More examples and presentation material available here.
- YouTube presentation (english).
- YouTube presentation (french).
Why is it cool?
If you can draw it, you can write it.
Mariana provides an interface so simple and intuitive that writing models becomes a breeze. Networks are graphs of connected layers and that allows for the craziest deepest architectures you can think of, as well as for a super light and clean interface.
There's no need for an MLP or a Perceptron or an Auto-Encoder class, because if you know what these things are, you can turn one into the other in 2 seconds.
- Very easy to use
- Work with high level machine learning abstractions (layers, activations, regularizations, ....)
- Export you models into HTML or DOT for easy visualization and debugging
- Great for Feed Forward nets: MLPs, Auto-Encoders, Embeddings, ConvNets, Momentum, ... (check out the examples)
- Completely modular and extendable, plug in your own activations, regularizations etc...
- Trainers can be used to encapsulate your training (even oversampling, ...) in a safe environement
- Easily save your models and resume training
- Free your imagination and experiment
- No requirements concerning the format of the datasets
Note that RNNs have not yet been implemented. Mariana is a project in active development. Bugs that find their way in the codebase are regularly squashed.
First, make sure you have the latest version of Theano (do a git clone not a pip install). I keep a version that is known to work with Mariana, and that I update regularly here:
git clone https://github.com/tariqdaouda/Theano.git cd Theano python setup.py develop
But if you are not against against a little bit of adventure and want the very latest Theano stuff you can checkout Theano's repository.
Then clone Mariana from git!:
git clone https://github.com/tariqdaouda/Mariana.git cd Mariana python setup.py develop
git pull #from Mariana's folder
If you run into a problem please try to update Theano first by doing a git pull in theano's folder.
Please have a look at the examples/mnist_mlp.py. It illustrates most of what this quickstart guide adresses. There's also examples/vanilla_mnist_perceptron_mlp.py, wich demonstrate how to train an MLP (network with one hidden layer) or a Perceptron on the MNIST database without the use of a trainer. You can also check the examples for the Convolutional nets, auto-encoders, embdeddings, ...
A word about the '>'
When communicating about neural networks people often draw sets of connected layers. That's the idea behind Mariana: layers are first defined, then connected using the '>' operator.
import Mariana.activations as MA import Mariana.decorators as MD import Mariana.initializations as MI import Mariana.layers as ML import Mariana.costs as MC import Mariana.regularizations as MR import Mariana.scenari as MS
The instant MLP with dropout, L1 regularization and ReLUs
ls = MS.GradientDescent(lr = 0.01) cost = MC.NegativeLogLikelihood() inp = ML.Input(28*28, name = "inputLayer") h = ML.Hidden(300, activation = MA.ReLU(), decorators = [MD.BinomialDropout(0.2)], regularizations = [ MR.L1(0.0001) ]) o = ML.SoftmaxClassifier(9, learningScenario = ls, costObject = cost, regularizations = [ MR.L1(0.0001) ]) MLP = inp > h > o
Training, Testing and Propagating:
for i in xrange(len(train_set)) : #train the model for output 'o' function will update parameters and return the current cost print MLP.train(o, inputLayer = train_set[i : i +miniBatchSize], targets = train_set[i : i +miniBatchSize] ) for i in xrange(len(test_set)) : #the same as train but does not updated the parameters print MLP.test(o, inputLayer = test_set[i : i +miniBatchSize], targets = test_set[i : i +miniBatchSize] ) #the propagate will return the output for the output layer 'o' print MLP.propagate(o, inputLayer = test_set[i : i +miniBatchSize])
This is an autoencoder with tied weights
ls = MS.GradientDescent(lr = 0.001) cost = MC.MeanSquaredError() inp = ML.Input(10, name = "inputLayer") h = ML.Hidden(2, activation = MA.Tanh(), decorators = [ MI.GlorotTanhInit() ]) o = ML.Regression(10, activation = MA.Tanh(), costObject = cost, learningScenario = ls) ae = inp > h > o #tied weights, we need to force the initialisation of the weight first ae.init() o.W = h.W.T
Another way is to use the Autoencode layer as output:
o = ML.Autoencode(inp.name, activation = MA.Tanh(), costObject = cost, learningScenario = ls)
Can it run on GPU?
At the heart of Mariana are Theano functions, so the answer is yes. The guys behind Theano really did an awesome job of optimization, so it should be pretty fast, whether you're running on CPU or GPU. This command should run your script en GPU:
THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python <my script>.py
To make sure your machine is GPU enabled, have a look at: tests/theano_device_check1.py By default, Mariana will also tell you wether it's running on GPU or CPU and give you warnings in case you asked for the GPU and have some rogue float64s.
Making life even easier: Trainers and Recorders
A trainer takes care of the whole training process. If the process dies unexpectedly during training it will also automatically save the last version of the model as well as logs explaining what happened. The trainer can also take as argument a list of stopCriterias, and be paired with a recorder whose job is to record the training evolution. For now there is only one recorder : GGPlot2 (default recorder).
This recorder will:
- Output the training results for each epoch, highliting every time a new best score is achieved
- Automatically save the model each time a new best score is achieved
- Create and update a CSV file in a GGPlot2 friendly format that contains the entire history of the training as well as information such as runtime and hyperparameter values.
Mariana is dataset format agnostic and uses DatasetMaps to associate layers with the data the must receive, cf. examples/mnist_mlp.py for an example.
Mariana layers can take decorators as arguments that modify the layer's behaviour. Decorators can be used for example, to mask parts of the output to the next layers (ex: for dropout or denoising auto-encoders), or to specify custom weight initializations.
Costs and regularizations
Each output layers can have its own cost. Regularizations are also specified on a per-layer basis, so you can for example enforce a L1 regularization on a single layer of the model.
Saving and resuming training
Models can be saved using the save() function:
Loading is a simple unpickling:
import Mariana.network as MNET mlp = MNET.loadModel("myMLP.mariana.pkl") mlp.train(...)
Getting the outputs of intermediate layers
By setting a layer with the argument saveOutputs=True. You tell Mariana to keep the last outputs of that layer stored, so you can access them using .getLastOutputs() function.
Cloning layers and re-using layers
Mariana allows you to clone layers so you can train a model, extract one of it's layers, and use it for another model.
h2 = h.clone()
You can also transform an output layer into a hidden layer, that you can include afterwards in an other model.
h3 = o.toHidden()
And a hidden layer to an output layer using:
o = h.toOutput(ML.Regression, costObject = cost, learningScenario = ls)
To simplify debugging and communication Mariana allow to export graphical representation of networks.
The easiest way is to export it as a web page:
#to save it mlp.saveHTML("myAwesomeMLP")
But you can also ask for a DOT format representation of your network:
#to simply print it print mlp.toDOT() #to save it mlp.saveDOT("myAwesomeMLP")
You can then visualize your graph with any DOT visualizer such a graphviz.
Mariana allows you to define new types of layers, learning scenarios, costs, stop criteria, recorders and trainers by inheriting from the provided base classes. Feel free to taylor it to your needs.