deepy: A highly extensible deep learning framework based on Theano
deepy is a deep learning framework for designing models with complex architectures.
Many important components such as LSTM and Batch Normalization are implemented inside.
Although highly flexible, deepy maintains a clean high-level interface.
From deepy 0.2.0, you can easily design very complex computational graphs such as Neural Turing Machines.
Example codes will be added shortly.
deepy now supports training on multiple GPUs, see the following example for training neural machine translation models.
- Python 2.7 (Better on Linux)
- scipy for L-BFGS and CG optimization
Tutorials (Work in progress)
# A multi-layer model with dropout for MNIST task. from deepy import * model = NeuralClassifier(input_dim=28*28) model.stack(Dense(256, 'relu'), Dropout(0.2), Dense(256, 'relu'), Dropout(0.2), Dense(10, 'linear'), Softmax()) trainer = MomentumTrainer(model) annealer = LearningRateAnnealer(trainer) mnist = MiniBatches(MnistDataset(), batch_size=20) trainer.run(mnist, controllers=[annealer])
MNIST Handwriting task
- Simple MLP
- MLP with dropout
- MLP with PReLU and dropout
- Maxout network
- Deep convolution
- Elastic distortion
- Recurrent visual attention model
- Train a model
- Visualization the output when varying the 2-dimension latent variable
- Result of visualization
Penn Treebank benchmark
- Baseline RNNLM (Full-output layer)
- Class-based RNNLM
- LSTM based LM (Full-output layer)
Char-based language models
- Char-based LM with LSTM
- Char-based LM with Deep RNN
Deep Q learning
- Start server
pip install Flask-SocketIO python experiments/deep_qlearning/server.py
- Open this address in browser
- Recurrent NN based auto-encoder
- Recursive auto-encoder
Train with CG and L-BFGS
# Train the model python mnist_training.py # Create animation python animation.py experiments/draw/mnist1.gz
python experiments/highway_networks/mnist_baseline.py python experiments/highway_networks/mnist_highway.py
Effect of different initialization schemes
python experiments/initialization_schemes/gaussian.py python experiments/initialization_schemes/uniform.py python experiments/initialization_schemes/xavier_glorot.py python experiments/initialization_schemes/kaiming_he.py
Sorry for that deepy is not well documented currently, but the framework is designed in the spirit of simplicity and readability. This will be improved if someone requires.
Raphael Shu, 2016