Roadmap

[fchollet]

Here are the key features that we will have in Keras by the time it hits v1.0.

• Visualization tools, possibly built on top of Bokeh. You should be able to see everything that's going on in your experiment, and maybe even manage your experiments from a GUI. Total visualization is key to doing good research.
• Easy support for Spearmint for hyperparameter search.
• Complete unit tests. The recent regularizers/constraints debacle that left Kaggle users confused highlights once more that reliable unit tests should be an absolute priority. We want to be able to develop quality code safely and with confidence.
• Better convnet features, including FFT convolutions, and maybe new padding options. In the medium term we should look into incorporating support for Nervana Systems' fast convolution kernels. Let's stay state of the art ;-)
• Support for non-sequential models, by the way of:
  • [DONE] a Merge container that takes a list of Sequential models and turns them into a single output
  • a Fork container that replicates the output of a Sequential model to a list of Sequential models
    (both would be trainable end-to-end, of course).

If you see anything you would like to work on, please post here with your thoughts on how you would incorporate it into the current architecture / data structures. This is our opportunity to make a big contribution to the deep learning ecosystem! : )

[pranv]

Sounds Great!

You have provided a high level direction.
Couple of (minor) things I would add to the list:

1. Interfacing with Caffe Models - loading, saving weights.
2. Automatic Tensor Size Calculation. (even in the densely connected layer)
3. Layer wise Learning Rates for fine tuning.
4. Having a Model Zoo with various models trained on a wide number of Datasets would be great.

My thoughts on your ideas:

1. GUI is a must as you have stated. I'm currently running a basic graphing and filter visualization code with keras + matplotlib , but thats very basic and a more robust approach is needed. Nvidia's DIGITS Library would give some good inspiration. It's runs on a local server, Bokeh would give us that, but its dependencies are enormous. But, it's a really versatile library that can support all ranges of work done on keras.
2. Adding native t-SNE somewhere in the loop would be a great addition.
3. Building a better abstracted Layer Class would help both now and the future demands. Recently there was minor issue with having Dropout Layer as the 1st layer. If there is a chance to improve this, it would be now. Once the core Layer Class is fixed and shipped, changing that will break a lot of things and create a lot of hassle. So I would suggest a through debate on the core layer class, so any future layers - be it fork, merge or whatever, that can be built.

[fchollet]

Interfacing with Caffe Models - loading, saving weights.

Definitely something that would be great to have. If you have experience with Caffe, is this something you would be interested in working on?

Automatic Tensor Size Calculation. (even in the densely connected layer)

I've been thinking about something like this. It would dramatically change the API, but since it would make it simpler, it would be worth it.

Layer wise Learning Rates for fine tuning.

I don't think we'll go in that direction, because it would clash too much with the modularity of the architecture. Optimizers, of which learning rates are one part, are supposed to be independent from the model you apply them to. Introducing layer parameters that would only be used by SGD... doesn't seem right. Also adaptative optimizers like Adam, Adagrad, Adadelta, do use layer-wise (even parameter-wise) learning rates, in a way.

Having a Model Zoo with various models trained on a wide number of Datasets would be great.

Also something that would be great to have : ) Having a set of automatically downloadable datasets (currently: IMDB, MNIST, Reuters, CIFAR10) is one step in that direction. Next we should definitely have a library of pre-trained models (code to build them + HDF5 weights) that perform well on these datasets.

[johncant]

I'd like to mention adding hessian free optimizers. Keras looks awesome, but as someone who has read a bit about the state of the art in neural networks but has fairly little practical experience, the lack of a hessian-free optimizer put me off. I would definitely consider submitting a pull request.

[fchollet]

Keras looks awesome, but as someone who has read a bit about the state of the art in neural networks but has fairly little practical experience, the lack of a hessian-free optimizer put me off.

I'm quite sure hessian-free optimization is being used non-ironically by anyone in the NN community today. Turns out SGD with momentum and decay fits pretty much every use case, as long as the weights are properly initialized. HF is very 2010.

But if you want to add an HessianFree optimizer (there are several Theano implementations floating around GitHub), then go ahead : )

[pranv]

I can take up the Caffe Interfacing, but I can't guarantee any deadlines as I have a busy schedule. But, I think I can have a basic version within 3 weeks.

[fchollet]

Awesome! : ) Don't worry about deadlines, code quality is more important.

[pranv]

Better convnet features, including FFT convolutions, and maybe new padding options. In the medium term we should look into incorporating support for Nervana Systems' fast convolution kernels. Let's stay state of the art ;-)

Regarding Nervana Kernels, I started an issue with the Theano group and here's their reply:
Theano/Theano#2919

Which refers this issue, with instructions to get Nervana Kernels to work with Theano:
Theano/Theano#2941

Seeing that only Maxas class GPUs are supported, I think having another Conv/FFT backend would be better.

[tdhd]

Regarding the GUI I would recommend Tkinter which is part of the python standard library and allows matplotlib figures to be integrated into it.
It cannot be more minimalistic from external dependencies viewpoint. :)

What kind of visualizations do you have in mind? I am thinking, especially for CNNs, maps of the learned filters and occlusion heatmaps like here http://cs231n.github.io/understanding-cnn/ which are good for understanding what image regions contribute how much for the (classification) task.

Generally I think it would be useful to be able to plot the training/validation loss over the epochs. It would be neat to be able to monitor these curves as the training runs. I am not 100% sure how to do it but using observable streams from https://github.com/ReactiveX/RxPY for it would definitely be an elegant solution.

[fchollet]

Hey, thanks for the suggestions. Plotting basic quantities (loss, accuracy) over training will be the very first thing we'll add. Being able to visualize the learned weights, and being able to plot clusters of features or/and inputs would be neat as well.

I was thinking of building the UI in JS/HTML5, and make it a completely separate app (that could potentially be re-used with a different library than Keras). This seems to offer maximum flexibility and would requires no unusual dependencies either.

[tdhd]

If you do not need it right away I can take up the task of basic plotting scripts. I would write them such that they could be used after fitting a model, using the output from the fit method.

What do you think?

[fchollet]

Sure, what do you have in mind exactly?

[tdhd]

Just a simple util class which takes the output of the fit method as input. The class would then provide a method for plotting the loss/acc over the epochs and optionally saving the figure to a file. This would make matplotlib a dependency of the project though.

I will create a pull request when i have the code done. :)

[tleeuwenburg]

The advantage of a web-based UI is the ability to easily share results with
others. Tools such as Ipython notebook show the power of this approach. I
also support the modular approach of a separate library. If Keras is
capable of providing the appropriate callbacks and data structures to
support plotting requests, that sounds appropriate. That kind of modularity
would (hopefully) allow easy integration into other kinds of GUI (e.g.
standalone) for people with that use case.

On 29 May 2015 at 03:46, François Chollet notifications@github.com wrote:

Hey, thanks for the suggestions. Plotting basic quantities (loss,
accuracy) over training will be the very first thing we'll add. Being able
to visualize the learned weights, and being able to plot clusters of
features or/and inputs would be neat as well.

I was thinking of building the UI in JS/HTML5, and make it a completely
separate app (that could potentially be re-used with a different library
than Keras). This seems to offer maximum flexibility and would requires no
unusual dependencies either.

—
Reply to this email directly or view it on GitHub
#100 (comment).

---

Tennessee Leeuwenburg
http://myownhat.blogspot.com/
"Don't believe everything you think"

[tleeuwenburg]

Also, why was Bokeh chosen specifically? Seaborn is another strong alternative which I think would be worth thinking about.

[fchollet]

Also, why was Bokeh chosen specifically? Seaborn is another strong alternative which I think would be worth thinking about.

I was thinking about Bokeh because it allows back-and-forth communication with Python, but really right now it's just one option among others. Thanks for suggesting Seaborn too!

[EderSantana]

I'm also building visualization tools. Maybe you guys want to check it out. Here is an example to visualize cost function and weights being adapted with Bokeh:
https://github.com/EderSantana/agnez/blob/master/examples/bokeh_plotting.py

It was using the old API (I'll be updating all my code to the new API soon). But it may work as a starting point.

For an MLP training on the MNIST, I had visualizations like this (sorry for the bad crop):

and this live demo: http://agnez.herokuapp.com (this link may change subject or go down at one point)

I'll build this in parallel to Keras to possibly serve other frameworks as well. Obviously it will serve Keras first. The License is compatible, so if anybody wants to adapt it for this repo, I may help.

[fchollet]

@EderSantana very cool stuff!

I'll build this in parallel to Keras to possibly serve other frameworks as well.

That's also what I had in mind: there is currently no web-based visualization tool for deep learning, and the community really needs one, whether for use with Caffe, Torch, Keras... And there is no reason to make it dependent on one DL framework or another, since it will communicate with models via an API. On the Keras side, integration is dead easy via callbacks.

[datnamer]

I recommend bokeh, manage plotting, gui, callbacks all in html 5 and in one package....and yes, I think a platform independent tool would be awesome.

[pranv]

TensorFlow port is really great idea. TF needs keras like API as much as Keras would need TF when ported!

[dbonadiman]

@pranv I think that @fchollet hinted us many times that something was coming
( e.g. #666 (comment) #793 )

We need to wrap up the force a bit and start abstracting the Keras API as soon as possible.
Having TensorFlow in mind because being the first framework to support it would be awesome since it have lot's of visibility right now.

We need to start thinking about that. A discussion thread should be opened here or on the google group to discuss how it should be achieved.

[farizrahman4u]

@dbonadiman Now that we have confirmation that TF will be the new backend for Keras, why dont we start a new issue and list out the basic TODO list?

[viksit]

Incidentally, @fchollet - I remember you once linked to some WIP web UI for graphing/visualization - is that an active project?

[Kkevsterrr]

Are there any plans for MultiNEAT/HyperNEAT type support? I've loved Keras, and the ability to utilize NEAT functionality inline would be tremendously useful.

[tiangolo]

I would suggest Hyperopt instead of Spearmint since Spearmint could have commercial licensing issues: "Academic and Non-Commercial Research Use License".

Hyperopt has a more "open" license: https://github.com/hyperopt/hyperopt/blob/master/LICENSE.txt

Nevertheless, I'm not sure how it should be integrated, apart from just using Hyperopt on top of Keras (that's what I've been doing).

[claymcleod]

Anyone have an update on visualization tools??

[EderSantana]

@claymcleod I did something on a separate project: Agnez
https://github.com/AgnezIO/agnez/tree/master/examples
it can be used for visualizing Keras models. Check it out if it has something you need.

[evelynmitchell]

The playing with tensorflow visualization is under the apache license:
https://github.com/tensorflow/playground
site: http://playground.tensorflow.org/

[hashbangCoder]

I think a nifty little feature that can be added (which doesn't require much modification) could be changing learning rate when you reach patience in Early stopping callback instead of stopping completely. It could passed as an optional flag that determines whether to stop training or run schedule function and set new learning rate for optimizer
?

[marcj]

http://aetros.com/trainer is now available for everyone, so you have basically a model designer and visualisation of your Keras model, also with insights during training (like accuracy, loss, convolutional layer, confusion matrix, etc.).