GUI version of tools?

[jyegerlehner]

Hi All,
I'm interested in developing a GUI front-end that does the equivalent of train_net.bin. Mainly what I want is a graph of training error and test error vs time or iterations as training progresses (along with possibly other statistics), with the opportunity to manually pause training, and save a snapshot and history of error vs iterations in a .csv file (as well as network weights).

My question is: if I were to develop this, would you wish me to submit a pull request that includes it under "tools" in caffe, or should I just create a completely separate project that uses caffe as a library? On the one hand, "make distribution" creates a nice library project to be linked to any application such as the application I envision. On the other hand, I think I may need to modify Caffe Solver so that it accepts an observer that receives notifications of statistics as training progresses.

Thanks for any thoughts,
Jim

[jamt9000]

I was also thinking of a UI. My idea was a web-based frontend to design and
train the nets. I was playing with flask and bootstrap a bit this weekend.

On Mon, Jun 9, 2014 at 1:50 AM, leelurch notifications@github.com wrote:

Hi All,
I'm interested in developing a GUI front-end that does the equivalent of
train_net.bin. Mainly what I want is a graph of training error and test
error vs time or iterations as training progresses (along with possibly
other statistics), with the opportunity to manually pause training, and
save a snapshot and history of error vs iterations in a .csv file (as well
as network weights).

My question is: if I were to develop this, would you wish me to submit a
pull request that includes it under "tools" in caffe, or should I just
create a completely separate project that uses caffe as a library? On the
one hand, "make distribution" creates a nice library project to be linked
to any application such as the application I envision. On the other hand, I
think I may need to modify Caffe Solver so that it accepts an observer that
receives notifications of statistics as training progresses.

Thanks for any thoughts,
Jim

—
Reply to this email directly or view it on GitHub
#481.

[jyegerlehner]

jamt9000-
Maybe you're way ahead of me, and the web thingy is more valuable. I have no idea what "flask" and "bootstrap" are. I was just thinking of a local app to run on my Ubuntu machine. I was thinking of using GTK+ as the UI library. Qt is nice, but I don't like the dependency on the non-standard C++ moc compiler.

[jamt9000]

Well, mine is just an idea and I'm not sure I'd have time to work of it. But a sleek interface with visualisations like [1] would be cool. Flask and Bootstrap are just web frameworks (backend and frontend respectively). Let us see what @Yangqing et al think.

[1] http://vis.berkeley.edu/courses/cs294-10-fa13/wiki/images/f/fd/DeepVizPaper.pdf

[jyegerlehner]

Yes, that it is a slick front end that they have.

I'm thinking it would make more sense to write a daemon exposing an interface via RPC or message passing over a socket or what-have-you. Then one could develop whatever kind of GUI communicating through that interface, be it a fat local client or a web app.

The fairly minimalist thing I had imagined is expanding its scope, and not sure how much I want to bite off. We haven't seen a ground swell of enthusiasm for the idea so far.

[shelhamer]

@leelurch

if I were to develop this, would you wish me to submit a pull request that includes it under "tools" in caffe, or should I just create a completely separate project that uses caffe as a library?

An interface to training / logging / testing seems core enough to me to include in tools provided it doesn't add dependencies or is optional. Other developers may have different opinions. Of course if the scope of this grows to an all-encompassing interface for configuring, training, and running Caffe models it could deserve to graduate into its own project.

I may need to modify Caffe Solver so that it accepts an observer that receives notifications of statistics as training progresses.

I suggest starting here. An observer for solving could unify learning rates and stopping (see #76 and #190), logging, and interfaces like you're proposing. Make a PR for this then decide what you want to do further.

Regarding the interface, I'd rather have a web interface like @jamt9000 is thinking of.

• cross platform is less of a hassle
• easily separable from core Caffe and doesn't need its own compilation
• flask and bootstrap are nice frameworks, and easier to customize than a C++ GUI

[1] is indeed nice but there's no reason not to start simply then build up.

[jyegerlehner]

@shelhamer
Thanks for the guidance. I'm not familiar with git for version control, nor with github and am perplexed by the mechanics of it. Trying to learn from tutorials. Will probably be a while before I can do anything useful.

[kloudkl]

In our team meeting last week, I proposed the democratization of the mysterious and daunting "DEEP LEARNING" among the users who don't want to struggle with the complicated technical details. The delivery is a website that looks like the AWS management console or the Cloudera Manager. Ideally, the users can easily build deep learning pipelines with various algorithm options. We can provide the users with introduction videos including step-by-step guides. New users of the website would be able to get started in a few minutes using the reasonable default parameters that can run off the shelf.

Flask, Bootstrap, and jQuery are all very lightweight and very popular for quick prototyping. @sergeyk has recently created the classification demo using this popular stack. I've also used them to write simple image retrieval and annotation websites. They usually took only a few days at most.

[sergeyk]

@leelurch Web UI is definitely the way to go, and is light weight enough to live under tools/ or examples/. Any other GUI would not be accepted as core part of Caffe -- we don't want dependency on GTk, Qt, etc.

@jamt9000 Great link to deepViz! These guys are just a building over from us. I have emailed them to see if they'd like to work with Caffe. In any case, their code is available: https://github.com/bruckner/deepViz

@kloudkl Is the goal (1) to provide machine learning researchers with a nice codebase on which to build, (2) to provide industry/hobby developers with a visual recognition framework, or (3) to provide non-developers with a deep learning approach to visual recognition?

Your vision seems overkill for (1). I don't understand why (3) would matter. (2) would seem better served by a business than an open source project -- and indeed the examples you provide are for-pay services from Amazon and Cloudera.

[kloudkl]

I understand Caffe mainly serves the research community. It seems that the only thing that is necessary for the researchers is easy visualization to debug algorithms and write papers.

[sergeyk]

Response from Evan Sparks:

"I spoke with Josh about this (Dan has moved on to industry for now). As you've seen - the code is here: https://github.com/bruckner/deepViz

It's fairly self-explanatory and pretty modular. There are four basic components.

1. A web frontend - this is written in javascript with jquery and bootstrap. Josh indicates that with a little effort this could/should be rewritten in a framework like AngularJS for extensibility/maintainability. It handles the UI magic and gets its data from the web service.
2. A component that loads model snapshots in from disk using DeCAF. We used cuda-convnet to train the models offline, but used DeCAF to parse/analyze them and apply the models to new input images.
3. A program that computes statistics about the model (things like image class probabilities and some k-means clusters derived from the last layer of features) and writes these to a file.
4. A web service that allows the front-end to request various kinds of state - what does layer 2 at epoch 10, or what do the activations look like at the first layer of my fully trained model applied to the input image, etc.

If you're interested in integrating the code with Caffe, the biggest hurdle is probably going to be in swapping out where we use DeCAF objects for Caffe objects.

You're welcome to use the code however you like - if you're concerned about licensing we can certainly add one (likely Apache) - and if you have questions please feel free to ask Josh and myself - if it makes sense to sit down and walk through the code we can set that up, too."

Daniel adds:
"To add one detail to Evan's spot-on overview---there's a local branch of DeepVis that adds partial support for models trained by Caffe. I'm traveling and don't have access to the branch right now, but can push it to github next week as a reference."

---

Let's wait for Daniel's caffe-based deepviz branch to be updated, and then someone can take a look at integrating some of the DeepViz functionality into caffe under tools/.

[sergeyk]

In case anyone wants to work on this, Daniel pushed up the changes to github under the branch caffe.
It's all in a single commit: bruckner/deepViz@5b377a2
"There's a file models.py that has classes that wrap the caffe python module, so probably best to start there."

Thanks @bruckner!

[rodrigob]

I am currently considering the same issue.
I think the front-end toolkit is up to debate (on a first round I would go for matplotlib plots), but indeed an interface like convnetjs would be nice
http://cs.stanford.edu/people/karpathy/convnetjs/demo/mnist.html
(only using caffe instead of javascript)

However the key point is changing the Solver class so that Solve(...) includes a callback function as parameter (from that the python wrapper can be extended to do magic).

My current guess would be to feed the callback with a read-only view (or copy ?) of the current network (with all its layers and blobs), this would be then used to show the current filters, etc.
What else should this callback provide ? Maybe the summary of the test runs ?

I suspect doing a "generic UI" might be overkill, and in the end very "network dependent" (do no underestimate how crazy people might go regarding network architectures).
My aim would be to have a UI-enabled lenet over mnist demonstration, and exposing the necessary pieces to python so that pycaffe users can write their custom UIs (using their favorite toolkit).

[rodrigob]

Pull request #1228 would also help having an UI by giving the control loop to python.
This is the alternative to the callback strategy.

[rodrigob]

Using the branch from the push requestion #1228 I did get a first version of caffe_train.py running.
Part of the UI is generic (e.g. loss decay), and the other part is network specific.
For post-CVPR deadline, my plan is to make the network specific part a program argument that takes in a python script with a predefined function in it (e.g. "plot_network") or similar.

Ideally I will port to one of caffe's MNIST demonstration and do a push request.
Other #1228, I also did some extensions, in particular, having data and mutable_data on the python API, i.e. read-only by default, read-write only if explicity about it. Did this change after getting bit by a python referencing issue (ploting a matrix changed its value, which destroyed the learning).

[kmatzen]

I just saw this issue today. I have a branch that might be relevant. It doesn't do everything that the OP wanted though.
https://github.com/kmatzen/caffe/compare/BVLC:dev...visualization

It defines a Visualizer interface (which should actually be called something like Monitor or Logger since a separate piece does the visualization) that you register with the solver.
e.g.

solver->SetLogFilename(FLAGS_log);
solver->AddVisualizer(shared_ptr<caffe::Visualizer<float> >(
    new caffe::BasicVisualizer<float>()));

and in the solver protobuf you specify something similar to display like

visualize: 200

This means that every 200 iterations, the solver will hand the net over to the visualizer and the visualizer can create some message to be logged to the log file.

Right now I have it logging the loss, accuracy, 0, 15, 50, 85, 100 percentiles of the weights and their gradients, and a few examples of filters and their activations from each layer. There's a script called python/viz_server.py that monitors these logs and serves a website.

[shelhamer]

@kmatzen sweet display! I'd vote for bundling a monitoring tool like that if you were to PR and the details were agreeable to everyone.

[hctomkins]

I'm not entirely sure if it's of use to anyone, but I have a node interface for generating the network prototxt files. They are a bit messy when generated, and I have no real idea how to use caffe (still learning), but I could attach the files if anyone is interested? Below is a convnet and autoencoder.

[sergeyk]

@Chasvortex that looks amazing, and will definitely be useful for Caffe developers and users. What is this interface?

[hctomkins]

It's built on top of an open source 3d software tool (blender), but I have stripped down and simplified the interface to make it caffe specific. I uploaded it here if anyone wants to clone it.

[hctomkins]

Sorry - link is below
https://github.com/Chasvortex/caffe-gui-tool

[futurely]

Microsoft Azure Machine Learning Studio is an amazingly simple GUI tool. To develop an experiment, only a few clicks, drags and drops are needed.

[hctomkins]

Microsoft copied my idea!
...
Joking aside - from what I can tell it isnt at all based on caffe? I may be wrong.

[ajtulloch]

https://github.com/ajtulloch/caffegraph/ may be interesting as well to some folks.

[jmozah]

@kmatzen Do you have any plans of PR'ing your visualize branch and merging to dev?
I need a tool like that and i can also help you in the efforts.

[Geekrick88]

We now have NVIDIA announce a web based visualization framework called digits. https://github.com/NVIDIA/DIGITS

[hctomkins]

If anyone is interested, my Network creator is now updated for RC2.
https://github.com/Chasvortex/caffe-gui-tool

[bhack]

/cc @mtamburrano

[yankov]

I've got the same idea as @Chasvortex, but wanted to do it in Flask + D3 (js visualization framework) and found this issue. Anyone is working on a similar thing?

To my understanding Digits doesn't allow to visually edit the network, only to visualize your configuration.

[thatguymike]

@lukeyeager I'm sure would be happy to talk about extensions and plugins for DIGITS if you did want to integrate there.

[lukeyeager]

To my understanding Digits doesn't allow to visually edit the network, only to visualize your configuration.

That's correct.

@lukeyeager I'm sure would be happy to talk about extensions and plugins for DIGITS if you did want to integrate there.

Also correct!

[hctomkins]

If it's of any use to anyone I have simplified the installation of my node GUI quite a lot. New Wiki

[jmcclaire]

Can this be used to create a custom model for Google's Deep Dream?

[hctomkins]

Not sure if it is entirely useful posting here, but my GUI tool now manages and trains networks asynchronously inside caffe, plotting and managing losses. If it is now of any use to anyone feel free to have a look!

[achalddave]

@kmatzen Your visualization looks super useful! Did you get around to submitting a PR for this that I'm not seeing?

I'd love to be able to use this, and would be happy to help in about a month if more effort needs to be put in before merging. (Though I'm not especially familiar with Caffe, right now.)

[jyegerlehner]

I created this issue offering to develop something so I reckon I ought to give a bit of explanation. Soon after creating this it was apparent others had already developed in the direction I had intended (e.g. Nvidia's digits, see above for other examples) and more than I would be able to do. At this point we have several options available. So I'm closing this issue. If the BVLC Brewers still want an open Milestone for this I guess they'll have to create a new one.

[omidsakhi]

Hi All, I know there are some awesome projects out there for the gui of the caffe. But most are a bit heavy weight. That is why I started one with Angular. Is is still a work in progress and has a long way to go. Would be glad if you take a look and give some feedback for course correction. Thanks. Here is the link: https://github.com/omidsakhi/caffe-maker

[KeyKy]

Does caffe have a tensorboard now? Could anyone give me some reference about the math of weight and output histogram visualization? I want to visualize the learnable weights and output of layer.

[sonack]

@KeyKy You can try nvidia digits for caffe