Import models from caffe,torch,tensorflow

[misaghkord]

Is it possible to import models that are trained on other frameworks??
If not, is there any plans to do so??

[davisking]

There aren't any tools to do this. I'm not going to add them but I'm not adverse to a PR :)

[OranjeeGeneral]

Caffe uses the horrible google protobuffer system That's why their models are so bloated as well.
If you want to add support Caffe models you probably need to bring in all those library dependencies that come with it. This I think would destroy one of the major benefit dlib currently has it it has basically zero dependencies which makes it very easy to build and distribute. Not sure about Torch and Tensorflow what they are using. But Tensorflow is google so they might have something similar going. Also don't forget they might also add a lot of layer or loss types dlib might have no equivalent for.

[davisking]

Yes, it would be a nightmare to support importing from other tools. That's why I'm not doing it. But if someone else somehow sets it up in a way that doesn't add dependencies or any irritations to dlib then that's fine with me. I highly doubt that would happen though, for all the reasons @OranjeeGeneral points out. But if someone can surprise me then that's great :)

[sandys]

@davisking so this is something we really need. it has real world implications on speed of training - running training on a 128 machine Tensorflow Google ML-Engine GPU cluster for 1 hour costs 100$ is cheap.

OpenCV already includes the code to import these - https://github.com/opencv/opencv/blob/master/modules/dnn/src/tensorflow/tf_importer.cpp and https://github.com/opencv/opencv/blob/master/modules/dnn/src/caffe/caffe_importer.cpp

However, there is another way to do it - can you use a standard format to generate your models ? for example, I have been struggling to create the shape_predictor_68_face_landmarks.dat in the right way from tensorflow so that it would be drop-in for dlib.

[davisking]

You can run dlib or any other library on a cloud service too. So that's not really important.

If someone wants to make a caffe to dlib import tool they can look at the dlib to caffe export tool I made recently: https://github.com/davisking/dlib/tree/master/tools/convert_dlib_nets_to_caffe. You can see a lot of the notes I put in that code about the inherent incompatibilities between dlib and caffe.

There is no such thing as "a standard format" for models. It's not like being in XML or some other thing like that makes any difference. If there is some important field or property in a model not considered by the software trying to import it, then the import won't work. There is no magic piece of code that is going to be able to execute all possible models learned by machine learning.

[sandys]

hi @davisking thanks for replying. My usage of standard was a bit different - i meant platform independent. dlib serialize is very c++ dependent. It would have been nice to use something like HDF5...even protocol buffers (sorry!).

It makes working with it easier from other frameworks like pythin, lua/torch,etc. Only advantage of tensorflow/caffe models is their protocol buffers usage which makes it much simpler to work with without pulling in a lot of dependencies.

Which is why your tool is actually the other way around. There are lots of papers, etc which are now using dlib pretrained models and it is very hard to subsititute the pretrained models with any other (even if im doing some research). I will try to get the other side working ... but it is perhaps going to be a big favor for a lot of us if dlib were to use something like HDF5 or PB for serialization.

[davisking]

There is nothing platform or hardware dependent about dlib's serialization. It works on all platforms. It's also only a small amount of code. You could put the dlib serialization code on a postcard. So talking about dependencies doesn't make sense. What about the serialization in dlib do you find confusing?

[sandys]

hi @davisking - again, thanks for bothering to answer!
im generally trying to create your shape_predictor_68_face_landmarks.dat file using my own algorithm (and make it possible for dlib shape_predictor to use it). Kind of drop-in replacement.
I am doing this in Keras and Caffe - im fairly unsure of how to produce that file.

i know that the serialization routines are here -

dlib/dlib/image_processing/shape_predictor.h

Line 411 in 465e954

inline void serialize (const shape_predictor& item, std::ostream& out)

, but they use dlib serialize - http://dlib.net/dlib/serialize.h.html and im finding it very hard to replicate it.

[davisking]

Did you read the big green comment at the top of dlib/serialize.h? It tells you what the serialization format is.