load DNN from memory buffer #9994

r2d3 · 2017-11-02T10:39:34Z

resolves #9765

This pullrequest changes

add an overloaded readNetFromTensorflow/readNetFromCaffe functions to read a TF/Caffe Net from data already in memory

dkurt · 2017-11-02T12:03:35Z

modules/dnn/include/opencv2/dnn/dnn.hpp

@@ -639,6 +639,11 @@ CV__DNN_EXPERIMENTAL_NS_BEGIN
      */
    CV_EXPORTS_W Net readNetFromTensorflow(const String &model, const String &config = String());

+    /** @brief Reads a network model stored in Tensorflow model in memory
+     */
+    CV_EXPORTS_W Net readNetFromTensorflowBuffer(const char *bufferModel, size_t lenModel,


Let's keep the same name for it, readNetFromTensorflow.

I think we can use brief description like at https://docs.opencv.org/master/d3/d63/classcv_1_1Mat.html#a2ec3402f7d165ca34c7fd6e8498a62ca: This is an overloaded member function... where only parameters are described.

Have you tested python / Java bindings for it?

dkurt · 2017-11-02T12:11:25Z

modules/dnn/src/tensorflow/tf_io.cpp

@@ -52,6 +52,18 @@ bool ReadProtoFromBinaryFileTF(const char* filename, Message* proto) {
    return success;
 }

+bool ReadProtoFromBinaryBufferTF(const char* data, size_t len, Message* proto) {


I think it's better to take existing two methods, remove std::ifstream creation from them and replace const char* filename argument to std::istream& is one.
In result, we'll have something like this:

bool ReadProtoFromTextFileTF(std::istream& s, Message* proto) { IstreamInputStream input(&is); return google::protobuf::TextFormat::Parse(&input, proto); }

bool ReadProtoFromTextFileTF(const char* buf, size_t len, Message* proto) { std::istringstream s(std::string(buf, len)); return ReadProtoFromTextFileTF(s, proto); }

bool ReadProtoFromTextFileTF(const char* filename, Message* proto) { std::ifstream fs(filename, std::ios::in); CHECK(fs.is_open()) << "Can't open \"" << filename << "\""; return ReadProtoFromTextFileTF(fs, proto); }

The same for binary version or just add boolean flags like text for each of them.

The problem with std::istringstream s(std::string(buf, len)) is that, in some cases, it creates a copy of the buffer into a std::string.

However ArrayInputStream input(data, len) guarantees that there is no copy
https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.io.zero_copy_stream_impl_lite#ArrayInputStream

@r2d3, we just vote to reduce a code duplication. There are two methods ReadProtoFromBinaryFileTF and ReadProtoFromBinaryBufferTF are different only at the beginning for now.

OK I will reduce code duplication

I put the common code in ReadProtoFromBinaryTF.

I simplified the code in other functions. raw_input and coded_input are now created on stack to avoid new/delete.

dkurt · 2017-11-03T10:35:40Z

modules/dnn/test/test_tf_importer.cpp

@@ -69,20 +69,61 @@ TEST(Test_TensorFlow, inception_accuracy)
    normAssert(ref, out);
 }

+static bool readFileInMemory(const string& filename, char** data, size_t* len)


@r2d3, do you mind if we return a content by std::string? In example,

static void readFileInMemory(const string& filename, std::string* content) { std::ifstream ifs(filename.c_str()); CV_Assert(ifs.is_open()); ifs.seekg(0, std::ios::end); content->resize(ifs.tellg()); ifs.seekg(0, std::ios::beg); ifs.read(&content->operator[](0), content->size()); }

No, I was thinking about it as it is how I do file loading in personal tests

dkurt · 2017-11-03T11:05:17Z

modules/dnn/test/test_tf_importer.cpp

+
+        net = readNetFromTensorflow(dataModel.c_str(), dataModel.size(),
+                                    dataConfig.c_str(), dataConfig.size());
+        ASSERT_FALSE(net.empty());


I think this check is useful for both ways of importing.

You are right, test is now done for both ways of importing

r2d3 · 2017-11-03T18:18:51Z

Hello @dkurt

As TensorFlow loading from memory is now working. I began working on Caffe one. I began moving all the ReadProtoFrom* functions to caffe_io.cpp and using them in tf_io.cpp.

r2d3 · 2017-11-03T19:50:24Z

So readNetFromCaffe is now also implemented with a corresponding test (I had to duplicate the readFileInMemory function, there is place to put these test utility functions)

I do not know if we could implement the same mechanism for DarkNet and Torch ?!

Regards

dkurt · 2017-11-04T12:02:16Z

modules/dnn/include/opencv2/dnn/dnn.hpp

+      * @param bufferModel buffer containing the content of the .caffemodel file
+      * @param lenModel length of bufferModel
+      */
+    CV_EXPORTS_W Net readNetFromCaffe(const char *bufferProto, size_t lenProto,


I had a problem with using it in python because of null characters in serialized protocol buffers. I think we should use just CV_EXPORTS.

If we use only CV_EXPORTS, the function will not be exposed to Python/Java interface.

I should look at how the binding generator works so the function accept Python str instead of string

It can't be exposed, because bindings don't work properly with native C++ pointers (at least they don't support buffers handling).

In modules/python/src2/hdr_parser.py, I find a CV_CARRAY that seems to be able to wrap arrays. But the corresponding #define has disappeared from OpenCV headers.

dkurt · 2017-11-20T07:22:07Z

modules/dnn/test/test_caffe_importer.cpp

@@ -55,6 +55,43 @@ static std::string _tf(TString filename)
    return (getOpenCVExtraDir() + "/dnn/") + filename;
 }

+static bool readFileInMemory(const string& filename, string& content)


Do you mind if we replace it to test_common.hpp to reduce duplicated code? (see test_tf_importer.cpp)

No problem. I was looking for this kind of common file and did not find it

dkurt · 2017-11-20T07:27:01Z

modules/dnn/test/test_caffe_importer.cpp

+    return true;
+}
+
+TEST(Test_Caffe, memory_read)


I think it'd be better to test an accuracy of one of the models too.

in the memory_read test case (as I have done for Test_TensorFlow) or in a separate test case ?

In example, replace TEST(Reproducibility_AlexNet, Accuracy) to

typedef TestWithParam<tuple<bool> > Reproducibility_AlexNet; TEST_P(Reproducibility_AlexNet, Accuracy) { bool readFromMemory = get<0>(GetParam()); ... } INSTANTIATE_TEST_CASE_P(Test_Caffe, Reproducibility_AlexNet, Values( Bool() ));

add a corresponding test

dkurt · 2017-11-22T18:16:43Z

👍
@r2d3, thank you! Looks good for me.

dkurt reviewed Nov 2, 2017

View reviewed changes

r2d3 force-pushed the dnn_memory_load branch 2 times, most recently from 1fec91e to 8425aec Compare November 2, 2017 17:05

dkurt reviewed Nov 3, 2017

View reviewed changes

r2d3 force-pushed the dnn_memory_load branch from 31b52e8 to d31f60f Compare November 3, 2017 17:41

r2d3 changed the title ~~[WIP] load DNN from memory buffer~~ load DNN from memory buffer Nov 3, 2017

dkurt reviewed Nov 4, 2017

View reviewed changes

r2d3 force-pushed the dnn_memory_load branch 7 times, most recently from 121e40b to 58520b6 Compare November 10, 2017 17:54

alalek assigned dkurt Nov 18, 2017

dkurt reviewed Nov 20, 2017

View reviewed changes

add loading TensorFlow/Caffe net from memory buffer

f723ced

add a corresponding test

r2d3 force-pushed the dnn_memory_load branch from a922ca6 to f723ced Compare November 20, 2017 15:29

dkurt approved these changes Nov 22, 2017

View reviewed changes

opencv-pushbot merged commit f723ced into opencv:master Nov 22, 2017

opencv-pushbot pushed a commit that referenced this pull request Nov 22, 2017

Merge pull request #9994 from r2d3:dnn_memory_load

f37f4cf

tani mentioned this pull request Mar 18, 2018

Reading Darknet from cv::FileStorage #11104

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

load DNN from memory buffer #9994

load DNN from memory buffer #9994

r2d3 commented Nov 2, 2017 •

edited

dkurt Nov 2, 2017

dkurt Nov 2, 2017 •

edited

r2d3 Nov 2, 2017

dkurt Nov 3, 2017

r2d3 Nov 3, 2017

r2d3 Nov 3, 2017

dkurt Nov 3, 2017 •

edited

r2d3 Nov 3, 2017

r2d3 Nov 3, 2017

dkurt Nov 3, 2017

r2d3 Nov 3, 2017

r2d3 commented Nov 3, 2017 •

edited

r2d3 commented Nov 3, 2017

dkurt Nov 4, 2017

r2d3 Nov 4, 2017

alalek Nov 4, 2017

r2d3 Nov 4, 2017

dkurt Nov 20, 2017

r2d3 Nov 20, 2017

r2d3 Nov 20, 2017

dkurt Nov 20, 2017

r2d3 Nov 20, 2017

dkurt Nov 20, 2017 •

edited

r2d3 Nov 20, 2017

dkurt commented Nov 22, 2017

load DNN from memory buffer #9994

load DNN from memory buffer #9994

Conversation

r2d3 commented Nov 2, 2017 • edited

This pullrequest changes

Choose a reason for hiding this comment

dkurt Nov 2, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dkurt Nov 3, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

r2d3 commented Nov 3, 2017 • edited

r2d3 commented Nov 3, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dkurt Nov 20, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dkurt commented Nov 22, 2017

r2d3 commented Nov 2, 2017 •

edited

dkurt Nov 2, 2017 •

edited

dkurt Nov 3, 2017 •

edited

r2d3 commented Nov 3, 2017 •

edited

dkurt Nov 20, 2017 •

edited