Evaluate options for a language-independent checkpoint/serialization format for the CLA

[rhyolight]

The current CLA model checkpoint uses the pickle module. As we move towards multiple language support and more external model sharing, we should define a language-independent format for serializing the CLA.

The major objectives for this would be cross-language implementation (i.e. we don't have to create serialize functions separately for each language), speed, checkpoint size, and ease of development and versioning.

Status

• Protocol Buffers - The most mature. Has implementations in many languages.
• Cap'n Proto - Has C++11 and Python implementations. Has speed advantages and potentially future ability for mmap. Much less mature than Protocol Buffers but seems better designed and has active development.
• Rejected Flat Buffers - From Google, similar in spirit to Capn Proto but less mature. Probably not as good as Capn Proto as it has a pretty specific use case driving the implementation details. Also has no Python implementation.
• Rejected Some form of binary JSON (MessagePack, BJSON) - Rejected because these options don't have easy backwards compatibility.
• Rejected Thrift - Rejected. Inferior to Protocol Buffers, more difficult to use.

[iandanforth]

Thrift is a PITA both conceptually and in implementation.

--Please excuse brevity, sent from phone.--
On Oct 24, 2013 5:01 PM, "Matthew Taylor" notifications@github.com wrote:

The current CLA model checkpoint uses the pickle module. As we move
towards multiple language support and more external model sharing, we
should define a language-independent format for serializing the CLA.

The major objectives for this would be cross-language implementation (i.e.
we don't have to create serialize functions separately for each language),
speed, checkpoint size, and ease of development and versioning.

Some options include:

• Protocol buffers
• Thrift
• Cap'n proto - a little early in development but has some nice
  properties
• Some form of binary JSON

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/333
.

[scottpurdy]

@iandanforth Yes I have heard (and experienced) similar opinions. Frank mentioned the same in JIRA I believe. Something like MessagePack is easy to use but is harder to maintain and doesn't work as well across languages (as far as I can tell).

I am leaning towards protobufs or similar. I quite like Cap'n Proto and there is active development but previously is was still a bit raw. One potential advantage of it over PB is that it has planned support for memory mapping that could make deserialize-run-serialize operations very fast (this is the current Grok use case).

[scottpurdy]

Here is my branch with initial Python spatial pooler test:
https://github.com/scottpurdy/nupic/tree/sp-capnp

The files are in nupic/proto.

My initial results weren't very good. I suspect there is type conversion going on. It would probably be better to do the first tests in C++ where it is more obvious what is happening. And I wasn't doing thorough profiling, just saw that the time to create the model and the time to feed a record in were longer. From a theoretical standpoint, I am fairly confident that we can make serialization/deserialization and runtime both faster though.

[utensil]

👍

[scottpurdy]

There are currently four different options we want to measure times for: C++ and Python protocol buffers and C++ and Python Cap'n Proto buffers.

There are two ways to use these. In the simple scenario, we leave the code pretty much the same and simply create and populate the buffers when we need to serialize. The other is to actually use the buffer in memory during execution. This doesn't work for some fields like sparse matrices but it works for most everything else.

In the former case, we want to measure the times for the following operations:

• Time to create and populate the buffer
• Time to serialize the buffer to disk
• Time to deserialize the buffer into memory
• Time to create the object and copy data from the buffer into it

In the case that we use the buffer in memory during execution, we would want to measure the time it takes to run records through in addition to the times for copying/serializaing/deserialization.

Finally, when doing these timing tests it is important to run some records through the pre-serialization and post-de-serialization objects and compare the results to ensure that everything is implemented correctly (wouldn't be fair timing tests is some pieces were left out!).

[utensil]

Yeah, seen the code and it's taking the approach of "actually use the buffer in memory during execution".

[utensil]

It would probably be better to do the first tests in C++ where it is more obvious what is happening. And I wasn't doing thorough profiling, just saw that the time to create the model and the time to feed a record in were longer.

It seems that Cap’n Proto does the encoding when feeding the data, and decoding on retrieval:

Cap’n Proto gets a perfect score because there is no encoding/decoding step. The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out to disk!

So, I guess it's definitely not an option to use Cap’n Proto as internal structure of SP and such, it will slow things down.

[vsinha]

Captain Proto is currently C++11 only and I haven't been able to get it to link with the current nupic.core (or even Marek's C++11branch)

Performance with protocol buffers looks good: both the stages of creating the protocol buffer and serializing, as well as deserialization and loading variables back into the class fields were about 2 times faster than the current implementation. Right now my implementation doesn't use the protobuf object in memory throughout SP execution, but allocates and populates it when the save function is called.

[h2suzuki]

I'm wondering... what kind of API should we need for language bindings?
Should it be an IDL kind of stuff to allow generic RPC --- procedural?
or should it be a memcached or SQL kind of text/binary protocol to allow generic data access -- data oriented?

It also depends on the medium to communicate upon; dynamic linking mechanism or mere network packets. We can do binding in multiple levels. Does the efficiency matter? The speed can be optimized for throughput, latency, operations per second, ..., etc.

[rhyolight]

When I get back from vacation, I want to have a discussion about updating to C++11. That might change the implementation of this issue significantly.

[utensil]

There's #130 taking the approach of using google protocol buffer. Just wonder if anyone has noticed and evaluate FlatBuffers which is also the successor of protocol buffer and developed by google. At first glance, it also provides "Access to serialized data without parsing/unpacking" and has miscellaneous efficiency improvements just like Cap'n proto .

[rhyolight]

@scottpurdy Would you call this ticket complete?

[utensil]

rhyolight commented a day ago

@scottpurdy Would you call this ticket complete?

If it's complete, what's the conclusion?

[scottpurdy]

We don't have a decision on this yet. I think it makes sense to keep tracking here. I will create a follow up issue to track the implementation to be done after we finalize the decision.

[scottpurdy]

Now that we have a C++11 nupic.core I am going to attempt Capn Proto again.

[scottpurdy]

We have more motivation for this issue from #1231 which is a somewhat serious bug in NuPIC.

[rhyolight]

@scottpurdy Once C++11 is finished across nupic and nupic.core, the decision is to start a Cap'n Proto implementation, right?

[scottpurdy]

@rhyolight - that is my current plan, yes

[rhyolight]

Shall we close this yet?

---

Matt Taylor
OS Community Flag-Bearer
Numenta

On Fri, Oct 3, 2014 at 10:28 AM, Scott Purdy notifications@github.com
wrote:

@rhyolight https://github.com/rhyolight - that is my current plan, yes

—
Reply to this email directly or view it on GitHub
#333 (comment).

[rhyolight]

Closing, assuming we're going with Cap'n Proto in #1336.