Figure serialization (remove capnproto?)

[breznak]

Also in https://github.com/chhenning/nupic.core/pull/4/commits
Replace with plain stream writes?

• remove zlib (minimal, non-breaking change)
• remove capnp & replace with << bitstream backups
  • validate tests

Part of #47

[dkeeney]

The Network class serializes using Yaml. everything else uses plain binary stream writes. It is fast, non-invasive, but not platform independent. I coded up plain stream serialization for the Network class but I backed it out. The Yaml is more readable and we need Yaml for parameter processing anyway. What are your thoughts?

…

On Mon, Aug 6, 2018 at 10:52 PM, breznak ***@***.***> wrote: Also in https://github.com/chhenning/nupic.core/pull/4/commits Replace with plain stream writes? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#48>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFBa_03-ct-W5hq3vxXsWVVuXLt5VqsZks5uOSsXgaJpZM4Vxhqc> .

[breznak]

The Yaml is more readable and we need Yaml for parameter processing anyway.

Could you point me to the param handling code? Actually, I'd like to replace all the individual setXX, setXY setters/getters with a {json: "style"} format and a factory. So a config can be saved in separate file and passed to other algorithms (useful for param optimization, etc)

everything else uses plain binary stream writes. It is fast, non-invasive, but not platform independent.

I'd like to ask about your thoughts on capnp? I dislike it for it bloats so much of the codebase, on the other hand, there's already a lot of code written and it works.

[dkeeney]

The Yaml parameter processing is in src/nupic/engine/YAMLUtils.cpp/hpp. The parameters are passed to the third argument of Network::addRegion(name, nodeType, nodeParams). Often this is written in JSON format but since Yaml is a superset of JSON it can handle both formats. I looked at replacing it with simple JSON because it would be much simpler (and we could remove the last dependency) but I think we would break a lot of people's existing implementations. In the forums people talk about their setup in ModelParams.yaml. And I think it might also break OPF if we removed this. I know OPF relies on the Network API for much of its implementation so I have tried to be careful to avoid changes that affect the API. One feature I would like to get rid of is the requirement that all plugins (subclass of RegionImpl) must implement the functions getParameterFromBuffer() and setParameterFromBuffer(). These rely on the IWriteBuffer and IReadBuffer classes which are an attempt to avoid strong parameter typing. If all plugins implemented the getParameterXXX() and setParameterXXX() functions those older functions would not be needed. We can fix all of the existing C++ plugins that still use the old functions but we would still break Python plugins that still use them.

…

On Tue, Aug 7, 2018 at 10:02 AM, breznak ***@***.***> wrote: The Yaml is more readable and we need Yaml for parameter processing anyway. Could you point me to the param handling code? Actually, I'd like to replace all the individual setXX, setXY setters/getters with a {json: "style"} format and a factory. So a config can be saved in separate file and passed to other algorithms (useful for param optimization, etc) everything else uses plain binary stream writes. It is fast, non-invasive, but not platform independent. I'd like to ask about your thoughts on capnp? I dislike it for it bloats so much of the codebase, on the other hand, there's already a lot of code written and it works. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#48 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFBa_737JBn8s4ocamX9p01u9vr3JVqQks5uOcgJgaJpZM4Vxhqc> .

[breznak]

looked at replacing it with simple JSON because it would be much simpler (and we could remove the last dependency) but I think we would break a lot of people's existing implementations. In the forums people talk about their setup in ModelParams.yaml. And I think it might also break OPF if we removed this.

https://codebeautify.org/yaml-to-json-xml-csv

I'd still consider that scenario (shifting from yaml to json), what do you think about asking people for one-time automated conversion of their config?

[dkeeney]

I will take on removing capnproto. My branch will be "remove_capnproto"

[breznak]

Awesome, I'm looking forward! Please open a PR even if you have some WIP, so we can review incrementally.