Does Cereal support streamed JSON deserialization?

[dkorolev]

Hello,

Is it possible to have Cereal output each input object as it has been parsed from the stream?

I have a code snippet that looks like this:

class X {
  public:
    X() = default;
    std::string s;

  private:
    friend class cereal::access;
    template<class A> void serialize(A& ar) {
        ar(CEREAL_NVP(s));
    }
};
// ...
{
    cereal::JSONInputArchive ar(std::cin);
    X x;
    while (true) {
        ar(x);
        std::cout << x.s << std::endl;
    }
}

(Full code: https://github.com/dkorolev/sandbox_cereal/blob/233396ef12dcb7feda88aa3e0efe6d8623353da3/standalone.cc)

This code snippet can parse a single piece of input:

echo '{"value0":{"s":"test"}}' | ./build/standalone

It can parse multiple inputs as well:

echo '{"value0":{"s":"this"},"value1":{"s":"test"},"value2":{"s":"passes"}}' | ./build/standalone

However:

1. All the output happens after all the input is processed. I have been unable to have it process streaming data.

In other words, this does not output this test before the whole code runs:

(echo '{"value0":{"s":"this"},"value1":{"s":"test"},' ; sleep 5 ; echo '"value2":{"s":"passes"}}') | ./build/standalone

2. The code crashes with the exception at the end.

terminate called after throwing an instance of 'cereal::RapidJSONException' what(): rapidjson internal assertion failure: IsObject() Aborted (core dumped)

I am using Cereal 1.0.0.

Any suggestions?

Thanks,
Dima

[randvoorhies]

Streaming is currently unsupported, unfortunately. This is because the JSON parser we are using does not support it. There is currently two open issues for this (#55, and #89), so hopefully we will find the time to address this soon.

[pah]

FYI, on its master branch, RapidJSON now supports streaming by using the kParseStopWhenDoneFlag parsing flag. With this, the parsing stops consuming the stream and returns to the caller. See upstream issues Tencent/rapidjson#75 and Tencent/rapidjson#83.

[AzothAmmo]

Great to hear about RapidJSON. It will save us a lot of hassle to be able to stick with it when we move to support streaming.