`std::optional` for cbor

[dcorbeil]

The cbor implementation doesn't seem to be handling std::optional as documented (for json at least). Consider the following struct example:

struct optional_test_t {
    int i;
    std::optional<int> opt_i;
};

Reading

const optional_test_t test =
    optional_test_t{
        .i = 42
    };

std::vector<uint8_t> cbor_bytes = {
    0xA1,      // map(1)
    0x61,      // text(1)
        0x69,  // "i"
    0x18, 0x2A // unsigned(42)
};
const std::string json_string = R"({"i":42})";

auto test_from_json_res = rfl::json::read<optional_test_t>(json_string);
auto test_from_cbor_res = rfl::cbor::read<optional_test_t>((char*)cbor_bytes.data(), cbor_bytes.size());

optional_test_t test_from_json = test_from_json_res.value(); // OK
optional_test_t test_from_cbor = test_from_cbor_res.value(); // Field named 'opt_i' not found.

Writing

const optional_test_t test =
    optional_test_t{
        .i = 42
    };

const std::vector<char> expected_cbor = {
    (char)0xA1,            // map(1)
    (char)0x61,            // text(1)
        (char)0x69,        // "i"
    (char)0x18, (char)0x2A // unsigned(42)
};
const std::string expected_json = R"({"i":42})";

const std::string json = rfl::json::write(test);
const std::vector<char> cbor = rfl::cbor::write(test);

assert(expected_json == json); // OK
assert(expected_cbor == cbor); // Fail. The written cbor bytes translates to "{"i": 42, "opt_i": null}"

Just like for rfl::json, I expected rfl::cbor::read() to successfully parse a buffer of bytes that doesn't contain std::optional fields leaving the missing optionals as std::nullopt. I also expected rfl::cbor::write() to not emit std::optional fields in the buffer of bytes.

Long story short, would it be possible for rfl::cbor to behave like rfl::json when it comes to std::optional?

Thanks!

[liuzicheng1987]

You are right...the reason this is happening is because msgpack and cbor require you to set the size of the object in advance, which makes it difficult, but not impossible to do this.

I think I could change it, and at the very least the behaviour should be documented.

[dcorbeil]

[...] because msgpack and cbor require you to set the size of the object in advance

Is this related to the expectation that maps are of finite length? This issue might not be the right place to talk about this but I was curious as to what the reason is behind that decision.

I think I could change it

That would be awesome if you did :)

[bryceschober]

msgpack and cbor require you to set the size of the object in advance

@liuzicheng1987 I can't speak to msgpack, but CBOR's arrays and objects ("maps" are "major type 5" in CBOR) have both explicit-length and indefinite-length variants.

I'm not that familiar with the jsoncons library, but I know that it supports encoding & decoding indefinite-length CBOR objects, because their first example and basic_cbor_encoder docs include them.

So it's easy to see that it's possible, but I'm not sure what the implications would be for your reflection-based implementation. For arrays, it seems like it would be relatively straightforward to deserialize into a std::vector without knowing the size up-front. For a CBOR map to map to a struct, it seems that it would be harder. Perhaps your reflection support can already discern whether a struct has any optional members, in which case it could just always encode with an indefinite length and not have to "look ahead" to count how many optional members are valid. On the decoding side, it would similarly need to reflect whether the struct has any optional members and accept either explicit-length or indefinite-length objects.

[kpwelsh]

Just stopping by to say that I am also affected by this. The library is super slick, but having to redefine my structs around this is very invasive.