byte_string seems broken

[geiseri]

Greetings, I upgraded and moved my code to the new tags interface and I seem to have a regression. Here is my json_type_traits in case I am doing something wrong.

template <class Json>
struct jsoncons::json_type_traits<Json, std::shared_ptr<EncryptedRecord>> {
    using allocator_type = typename Json::allocator_type;

    static bool is(const Json &val) noexcept
    {
        return (val.is_object() && val.contains("id") && val.contains("data"));
    }

    static std::shared_ptr<EncryptedRecord> as(const Json &val)
    {

        SPDLOG_DEBUG("type {} tag {}", val.at("data").type(), val.at("data").tag());
        auto id    = val["id"].template as<std::string>();
        auto data  = val["data"].template as<std::vector<uint8_t>>();
        return std::make_shared<EncryptedRecord>(id, data);
    }

    static Json to_json(const std::shared_ptr<EncryptedRecord> &val, allocator_type allocator = allocator_type())
    {
        Json json(jsoncons::json_object_arg, jsoncons::semantic_tag::none, allocator);
        json["id"]    = val->id;
        json["data"]  = Json(jsoncons::byte_string_arg, val->data, jsoncons::semantic_tag::base64url, allocator);
        SPDLOG_DEBUG("type {} tag {}", json.at("data").type(), json.at("data").tag());
        return json;
    }
};

I then do the following:

auto j = jsoncons::json(r).to_string();
SPDLOG_DEBUG("Buffer1 {}", j);
auto out = jsoncons::json::parse(j).as<std::shared_ptr<EncryptedRecord>>();

The output from Buffer1 {} is correct:

{
  "data": "ny2hSd09EU0zbuyk7pb2CjfV8mC9fN4aNXQEIgDft0dfAfB-jv4-kAsXbqHf14bQr0r8vHjgLtRZiiMAtYQQk8erTWgckuQ4JNnK7tDby2IsLFgHdYdqvsimXBSlrDEJo5p5rwRwSv7CULClK2zZ63sQog1ajRpp6eORsqFnAHUr9p41M9Fgmu9XqXDaW0tFTE1i1WyF_c0oCc3Wq3JBmtrBDgCRXva-WY4GcQSiiXIVs0kWez4BmG2JIPtRFRf_ufKRsCiDHj8_rz9kpG5TGTv4",
  "id": "test",
}

The issue is when I try to parse it I get a json cannot be converted to vector. I also noticed the following on the printing the tags:

to_json:

[13:01:34][D][105340:105340][record_tests.cpp:124] type 7 tag 9

as:

[13:01:34][D][105340:105340][record_tests.cpp:108] type 6 tag 0

I assume the string is correct on the read, but shouldn't the parser be able to try to convert the string to a byte_string/vector<uint8_t>?

[danielaparker]

I don't think that's a regression. jsoncons has never supported reading a string from a JSON text into a json value j and subsequently allowing j.as<std::vector<uint8_t>>() or j.as<jsoncons::byte_string>().

Two-way byte string support is only supported for binary formats, such as CBOR or MessagePack, that accomodate byte string formats. When encoding to JSON, jsoncons encodes byte strings to JSON strings as base64url, base64 or base16, according to the associated semantic tag, defaulting to base64url if no tag is specified. When decoding to a json value, it decodes them as strings. It's left to you to retrieve it as a string and decode the base64url, base64 or base16 encoded string.

[geiseri]

I was going to say I swear I had used it before but I was only reading it in cbor. I wonder if it would be possible to leverage the semantic tags like encoding. So if you do foo.as<byte_string,semantic_tag::base64url> it would know to do the "right thing"?

[danielaparker]

I suppose you want a strategy that will work for both CBOR (where you have a byte string) and for JSON (where you have a string in an encoding scheme that jsoncons doesn't know but you do.) One option would be to support j.as<std::vector<uint8_t>>(semantic_tag::base64url), where the parameter would be considered a hint. I'd have to think about this though.

[geiseri]

I actually really like the idea of the semantic_tags in general. I think they give a very clear articulation of specialized intent, without messing up the implementation of the API with horrible if/else trees.

[danielaparker]

I've added a new member function to basic_json

template <class T>
T as(byte_string_arg_t, semantic_tag hint) const; 

which participates in overload resolution if uint8_t is convertible to T::value_type.

So you can write

    std::vector<uint8_t> u = {'H','e','l','l','o'};

    json j(byte_string_arg, u, semantic_tag::base64);

    auto bytes = j.as<std::vector<uint8_t>>();
    std::cout << "(1) ";
    for (auto b : bytes)
    {
        std::cout << (char)b;
    }
    std::cout << "\n\n";

    std::string s;
    encode_json(j, s);  // tag information is lost
    std::cout << "(2) " << s << "\n\n";

    auto sj = decode_json<json>(s); 

    // provide hint
    auto v = sj.as<std::vector<uint8_t>>(byte_string_arg,
                                         semantic_tag::base64);

    assert(v == u);

Output:

(1) Hello

(2) "SGVsbG8="

Currently on master.