Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming binary representations #477

Closed
RaviGaddipati opened this issue Feb 28, 2017 · 5 comments
Closed

Streaming binary representations #477

RaviGaddipati opened this issue Feb 28, 2017 · 5 comments
Assignees
Labels
aspect: binary formats BSON, CBOR, MessagePack, UBJSON kind: enhancement/improvement solution: proposed fix a fix for the issue has been proposed and waits for confirmation
Milestone

Comments

@RaviGaddipati
Copy link

Currently the CBOR and MessagePack requires a vector of bytes for reading/writing. It would be beneficial to enable byte by byte consumption, like in standard file streams. This helps prevent allocating more memory than necessary (large JSON files).

Looking at the implementation this doesn't seem like it would be too difficult, some additional checking would be needed to make sure enough bytes are available.

@nlohmann
Copy link
Owner

nlohmann commented May 8, 2017

You mean a signature like

static void to_cbor(const basic_json &j, std::ostream &o)

for writing?

@RaviGaddipati
Copy link
Author

Yup. It would help if you had a multi-GB JSON file. By creating a vector and then parsing, essentially 2 copies are held in memory until parsing is complete and the vector can be destroyed.

@nlohmann
Copy link
Owner

Opened #623 as place to discuss the structure of the parsing functions.

nlohmann added a commit that referenced this issue Jul 23, 2017
- You can now pass a reference to a vector to the to_cbor and to_msgpack functions. The output will be written (appended) to the vector. #476

- You can now pass an output stream with uint8_t character type to the to_cbor and to_msgpack functions. #477

- You can now read from uint8_t */size in the to_cbor and to_msgpack functions. An input adapter will be created from this pair, so you need to use braces. #478
@nlohmann nlohmann added the solution: proposed fix a fix for the issue has been proposed and waits for confirmation label Jul 23, 2017
@nlohmann
Copy link
Owner

Function

static void to_cbor(const basic_json& j, detail::output_adapter<uint8_t> o)
{
    binary_writer(o).write_cbor(j);
}

now allows to write CBOR (and similarly, MessagePack) to the following types from which an output_adapter<uint8_t> can be constructed:

  • std::vector< uint8_t >&
  • std::basic_ostream< uint8_t >&
  • std::basic_string< uint8_t >&

Note there is no function yet to write to std::ostream(i.e., where the character type is char). This is still open.

@nlohmann nlohmann self-assigned this Jul 30, 2017
@nlohmann nlohmann added this to the Release 3.0.0 milestone Jul 30, 2017
@nlohmann
Copy link
Owner

Done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aspect: binary formats BSON, CBOR, MessagePack, UBJSON kind: enhancement/improvement solution: proposed fix a fix for the issue has been proposed and waits for confirmation
Projects
None yet
Development

No branches or pull requests

2 participants