Support for MessagePack (msgpack)

[bflad]

Description

fq looks like a great tool! Checking out the supported formats, it doesn't appear that https://msgpack.org/ is supported at the moment.

MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON.

Is this format something that would be considered for this project? It would be great to be able to inspect and filter against msgpack data.

[wader]

Hello, that should be possible. Usually fq decoders decode to something that tries to balance usability and access to many parts as possible. For self describing formats like msgpack that would deciding if all values (even scalars) should be decoded into a structs that provide access to its "encoded" length and type (fixint, bin8, etc) in addition to its "JSON" value or should it be just the "JSON" value somehow?
A compromise could be to decode into a detailed structure but then provide some helper jq function to turn the detailed structure into JSON. Usage would then be something like:

Query over the detailed structure:

# find all values encoded as fixstr
fq -d msgpack '.. | select(.type=="fixstr")' file.msgpack

Query over encoded JSON:

# find all objects with .name equal "abc"
fq -d msgpack 'msgpack_json | .. | select(.name="abc")' file.msgpack

Interestingly msgpack_json could probably be done at any level so you could query for encoding specific things like objects that has keys using fixstr encoding then turn those into json for further querying.

[tmccombs]

CBOR support would be related, since it is very similar to (and based on) msgpack.

[wader]

@tmccombs had q quick look at CBOR. Yeap some things seems similar, other things not so much. Hopefully it's possible to reuse part of the msgpack decoder at least.

[wader]

With #69 you can do:

# display detailed tree
fq -d msgpack d file.msgpack
# value version of detailed tree
fq -d msgpack tovalue file.msgpack
# represented value
fq -d msgpack msgpack_torepr file.msgpack

[wader]

Added a CBOR issue #71

[wader]

@bflad hey, in fq master you now use torepr instead of per format torepr

[bflad]

Sweet, thanks! That's really helpful. Super appreciative of the work you are doing on this project.

[wader]

👍 Glad you like it! now there is CBOR support also #75 if that is interesting