Add support for alternate encodings of bytes types

[jethrogb]

JSON does not natively support binary data. serde_json currently just supports integer arrays and a deserialization mode for raw strings. However, protocols using JSON specify their own mechanisms to handle binary data in JSON, and it's likely not to be what serde_json currently does.

How types in the serde data model are to be encoded should be specified by the (de)serializer, not the type. Suggestions such as #360 (comment) are therefore not appropriate. By specifying the encoding that way in the type, you can't (de)serialize that type anymore with a (de)serializer that does natively support bytes (such as CBOR).

This PR introduces the concept of alternate encodings for bytes types. When creating a JSON (de)serializer, you can specify the mode you want. The default mode is still integer arrays. This PR adds base64-encoding as an option. All changes can be enabled with features and should be backwards-compatible and zero-cost for existing users. The functionality is thoroughly documented with the BytesMode type.

There are a couple of TODO items but I wanted to gather feedback before continuing the work.

TODO:

• some kind of support for Value as Deserializer
• tests

cc @jseyfried

[jethrogb]

Not too happy about having this in lib.rs, but no existing module seems like a good place.

[jethrogb]

Version requirement due to https://github.com/serde-rs/json/pull/656/files#diff-b4aea3e418ccdb71239b96952d9cddb6R470

[agausmann]

How types in the serde data model are to be encoded should be specified by the (de)serializer, not the type. Suggestions such as #360 (comment) are therefore not appropriate. By specifying the encoding that way in the type, you can't (de)serialize that type anymore with a (de)serializer that does natively support bytes (such as CBOR).

I don't agree entirely; I haven't encountered a scenario where I have to implement a specific protocol and efficiently support multiple data formats simultaneously.

My focus when implementing a protocol (de)serializer is making it compatible with that specific protocol, and I am fine with creating custom Deserialize impls to deal with that. If I want to be able to (de)serialize some of that data in another format or send/receive it using another incompatible protocol, I will create another data model to match that other structure and convert between the two.

As an example, I've recently been implementing a game data parser for one of my projects. I wanted to pick out the relevant bits from a dump of the game data, and so I made a parser for that (raw.rs) which required some special logic on top of the existing Deserializer impl that I selected. However, those complex parsing rules and special helpers make it difficult to use as a general-purpose (de)serializable model, and I'd prefer a simpler model that is easier to use for the other modules. So, I implemented a more high-level data model (mod.rs) that is meant to be persistent/cached, and that doesn't use any special serde rules, just (de)serializing the data using whatever the selected format thinks is the best.

[jethrogb]

@dtolnay ping for review?

[dtolnay]

Thanks for the PR!

I would prefer not to build this into serde_json in this way, but I opened dtolnay/request-for-implementation#48 to propose how a separate library could wrap our serializer/deserializer (or any other serde format's) to expose control over the representation of the bytes type.