Serialize [u8] as a base64 or hex?

[dpc]

Hi,

I want to serialize sodiumoxide's public and secret keys. They do have Serialize trait implemented, but they serialize as a [u8]. I'd like them to be serialized to hex or base64 string instead.

I'm not sure how to go about it. Especially that I can't even find a Hex/Base64 encodings for serde. :(

[dtolnay]

You could use serialize_with and deserialize_with attributes to control how the public and secret keys are serialized. Something like:

#![feature(proc_macro)]

#[macro_use]
extern crate serde_derive;

extern crate base64;
extern crate serde;
extern crate serde_yaml;
extern crate sodiumoxide;

use serde::{Serializer, Deserialize, Deserializer};
use sodiumoxide::crypto::sign::{PublicKey, PUBLICKEYBYTES};

#[derive(Serialize, Deserialize)]
struct Config {
    #[serde(serialize_with = "as_base64", deserialize_with = "from_base64")]
    key: PublicKey,
}

fn as_base64<S>(key: &PublicKey, serializer: &mut S) -> Result<(), S::Error>
    where S: Serializer
{
    serializer.serialize_str(&base64::encode(&key[..]))
}

fn from_base64<D>(deserializer: &mut D) -> Result<PublicKey, D::Error>
    where D: Deserializer
{
    use serde::de::Error;
    String::deserialize(deserializer)
        .and_then(|string| base64::decode(&string).map_err(|err| Error::custom(err.to_string())))
        .map(|bytes| PublicKey::from_slice(&bytes))
        .and_then(|opt| opt.ok_or_else(|| Error::custom("failed to deserialize public key")))
}

fn main() {
    let config = Config {
        key: PublicKey::from_slice(&[1; PUBLICKEYBYTES]).unwrap(),
    };
    let yaml = serde_yaml::to_string(&config).unwrap();
    println!("{}", yaml);
    let _: Config = serde_yaml::from_str(&yaml).unwrap();
}

[dtolnay]

A reusable version of the same thing if you need it for multiple types:

fn as_base64<T, S>(key: &T, serializer: &mut S) -> Result<(), S::Error>
    where T: AsRef<[u8]>,
          S: Serializer
{
    serializer.serialize_str(&base64::encode(key.as_ref()))
}

[dpc]

Oh. Thank you so much. It looks really convenient and somehow I missed it in the documentation.

[dpc]

These examples practically did the homework for me. Thank you!

[demurgos]

This issue was really helpful. If anyone's looking for the hex version using the latest version of Serde, I managed to make it work with the following code:

use hex::{FromHex, ToHex};
use serde::{Serializer, Deserialize, Deserializer};

/// Serializes `buffer` to a lowercase hex string.
pub fn buffer_to_hex<T, S>(buffer: &T, serializer: S) -> Result<S::Ok, S::Error>
  where T: AsRef<[u8]>,
        S: Serializer
{
  serializer.serialize_str(&buffer.as_ref().to_hex())
}

/// Deserializes a lowercase hex string to a `Vec<u8>`.
pub fn hex_to_buffer<'de, D>(deserializer: D) -> Result<Vec<u8>, D::Error>
  where D: Deserializer<'de>
{
  use serde::de::Error;
  String::deserialize(deserializer)
    .and_then(|string| Vec::from_hex(&string).map_err(|err| Error::custom(err.to_string())))
}

Note: I don't know how to make it fit with Option<Vec<u8>>

[theory]

I was trying to figure out how to get serde to serialize and deserialize a Vec<Vec<u8>> as an array of Base64 (or hex). Is writing a custom control as described in the initial reply still the recommended method to do so?

[jonasbb]

@theory Would this work for you, based on the serde_with crate? base64 is not included, but is easy enough to add.

#[serde_with::serde_as]
#[derive(serde::Serialize)]
pub struct Container {
    #[serde_as(as = "Vec<serde_with::hex::Hex>")]
    pub inner: Vec<Vec<u8>>,
}
// De-/Serialize from/into this JSON: {"inner":["48656c6c6f","576f726c64"]}

There are other crates which you can use too, e.g., base64_serde. So you don't have to implement everything from scratch. They might not work on all types though, e.g., only support Vec<u8>, but you still need custom code for Option<Vec<u8>> or Vec<Vec<u8>>.

[theory]

Yeah, that looks super useful. I saw base64_serde, but wasn't able to see at a glance how it could work with a vector of Vec<u8>s. The syntax you show here for serde_with looks pretty ideal to my eye.

[theory]

Looking at the serde_with hex docs, I don't think I would have figured out how to use it on a Vec<Vec<u8>>, either. Perhaps it's a common pattern to use a type syntax in the macro, but I'm new enough to Rust not to notice it. I do see it implied by the #[serde_as(as = "Vec<DurationSeconds>")] example in the top-level docs, though, and explained in the guide, which 😍. Super happy to find that.