deser: an experimental serialization and deserialization library for Rust

Deser is a serialization library for Rust for self describing formats such as JSON, YAML, TOML, INI, CBOR, MessagePack, XML, property lists, PHP's serialize, Python's pickle, CSV and query strings. It takes the user experience of serde, the problems that years of running serde in production turned up and the Rust of today, and tries to solve them with a different architecture. If you know serde you will feel at home: you derive Serialize and Deserialize, pick a format crate, and most attributes have the names you already know.

use deser::{Serialize, Deserialize};

#[derive(Debug, Serialize, Deserialize)]
#[deser(rename_all = "camelCase")]
pub struct Account {
    id: u64,
    account_holder: String,
    #[deser(default)]
    is_deactivated: bool,
}

let json = r#"{"id": 42, "accountHolder": "Jane"}"#;
let account: Account = deser_json::from_str(json).unwrap();
assert_eq!(account.account_holder, "Jane");
assert_eq!(
    deser_json::to_string(&account).unwrap(),
    r#"{"id":42,"accountHolder":"Jane","isDeactivated":false}"#
);

Deriving requires the derive feature, which is not enabled by default:

cargo add deser --features derive
cargo add deser-json

The same type works unchanged with every format (CSV as long as it's flat).

• Core: deser (the crate you depend on), deser-derive (the derive feature) and deser-core (internal, what format crates depend on)
• Formats: deser-json, deser-jsonc, deser-json5, deser-hj, deser-yaml, deser-toml, deser-ini (INI and git's config files), deser-cbor, deser-msgpack, deser-xml, deser-plist (XML, binary and OpenStep), deser-php (PHP's serialize), deser-pickle (Python's pickle), deser-csv (CSV and TSV), deser-urlencoded (query strings and forms), deser-env (environment variables), deser-debug (debug formatting)
• Layers and adapters: deser-path (paths in errors), deser-location (line and column of values), deser-validate (validation), deser-encoding (hex and base32)
• Integrations: deser-value (dynamic values), deser-transcode (converting between formats), deser-tokio (async IO), deser-serde (serde interop)

Serde is one of the most important crates in the Rust ecosystem and its stability is a big part of why. That same stability also means that some of its problems cannot be fixed without breaking every format and every hand written implementation. Deser is an experiment to see what a serialization system looks like that is allowed to start over:

• Buffering does not lose information. Internally tagged and untagged enums record values as events together with everything the format knows about them, so u128, exact numbers, numeric keys and error locations survive.
• Flattening is native and does not buffer at all, also with deny_unknown_fields.
• No stack overflows. Deser does not recurse on the call stack, so deep nesting needs no recursion limit.
• Enums are more capable. Catch-all variants can hold data and round trip, a default variant can be picked if the tag is missing and tags can be integers and booleans.
• Customizations compose. Adapters nest (as = Option<Vec<DisplayFromStr>>) and validation is an adapter too.
• Attributes are Rust, not strings. Defaults, names, adapters and bounds are expressions and types the compiler checks.
• Ready for async. An ongoing deserialization is Send and can be fed while the input arrives.
• Errors point at the problem with line, column and the path to the value, also inside buffered values. Their category tells malformed input apart from input that does not fit your types (HTTP 400 vs. 422).
• Layers sit between the format and your types and can track paths, rename keys or reject input.
• Safe defaults for untrusted input: duplicate keys are an error by default, limits for depth, size and length are configured once in a context.

Many of these came up while building Sentry Relay, which processes untrusted JSON at scale. SERDE.md goes through them in detail and links the serde issues they correspond to.

A single enum that shows a few things that would need hand written code or extra crates with serde:

use std::net::IpAddr;
use deser::{Deserialize, Serialize};
use deser::adapters::DisplayFromStr;
use deser::de::Recording;
use deser_validate::{Check, validator};

mod keys {
    pub const KIND: &str = "@type";
}

validator!(NonZero(port: &u16) => *port != 0, "must not be zero");

#[derive(Debug, Serialize, Deserialize)]
#[deser(tag = keys::KIND, tag_alias = "type", rename_all = "snake_case")]
pub enum Listener {
    // picked if the tag is missing
    #[deser(default)]
    Tcp {
        #[deser(as = DisplayFromStr)]
        host: IpAddr,
        #[deser(default = 8080, as = Check<NonZero>)]
        port: u16,
    },
    Unix { path: String },
    // kinds this version does not know are kept and written back
    #[deser(other)]
    Other(#[deser(tag)] String, Recording),
}

let listener: Listener =
    deser_json::from_str(r#"{"host": "127.0.0.1"}"#).unwrap();
assert!(matches!(listener, Listener::Tcp { port: 8080, .. }));

let input = r#"{"@type":"quic","host":"::1","alpn":["h3"]}"#;
let listener: Listener = deser_json::from_str(input).unwrap();
assert_eq!(deser_json::to_string(&listener).unwrap(), input);

let input = r#"{"host": "::1", "port": 0}"#;
let err = deser_json::from_str::<Listener>(input).unwrap_err();
assert_eq!(
    err.to_string(),
    "InvalidValue: invalid value: must not be zero at line 1 column 25"
);

Here the tag of an internally tagged enum comes last, so the values have to be buffered until it is known. In serde this is where locations and paths get lost, in deser they are retained:

use deser::Deserialize;
use deser_path::{Path, PathLayer};

#[derive(Debug, Deserialize)]
struct Config {
    servers: Vec<Server>,
}

#[derive(Debug, Deserialize)]
#[deser(tag = "type", rename_all = "lowercase")]
enum Server {
    Http { url: String, timeout: u32 },
    File { path: String },
}

let toml = r#"
[[servers]]
type = "file"
path = "/srv/www"

[[servers]]
url = "https://example.com/"
timeout = "30s"
type = "http"
"#;

let err = deser_toml::Deserializer::from_str(toml)
    .deserialize_with::<Config, _>(|driver| {
        driver.push_layer(PathLayer::new())
    })
    .unwrap_err();
let path = err.attachment::<Path>().unwrap();
assert_eq!(path.to_string(), "servers[1].timeout");
assert_eq!((err.line(), err.column()), (Some(8), Some(11)));

More practical examples are in the examples folder.

Instead of visitors calling into each other recursively, deserializing a type creates a sink that receives events and serializing it produces emitters that hand out nested values. Nested sinks and emitters are returned to a driver which keeps them in an arena instead of on the call stack, which is why nesting cannot overflow the stack and a deserialization can be suspended between events. The data model is small (atoms, maps and sequences) and can be extended with extension atoms that carry a fallback for formats which do not understand them. Where buffering cannot be avoided, events are recorded together with the state the format published for them and replayed as such.

Deser only supports self describing formats, so bincode, postcard and similar formats are out of scope. Known limitations, performance numbers and notes on unsafe code are in LIMITATIONS.md.

This crate heavily borrows from miniserde, serde and Sentry Relay's meta system. The general trait design was modelled after miniserde.

• GitHub Repository
• Issue Tracker
• Documentation
• Development Documentation
• License: Apache-2.0