Support for Optional in ASN.1

[fjarri]

Currently picky-asn1-der simply skips any None-valued Optional fields during serialization, which causes a subsequent deserialization to fail. Would it be possible to serialize a NULL value in this case, or does it somehow contradict the standard?

[CBenoit]

Hi @fjarri, thank you for opening this issue.

I understand from where you are coming from. Indeed, this behavior is following the ASN.1 standard: a component inside a SEQUENCE or SET denoted as OPTIONAL is simply not serialized at all. The approach to deal with this kind of optional fields in ASN.1 is to make use of tagged types in order to distinguish optional components.

See for instance, the TBSCertificate definition from RFC 5280:

TBSCertificate  ::=  SEQUENCE  {
        version         [0]  EXPLICIT Version DEFAULT v1,
        serialNumber         CertificateSerialNumber,
        signature            AlgorithmIdentifier,
        issuer               Name,
        validity             Validity,
        subject              Name,
        subjectPublicKeyInfo SubjectPublicKeyInfo,
        issuerUniqueID  [1]  IMPLICIT UniqueIdentifier OPTIONAL,
                             -- If present, version MUST be v2 or v3
        subjectUniqueID [2]  IMPLICIT UniqueIdentifier OPTIONAL,
                             -- If present, version MUST be v2 or v3
        extensions      [3]  EXPLICIT Extensions OPTIONAL
                             -- If present, version MUST be v3
        }

You can see that all optional fields also have a context-specific tag denoted by the square brackets [X].

Serializing None values as NULL in ASN.1 would be like using a CHOICE type such that:

    RustOption ::= CHOICE {
        ok             OkType,
        none           NULL }

This is arguably a better default for the Rust Option when you don’t need to deal with RFC implementations and especially for a library that is built on top of serde. If I were to build a Rust-centric serde-based der serializer / deserializer, that is the approach I would go for. This results in much better ergonomics when one does not need to implement a specification but merely needs some binary encoding.

That being said, I think RustCrypto’s der crate is now doing a better job at providing a good API when following a specification by not using serde. We originally used serde to benefit from the ecosystem, reusing the well crafted procedural macro, but truth is that serde data model is not able to represent the whole ASN.1 data model and we sometimes need to resort to ugly hacks when following specifications.

If you don’t care about ASN.1 / DER in particular and merely need a fast binary encoding as an implementation detail, maybe bincode or postcard will do a better job for you?

I hope it helps!

[fjarri]

Thank you for the detailed reply, I suspected there was some reason for this behavior because of the standard. I wanted to use DER in my library to be a little more future-proof and allow easier introspection, but if it can't be mapped to Rust data model, I guess I'll have to stick with MessagePack. Feel free to close the issue if you are going to keep things as they are.

[CBenoit]

I wanted to use DER in my library to be a little more future-proof and allow easier introspection, but if it can't be mapped to Rust data model, I guess I'll have to stick with MessagePack.

Actually, I like the idea and am interested in forking the project with the explicit goal of serializing / deserializing DER using the serde data model only. This means that being able to implement most ASN.1 definitions from RFC is an explicit non-goal, but you would still get the advantage of using a wide-used binary format (future-proof, tooling…). One selling point of this crate would be the aforementioned Option serialization for Rust-side ease of use. Is that something you would be ready to use in rust-umbral?

[fjarri]

Theoretically, yes, but this was a convenient time for me to switch to another format, since ABI was already being changed for other reasons. Plus the application where it is used is in a semi-beta stage, so there are no migration problems. I may or may not find another convenient moment later. But I am in the process of writing another library where I could use DER serialization, and I'm sure a lot of other people will find it useful too.

Also, it would be great if you could take over https://crates.io/crates/serde_asn1_der since that's the place where everyone searching for "ASN1 DER serde" ends up.

[CBenoit]

Hi again! Sorry for the delay. I had a look at https://crates.io/crates/serde_asn1_der
Actually, it does what I mentioned above already so there is in fact no point in forking picky. It may look like it’s semi-abandoned because there is not a lot of activity, but I believe the reason is simply because the crate is pretty much feature complete at this point.
I’ll close the issue for now.

[fjarri]

A big issue with serde_asn1_der is that it reports the serializer as human-readable. Also it does not serialize mappings.

[CBenoit]

A big issue with serde_asn1_der is that it reports the serializer as human-readable. Also it does not serialize mappings.

I opened a PR to address the first point: KizzyCode/serde_asn1_der-rust#5
Regarding "mappings", I understand you refer to types such as std::collections::HashMap. IIRC, I don’t think picky supports this either. DER doesn’t really support mappings like JSON, but I guess this can be serialized as a SEQUENCE of Key/Value pairs.

cc @KizzyCode

[KizzyCode]

A big issue with serde_asn1_der is that it reports the serializer as human-readable. Also it does not serialize mappings.

As for the human-readable part, this is a bug since DER is definitively not human readable – this will be fixed in any case once I understand the implications (this could be a breaking change in some circumstances).

With regards to mappings: This is a limitation of the DER format; there is no standardized representation for mappings. Like @CBenoit said, a common "canonical" solution for this is a sequence of key-(attribute-)value-sequences. For example, X.509 v3 extensions are represented as a sequence of (key,attribute,payload)-tuples:

Extension  ::=  SEQUENCE  {
     extnID      OBJECT IDENTIFIER,
     critical    BOOLEAN DEFAULT FALSE,
     extnValue   OCTET STRING  }

[fjarri]

DER doesn’t really support mappings like JSON

I don't think it's the right way of looking at things. DER does not support structures either, but we serialize structures into it as sequences of fields. Then the deserializer knows to pack it back into the structure. A structure is just a sequence with an implied contract (say, that the second value must be an integer), and the contract is enforced by the deserializer.

Same for the mapping. It is again just a sequence with some implied contracts (every element is a 2-sequence, and all the first elements are unique), which will be enforced by a deserializer. DER itself does not need to support all the multitude of possible data structures. Even JSON does not really "support" mappings completely, there is no syntactical enforcement of key uniqueness, it is checked by the deserializer.

there is no standardized representation for mappings

That is true, but: there is a de-facto standard, as you mentioned; and how many possible ways are there to serialize a mapping, anyway? Probably another one is a sequence of keys and a sequence of values, but that seems really forced.

Moreover, even if it is not standard, it won't affect the people who implement existing specifications, because they know their target representation does not have a mapping, so they will not use it. And @CBenoit mentioned that

I like the idea and am interested in forking the project with the explicit goal of serializing / deserializing DER using the serde data model only. This means that being able to implement most ASN.1 definitions from RFC is an explicit non-goal, but you would still get the advantage of using a wide-used binary format (future-proof, tooling…).

[KizzyCode]

I don't think it's the right way of looking at things. DER does not support structures either, but we serialize structures into it as sequences of fields. Then the deserializer knows to pack it back into the structure. A structure is just a sequence with an implied contract (say, that the second value must be an integer), and the contract is enforced by the deserializer.

Well, there is a difference: ASN.1's sequences are designed as a direct 1:1 analogon to typed, ordered structs, whereas there is no native equivalent to mappings.

Same for the mapping. It is again just a sequence with some implied contracts (every element is a 2-sequence, and all the first elements are unique), which will be enforced by a deserializer. DER itself does not need to support all the multitude of possible data structures. Even JSON does not really "support" mappings completely, there is no syntactical enforcement of key uniqueness, it is checked by the deserializer.

For example there is one problem with the simple map-representation above: DER has the contract that encodings are reproducable – each input has one and only one valid DER representation (this was actually one of the design goals behind DER as opposed to e.g. BER). This means: If you encode the same input twice, it must produce the same binary output every time – even on different systems with different environments and different implementations etc.

One problem is that you need to define a strict ordering of elements. This is fine for types like arrays, vecs, tuples or structs, which carry information about their ordering we can use. However, other types – like HashMap – do not, so we would need an additional information on how to order the elements. We could use Ord, however Ord's behavior is Rust-specific and such an implementation would by default only be interoperable with Rust.

In an early version of asn1_der I actually had a map feature, and this caused a surprising amount of trouble, so I decided to scrap this and leave it to the library user. And especially with #[derive(Serialize, Deserialize), there is no real API interaction between the library user and serde_asn1_der and tbh I'm a bit reluctant to overload this API with even more Rust-specific implicit contracts.

In general, I'm a bit torn apart... One of my backgrounds is cryptography and embedded development, and in this area implicit contracts like this are a no-go. However on the other hand, I realize that DER is tempting as efficient general-purpose serialization where requirements may not be as strict. It could be worth a shot to implement a best-effort map serialization behind a feature gate and see how this works IRL; but no promises 😅

---

Edit: One could also argue that I already made a similar implicit assumption for Option – for me it seemed much more straight-forward than maps at this point, but... 🙈