Integrity requirement: how do we verify that no bits have been changed?

[shalka]

DISCLAIMER: This will likely not be included in v0.1 of the specification, barring further discussion on this issue.

When an event is published, should the publisher be required to include fields related to the integrity of the event envelope, payload, or both (such as checksums), that allow consumers to verify that the event has not changed? There are two primary concerns related to this issue:

• Malicious middleware that intentionally changes payload values
• Bits have been dropped/mangled during transport or conversion between formats

[erikerikson]

I feel this is important. Also, there is a challenge here: what bars malicious middleware intentionally changing payload values and also checksum?

Cryptographic signatures sprint to mind as a potential solution but this risks scoping out the consideration into a larger set of concerns.

[clemensv]

In AMQP, we introduced the footer for this purpose, but there has been little vendor momentum to fill that with life, because - as it turns out - doing so is rather complicated.

RFC7515 defines the JSON Web Signature and the metadata you'll need to carry to tell a receiver how to check that signature. JWS is an evolution of several prior standards, including XML Signatures. So that's the metadata set that would have to go into said footer.

Assuming having JWS, you still don't have a canonical way of forming a signature over structured content across different encodings, i.e. a JSON event would need to compute to the same signature as a MessagePack or Avro event, so you'll need a canonical type system and you need robust bi-directional type mappings into and from that type system if you want to maintain signature integrity while allowing transcoding.

The monster lurking under the surface of this is also authentication, because a valid signature doesn't do much for you if you can't also tell that the signing party is who you expect them to be. Without authentication, someone could manipulate an event, sign the manipulated event properly, and tell you the signing key. All you can tell then is that the forgery is authentic.

Where that leads me is to punt on the problem here for the time being and push this concern into the event payload. The event payload can use a full Auth/Sign/Crypt framework that fits the given event data encoding and it can indeed protect the event metadata by embedding copies of that data and signing those. If the payload-embedded copies and the metadata on the event envelope diverge, you can tell someone manipulated it. I'm not opposed to trying to solve it, but doing this right is a ton of work, and the memories of WS-Security throw dark, cold shadows.

[erikerikson]

This is deeply helpful Clemens.

The multi-format concern is indeed problematic and my inexact "cryptographic signatures" statement does imply key pair based identity and public key distribution, which truly does expand scope and allow for a vast amount of disagreement and more particularly the risk of fragility in the standard.

Thank you very much for this: continuing to digest.

[duglin]

At the 6/15 f2f we agreed that this wasn't required for v1.0, so I removed the v1.0 tag and added not-required-for-v1.0

[dmlb2000]

@clemensv Thanks for the info, I second the idea of punting on signing/encryption/authentication that come along with these issues. I would also add that many of the transports these events would go over already have done this work. It might be good to document the punt on this feature and provide pointers in the transport documents for how to configure them to handle these kind of issues.

[erikerikson]

@dmlb2000 Transports do address these issues. However, some use cases involve persistence. I've opened an issue to discuss whether persistence is considered within scope for the effort here.

[kfox1111]

Signing does have some advantages when its end to end, as it can be hard/impossible for a user to validate all parts in a complex routing deployment are covered by transport security. it may be plumbing that is never exposed to the user (for example when a cloud provider is used). Persistence is just one case where an implementation detail of the plumbing may make the risk of tampering or silent corruption worse.