Maturing the API

[dhensby]

This signifies a substantial re-write of the library to more completely follow the httpbis message signatures specification.

Approach:

There are two main functions, signMessage() and verifyMessage(). Each accepts: a config to dictate some parameters for the signing/verification logic; a message (to be signed); [optionally] a third message (the request to bind to if signing a response and binding is required).

Signing Config:

• key: The signing key to sign your request with
• name: [optional] The name of the signature in the Signature and Signature-Input headers - collisions are automatically avoided
• params: [optional] A list of the parameters to include in the signature (eg: created, expires, etc)
• fields: [optional] A list of fields to sign, these can either be derived components or headers or a "structured field" to indicate encoding or other properties of the component to be signed.
• paramValues: [optional] The parameter values to use instead of allowing the library to determine them itself, eg: specify a created or expires time if you want your own one.

Verifying Config:

• verifier: The verifier callback
• tolerance: [optional] A clock tolerance for the time comparisons (in seconds). default: 0
• notAfter: [optional] Dictates a maximum date for the created key - this stops us validating signatures that are too old. default: Date.now() + tolerance
• maxAge: [optional] Essentially overrides the expires time on the signature - if provided we won't validate signatures that are older than this time (seconds).
• requiredParams: [optional] A list of parameters that must be signed for us to consider the signature valid
• requiredFields: [optional] A list of derived components/headers that must be signed for us to consider the signature valid
• all: [optional] Require all signature to validate (default: false) - NB: If a message has multiple signatures, it is expected that a party validating the signature only cares about the signatures that it can validate (ie: ones where it knows the keyid), as those are trusted keys. Other signatures may have been added by other applications that the current verifier has no knowledge of (and thus no access to the key to perform validation) so they can be ignored whilst still validating the signature. In some circumstances where the application is prepared to tightly couple with logic around expected signatures, this flag can be used to force errors if there is any signature that either can't be validated because it came from an unknown source or because it is invalid. This is raised in section 7.2.6

This includes full support for:

• Signing and verification of requests and responses
• Request-response bound signatures
• Full support of the derived components, including @query-param type (which was previously not supported), and header encoding rules.
• Added ed25519 key support and also rsa-sha1 signatures (as per the legacy spec).

Outstanding items:

• Add an ability for consumers of the library to provide a component parser. The specification leaves open the ability to add components to signatures that are not (yet) defined in the specification. It's important that the application can parse these and act on them as required. It's also important that, if consumers of the library want to rely on a newly defined component, but the library is not updated fast enough, they can add support for these without needing the upstream library to be updated.
• A bit of consideration needs to be given to how consumers look up the key to perform verification with
• A bit more code tidying - I think there's a bit too much repeated encoding/decoding of the structured headers and this could probably be made more efficient/streamlined.
• Guidance/planning on how to correctly build the url parameter for incoming requests (perhaps a function that can do this from a raw IncomingMessage object)
• Go through the spec again making sure that any MUST items are followed appropriately and any examples are followed
• Proper integration tests, including actual HTTP requests/responses (run a HTTP server as part of tests and actually assert it works)
• Documentation! Proper documentation of the API plus examples of how to sign and verify requests

Possible future improvements (out of scope of the rewrite):

• It may be useful if, when validating request/response message signatures, that the library return a "new" request that only includes the signed values of the request - ie: a "trusted" message is returned, preventing the application from accidentally processing untrusted headers/values. There may be a need to provide a list of "always trusted" headers (eg: content-type) for operational purposes but ideally there would be agreement between parties that these operationally required headers be signed ahead of time. see https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-message-signatures#section-7.2.1
• Rather than trying to validate all signatures in a message, the consumer could provide a filtering function that determines if the signature should be processed at all (eg: by looking up a keyid in a list of known keys), thus preventing signatures that could cause an error from breaking the validation of a good signature. - EDIT: This is essentially done and the job of the keyLookup function - it can throw errors or return null based on the signature params provided.
• Accept-Signature support - The Accept-Signature header could be used to "automatically" configure signing. By parsing this header we can determine:
• Required fields to be signed, required parameters to use (eg: keyid, alg, nonce, name), and even multiple signatures. - EDIT: done

[dhensby]

A checklist from the HTTPbis specification (note they only just released an update: https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-message-signatures-13)

An aspect not currently supported is JWS Algorithm support . It is not currently clear to me how this would be used; it's implied that the algorithm would be defined by a JWK or "other mechanism common to JOSE implementations", as such this may be that there is a JSON Web Key Set (JWKS) available where the key could be found by its keyid (kid) and then validated that way. This is left without any guidance at all, so this kind of implementation will have to stay firmly in the application logic and not within the scope of this library (for now).

High level requirements of a HTTP Message Signing Application/Library (see Section 1.4 of spec):

• Allow specification of required/expected component identifiers
• Allow specification of required/expected signature parameters
• Be able to retrieve keys to verify the signature (eg: by keyid param)
• Be able to determine signature algorithms (eg: by alg param)
• Able to reject signatures that are signed with inappropriate algorithms for the key in hand
• Be able to determine the "context" for a signature and derive components from it

Message components

• Component names MUST appear only once (at most, I presume) where a component identifier consists of a unique combination of component name and parameter(s)
• Component parameter order MUST be preserved, but is not relevant when comparing equality (eg: component;parm1;param2 is the same as component;param2;param1)
• Component values MUST NOT contain \n characters

HTTP Fields

• HTTP Field component names MUST be lowercased
• Unless there is specific strict formatting parameter present, multiple HTTP Fields must be combined into a single combined value
• If a field has a sf parameter, the value must be serialized as per sf rules - sf may only be used on values that are known to conform to a valid type for a structured field.
• If multiple field values exist for a sf header, the value must be combined into a single value before serialization
• If a field has a bs parameter, the value must be serialised as specified in 2.1.3

Derived components

• Derived components (those beginning with an @) MUST be treated separately from HTTP fields
• @method must be uppercased
• @target-uri support (it is not clear to me if the domain name in this should be normalised in anyway)
• @authority must be lowercase
• @scheme must be lowercase
• @request-target should use the request-target portion of the request line (however this is difficult to determine in node)
• @path must be absolute path of request and no query params
• @query must be the complete query string where percent encoded octets are decoded and with leading ?
• @query-param allows addressing of individual query parameter values, where percent encoded octets are decoded
• If a @query-param parameter is named but does not exists, an error must be thrown
• If a @query-param has many values, it must be included on separate lines in the signature base in the order they occur
• @status MUST not be used on requests
• @status MUST be 3 digit response code (no description text)
• @signature-params MUST NOT be specified in derived components

Signature parameters

• created MUST be UNIX timestamp (in seconds) integer value
• expires MUST be UNIX timestamp (in seconds) integer value
• nonce MUST be a string
• alg MUST be a string
• keyid MUST be a string
• tag MUST be a string
• @signature-params is a serialised InnerList as per structured-field spec

Request-Response binding

• When signing a response, a req parameter can be added to a derived component or http header to indicate the value came from the triggering request
• The same component name MAY be included for both the request and response (eg: "content-type";req "content-type" is allowed)
• The req parameter MUST NOT be used in signatures for requests themselves

Signature base

• Unknown parameters must produce an error
• Incompatible parameters must produce an error (eg: sf and bs flags together are incompatible)
• req parameter for a target message that is a request must produce an error
• If a component value cannot be derived, an error MUST be produced
  • Unrecognised component name
  • Component name not present in the message or the value is malformed
  • A parameter is unknown or does not apply to the identifier
  • The field has an sf parameter but the field is known not to be a structured field
  • A dictionary member is not present or the underlying dictionary is malformed

Creating Signatures

• Signers must choose appropriate keys and algorithms for signing
• Signature creation time is set to current time
• Signature expiry time set if applicable
• Once order of components is decided, it MUST NOT change
• Each covered component MUST be a derived component or http header field
• @signature-params MUST NOT be in the component identifiers and it MUST be the last item in the signature base

Verifying Signatures

• Ensure signatures have corresponding signature-inputs
• Validate signature parameters (they are valid and all required components/parameters are present)
• Potential application configuration options:
  • Allow a set of required headers in a signature to be specified
  • Allow a maximum age for a signature to be specified
  • Reject any expired signatures
  • Reject based on parameter values (eg: alg)
  • Ensure key is found via keyid (if present)
  • Allow certain algorithms to be prohibited or mandated
  • Require keys to be of a certain size
  • Enforcing a unique nonce parameter
  • Require an application specific value for the tag parameter
• Additional configuration options MUST be enforced and MUST NOT accept signatures that don't conform

Signature algorithms

• Signatures MUST use appropriate signatures/MAC methods for the key type provided
• Support rsa-pss-sha512
• Support rsa-v1_5-sha256
• Support hmac-sha256
• Support ecdsa-p256-sha256
• Support ecdsa-p384-sha384
• Support ed25519

Message Signature Headers

• Signatures are in the Signature header as a dictionary with a unique label
• Signature input in the Signature-Input header as a dictionary with a unique label
• Label is determined by signer except where the label is dictated by protocol
• The Signature-Input value MUST be the same value used in the signature base
• Signature-Input may be a trailer (but recommended not to be)
• Signature-Input may be included multiple times, the value should be combined before processing
• The Signature header may be a trailer (but recommeneded not to be)
• Signature may be included multiple times, the value should be combined before processing
• Multiple signatures are allowed and a signature may sign another signature

Other security measures

• Signatures by all known signers should be validated
• Signature downgrade attacks should be prevented (ie: A key should specify which algs it is prepared to work with), an RSA key should not be able to be used with HMAC (for example)
• Must not get confused with header names that may start with an @

[dhensby]

After going through the specification further and thinking of use cases where multiple signatures may be added to a message, I think that it would be better for the signature verifier to attempt to validate as many signatures as it can. Essentially, every signature with a key that is known to the application should be validated. This is because there are instances where all the signatures (produced by known keys) do need to be verified to ensure complete coverage of trust for the request:

1. When using Accept-Signature, the specification currently dictates that no extra components can be added to the requested signature. As such an application may choose to add two signatures - one to meet the requested signature specification, and another to cover other components that it wants to provide guaranteed integrity over.
2. If some trusted intermediary is also adding a signature to the request (eg: a proxy is writing some new headers and wants to sign them for authenticity), then the application would want to validate both of those to ensure the integrity of all the headers.

Essentially, different signatures can be covering different parts of the message and to guarantee the message as a whole all those signatures need to be validated. If you just accept the first one you see, you may only be validating the proxy's signature (which only covers 1 or 2 components) and ignoring an invalid signature that attests to cover a more comprehensive list of components.

[adrianhopebailie]

Awesome work!

[frankh077]

Hello, thanks for all the work and effort in the implementation of the draft. Could you upload a full example of how to sign and verify a request and response please?

[wilsonianb]

👋 I've tested this out on the project that @adrianhopebailie is involved with:

• feat(open-payments): add verifySignatureHeaders interledger/rafiki#752

It only uses httpbis ed25519 signing and verifying, so it's not a comprehensive test of this PR's changes.
But the parts I'm using work as expected.

Looking forward to the eventual release. Thanks!

[dhensby]

I'm going to take this in as is, and then other improvements can come later