Payload processing specification

[aidantwoods]

Since there's no issue open specifically for this:

Currently the documentation only goes so far as to specify the Paseto message format, but does not specify how to process this payload after decrypting or verifying it.

From reading the reference implementation I can probably answer most of the following myself, the goal here is to highlight questions that the spec should answer if processing the payload is going to be part of the specification, it's not just a series of personal questions :p

• How payloads represented as strings should be encoded into bytes (utf8?)

• Is a JSON encoded payload part of the spec? Is it required that (received||sent) payloads are in this format, or optional?

• Specify the type (or even data) structure (if any) that the JSON payload should conform too

  • Reference implementation appears to hint that this is array<string, string>, but I think it actually allows array<string, mixed> due to a type hint diverging with the doc block. See also probable need for type validation on bulk setting of claims.

• Are arbitrary keys allowed to be set? (or is there a reserved character set for example?)

• Are any keys reserved for certain purposes? (e.g. the ones used for rules like expiry)

• Are these reservations case sensitive? (e.g. should an expiry key with incorrect case be ignored?)

• Should users be allowed to set reserved keys like arbitrary other keys, or should distinct mechanisms be enforced?

• If users can set these, do we fail if the user puts unexpected (e.g. unparseable) data in these reserved fields (i.e. should writes to reserved fields be validated)?

• Are rules part of the spec? (if so what are they?)

• Are rules specified as reserved keys, or are they in a structurally different part of the JSON payload to user data?

• Is rule validation required? (if no, what should be the default?)

• Do we fail open or closed if a rule is of an unexpected format?

• Will payload processing receive spec updates (e.g. potential changes to rules, addition of rules), how should this be versioned?

• If payload processing is versioned, where do we put this version?

Can probably ask at least the encoding question for processing the footer too, though that seems really intended just to be a string.

[paragonie-scott]

Let me attempt to answer this line-by-line, then document it:

---

• How payloads represented as strings should be encoded into bytes (utf8?)

  • Yes, UTF-8.

• Is a JSON encoded payload part of the spec? Is it required that (received||sent) payloads are in this format, or optional?

  • Originally it was going to be optional so people could use Protobuf, etc. However, I've since decided to just use JSON.

• Specify the type (or even data) structure (if any) that the JSON payload should conform too

  • Reference implementation appears to hint that this is array<string, string>, but I think it actually allows array<string, mixed> due to a type hint diverging with the doc block. See also probable need for type validation on bulk setting of claims.
  • OK.

• Are arbitrary keys allowed to be set? (or is there a reserved character set for example?)

  • By keys do you mean in the array<key, value> sense? If so, there are "registered claims" which are very similar to JOSE's, except we use ISO 8601 datetime strings rather than unix timestamps.

• Are any keys reserved for certain purposes? (e.g. the ones used for rules like expiry)

• Are these reservations case sensitive? (e.g. should an expiry key with incorrect case be ignored?)

• Should users be allowed to set reserved keys like arbitrary other keys, or should distinct mechanisms be enforced?

• If users can set these, do we fail if the user puts unexpected (e.g. unparseable) data in these reserved fields (i.e. should writes to reserved fields be validated)?

  • Users should not (but not 'MUST NOT') write arbitrary/invalid data to reserved fields (exp, iat, nbf, etc.)

• Are rules part of the spec? (if so what are they?)

  • No, we only provide them in the reference implementation for convenience.

• Are rules specified as reserved keys, or are they in a structurally different part of the JSON payload to user data?

• Is rule validation required? (if no, what should be the default?)

  • No, that entirely depends on what the application is doing with the tokens. Some people will only want tamper-proof cookies. Some people want the full shebang of JOSE features.

• Do we fail open or closed if a rule is of an unexpected format?

• Will payload processing receive spec updates (e.g. potential changes to rules, addition of rules), how should this be versioned?

• If payload processing is versioned, where do we put this version?

---

Anything unanswered (bold) is something I'm not sure about yet.

[aidantwoods]

Are arbitrary keys allowed to be set?

• By keys do you mean in the array<key, value> sense? If so, there are "registered claims" which are very similar to JOSE's, except we use ISO 8601 datetime strings rather than unix timestamps

Indeed, assuming the JSON structure is array<key, value>, this question is referring only to the key portion.
This is to ask whether:

1. Should conforming implementations allow users to pick their own keys, or is there some sort of whitelist (as fas as I know, answer is: user may pick keys)
2. Are there restrictions on what a key can look like (e.g. character set, casing, whitespace, ...).

With regard to the values, e.g. date formats, this comes under:

do we fail if the user puts unexpected (e.g. unparseable) data in these reserved fields (i.e. should writes to reserved fields be validated)?

which you've already answered :)

---

It's more than okay for the answer to some of these to be: "undefined", or "application specific", ... (some of these questions are really just aimed at pulling spec details out of the reference implementation)

[rlittlefield]

Originally it was going to be optional so people could use Protobuf, etc. However, I've since decided to just use JSON.

I'll assume that means I should make the JSON-specific route the favored option in my examples, and provide the raw-payload binary data only through a secondary route. Currently pypaseto only handles the raw portion, but I intend to build support for JSON and validation of registered claims soon.

I'm interested in getting msgpack functional as well, although due to the mostly-compatible nature, the processing of claims could actually be identical just with the middle encoding layer swapped out.

[paragonie-scott]

I'll assume that means I should make the JSON-specific route the favored option in my examples, and provide the raw-payload binary data only through a secondary route.

👍

[paragonie-scott]

This should be unambiguous enough now that #59 is merged. I'll reread this tonight and make sure before I close it.

[aidantwoods]

This should be unambiguous enough now that #59 is merged. I'll reread this tonight and make sure before I close it.

Yeah, I think #59 pretty much answers everything. I've checked everything above that I think has been explicitly answered. For what remains I would assume the following answers by omission:

---

Are these reservations case sensitive? (e.g. should an expiry key with incorrect case be ignored?)

Yes, case-sensitive.

If users can set these, do we fail if the user puts unexpected (e.g. unparseable) data in these reserved fields (i.e. should writes to reserved fields be validated)?

Up to implementation to come up with a means of discouraging invalid writes.

Do we fail open or closed if a rule is of an unexpected format?

Closed. (i.e. rule fails)

Will payload processing receive spec updates (e.g. potential changes to rules, addition of rules), how should this be versioned?

Unknown. (no mention of versioning for payload processing, leaning towards: No, this is a static requirement).

If payload processing is versioned, where do we put this version?

Unknown / n/a

[paragonie-scott]

060491a