Field-name syntax

[mnot]

Header field-names are defined as tokens. This is an extremely permissive syntax, including characters that will cause confusion and likely break some senders/recipients.

Most of the special characters allowed are not in the registry or seen "in the wild." Some research would be good to substantiate their use, but a starting point might be:

"-" / "_" / "." / "+" / DIGIT / ALPHA

There are a number of strategies we could take to the transition:

1. Like OWS / BWS, mark some characters as "do not generate" but "should consume"

2. Disallow registration of header fields with those characters, and discourage their use in unregistered headers

3. If we have more confidence that they're not in use, just ignore headers containing those characters.

[royfielding]

My opinion is that Apache httpd team would prefer to reduce the syntax to reduce the security issues.

[mnot]

Discussed in Montreal; interest in doing something, take to list.

[annevk]

I'm not sure why this is being considered. All browsers support headers such as !#$%&'*+-.^_`|~0123456789abcdefghijklmnopqrstuvwxyz. Changing that seems likely to break applications.

[royfielding]

The restrictions exist to protect legacy server gateways (like CGI) that are far more restrictive than browsers need to be due to the terrible idea of passing names through env vars and command-lines.

[mcmanus]

in bkk: no real consensus, concern about compatibility. reconsider on impact of a case by cases

[PiotrSikora]

For the record, NGINX drops request header fields with names containing characters other than letters, digits, hyphens and optionally underscores (if configured using underscores_in_headers directive). Unfortunately, virtually anything seems to be accepted and forwarded in the response headers.

[wtarreau]

Since I'm seeing the conversation moved from the WG to GH, I'm just pasting what I sent there for completeness. Right now haproxy only accepts :
"-" / "_" / "." / "+" / DIGIT / ALPHA / "!" / "#" / "$" / "%" / "&" / "'" / "*" / "^" / "`" / "|" / "~"
i.e. everything matching a token. Quite honestly, seeing any character from this extra list in a field name would look extremely suspicious to me, and I'd rather get rid of them.

As Roy mentioned, the restriction is to avoid trouble with CGIs doing :
eval "hdr_name=$value"

Characters like backquote (`), dollar ($), or pipe(|) have long been abused to attack servers...

[wtarreau]

Also I proposed that we could do something less extreme than blocking messages containing such header field names, we could recommend to simply drop these fields by default. This will have no impact if they're here by accident. Then we can let the agents decide if they want to let them pass or not. Thus we could make the difference between "forbidden characters" (the historical ones) and "unusual characters" (those excluded by the new, more restrictive list).

[mnot]

Discussed in Bangkok, but no consensus on an approach.

[mcmanus]

ietf104: still seeking data for characters used in the wild. http archive mentioned as a possible source.

[reschke]

I'm sceptical about the "protect CGI servers" (and servers similar to those). We have 2020; shouldn't these all have their own protections by now?

[mnot]

Discussed in Basel; suggestion is to document "safe" characters in prose (for now), both below the ABNF and in the new header field recommendations.

[PiotrSikora]

I'm sceptical about the "protect CGI servers" (and servers similar to those). We have 2020; shouldn't these all have their own protections by now?

Those CGI servers cannot protect themselves. The issue is that both hyphens and underscores are converted to underscores in CGI servers, so both Content-Length and Content_Length are converted to CONTENT_LENGTH at the protocol level, and CGI servers cannot tell which HTTP header did it originate from, since that information is lost.

[mnot]

Hey Piotr,

We chatted about that here and tend to agree -- will remove underscore and see how that goes down.

[royfielding]

Well, to be clear, it is the HTTP server that invokes CGI that is creating the environment variables from the received header fields. That server is responsible for avoiding bad transformations, such as the underscore problem above. However, since we are just talking about the characters that are good practice, we can exclude underscores since they are not commonly used in field names.