0x0B and 0x0C stripped from header values

[wpt-issue-mover]

Originally posted as w3c/wptserve#111 by @annevk on 11 Jan 2017, 14:05 UTC:

These are not HTTP whitespace. They should be preserved I think. Where is the header value parsing code?

[wpt-issue-mover]

Originally posted as w3c/wptserve#111 (comment) by @Ms2ger on 11 Jan 2017, 14:36 UTC:

In some python standard library thing, I'm afraid :/

[wpt-issue-mover]

Originally posted as w3c/wptserve#111 (comment) by @jgraham on 11 Jan 2017, 15:22 UTC:

Yeah it's not trivial to change, but maybe we want to move away from using the stdlib parts here in general, especially if we want to allow early responses and so on.

Is this affecting a test that you're writing right now?

[wpt-issue-mover]

Originally posted as w3c/wptserve#111 (comment) by @annevk on 11 Jan 2017, 15:28 UTC:

Yes, this affects #4525.

[wpt-issue-mover]

Originally posted as w3c/wptserve#111 (comment) by @gsnedders on 11 Jan 2017, 17:21 UTC:

Yeah, I'm increasingly in favour of moving away from it as well. We could possibly refactor stuff once we're away from that too…

[wpt-issue-mover]

Originally posted as w3c/wptserve#111 (comment) by @annevk on 11 Jan 2017, 17:51 UTC:

What standard library are we using now? It seems we should file a bug there too if they strip 0x0B and 0x0C (if they use GitHub...).

[wpt-issue-mover]

Originally posted as w3c/wptserve#111 (comment) by @gsnedders on 11 Jan 2017, 17:54 UTC:

@annevk unless you can make it out to be a security issue, it's almost certainly not getting fixed in Python 2.

[wpt-issue-mover]

Originally posted as w3c/wptserve#111 (comment) by @gsnedders on 11 Jan 2017, 20:20 UTC:

(It seems to already have been fixed in Python 3.)

[wpt-issue-mover]

Originally posted as w3c/wptserve#111 (comment) by @annevk on 12 Jan 2017, 06:31 UTC:

https://bugs.python.org/issue22233 was the issue. It's a little strange they don't consider the CR LF sequence a single newline though. I wonder if that leads to issues. Anyway, moving to Python 3 is not an option so I guess we'll have to fix this here.

[wpt-issue-mover]

Originally posted as w3c/wptserve#111 (comment) by @gsnedders on 12 Jan 2017, 07:34 UTC:

Nah, that bug was only introduced in Python 3, but yes, would've broken this too!

python/cpython@15a312f is where the bug that breaks this in 2.7 was fixed in Python 3.

[annevk]

This doesn't affect much:

fetch/api/headers/header-values-normalize.html
mimesniff/mime-types/README.md

I think I can fix both by simply writing out the headers myself. It seems to me the header API could be reimplemented on top of the "output bytes asis" mode as well, but I'll leave that to someone else.

[annevk]

Unfortunately that simple fix does not work. The problem is that request header parsing is dropping these bytes and I'm not interested in meddling with that.