improves http body decoding and enforces max length #295
Conversation
bwireman
force-pushed
the
bw/improve-http-body-decoding
branch
from
5ed7568
to
ea5fb9d
Compare
| decoder := encoder.NewDecoder() | ||
|
|
||
| //"windows-1252" is the default value and will likely not decode correctly | ||
| if certain || encoding != "windows-1252" { |
There was a problem hiding this comment.
Why won't windows-1252 decode correctly? Did you run into specific issues with that encoding?
There was a problem hiding this comment.
Sorry I should have explained this better. windows-1252 is what determineEncoding returns when it has no idea and is causing issues like the one we saw this morning. So unless it's certain I don't want to use it
There was a problem hiding this comment.
There was a problem hiding this comment.
An edge case would be a detection of windows-1252 earlier than the default (via prescan, I guess). But given the returns of charset, I guess there is no other way to know that this is the default rather than computed guess (other than manually re-implementing charset.DetermineEncoding). So your solution is probably best.
improves http body decoding and enforces max length
How to Test
Notes & Caveats
I decided the initial read should still be limited and that we should just re-enforce length afterwards, but I can see the argument for not doing that
qq.com is a good example because it decodes correctly even though it isn't certain