Getting a request with a BOM mark

[GP89]

I was using requests to get some json form a url. I noticed at the start it had a weird unicode character. The start of the response text was u'\ufeff[{"Hos... (the message was sent with content-type of text/javascript; charset=utf-8)

After looking into it a bit, it looks like it was a BOM mark. Looking for solutions I came across this post which suggests decoding the string, which would work fine using urllib2 as it gives back the byte string. But in requests the response objects text attribute is already decoded with the BOM mark included.

I was wondering if this was something that could be addressed automatically? Or maybe it might end up biting people who expected to see that mark in the response text?

[Lukasa]

If the BOM isn't getting stripped, the problem is that we're decoding it as non-UTF-16. In such a case, the BOM is strictly unnecessary, and a leading BOM is in fact not a BOM but a unicode zero-width non-breaking space character.

Now, the content-type header seems to suggest it's not a BOM, so we're doing what it says and decoding as UTF-8. In that situation, as mentioned above, the BOM is not meant to be present and so has meaning.

We shouldn't automatically strip it off because we can't unconditionally guarantee that it shouldn't be there. If users want it stripped they can easily do so themselves by unconditionally removing leading instances of u'\ufeff'.

[GP89]

That was my gut feeling - I didn't realise that char could mean something else other than a BOM though - so I'd agree to leave it as it is 👍