Incorrect encoding detected when no encoding is given

[untitaker]

On https://mozorg.cdn.mozilla.net/media/caldata/FrenchHolidays.ics, the encoding is set to ISO-8859-1, although no encoding was given in the headers, which is why i think it should be None instead.

[Lukasa]

This is well-known behaviour in requests, and comes right out of RFC-2616. We go into it here in the documentation, but in short: RFC 2616 says that if a Content-Type is received that is of type text/* (any text content type) but no charset is present, we must assume ISO-8859-1.

[untitaker]

Is there a way to detect this fallback as a user of requests?

[Lukasa]

Sure:

fallback_to_latin1 = (not 'charset' in r.headers['Content-Type']) and (r.headers['Content-Type'].startswith('text/')

[untitaker]

Okay, thanks!

[untitaker]

With http://evertpot.com/http-11-updated/, could there be any change on this?

[Lukasa]

@untitaker Way ahead of you, see #2086. =)