Fix error caused by invalid charset in response headers

[deosrc]

Ran into an error where we were having an exception raised from the requests library: 'bool' object has no attribute 'strip'

Seems the server was returning an empty charset value in the Content-Type response header of application/json;charset. This was causing the line below to default the value to a boolean True:

requests/requests/utils.py

Lines 524 to 529 in 2a6f290

	key, value = param, True
	index_of_equals = param.find("=")
	if index_of_equals != -1:
	key = param[:index_of_equals].strip(items_to_strip)
	value = param[index_of_equals + 1 :].strip(items_to_strip)
	params_dict[key.lower()] = value

This meant that attempting format the charset on the line below, the value was a boolean rather than a string:

requests/requests/utils.py

Lines 545 to 548 in 2a6f290

	content_type, params = _parse_content_type_header(content_type)
	if "charset" in params:
	return params["charset"].strip("'\"")

This change adds a type check so that the charset is ignored if a value is not provided. The unit test has been modified for coverage, along with expanding the other test scenarios slightly.

[nateprewitt]

Thanks for the PR, @deosrc! You've chosen an interesting time to raise this. We're currently working on options for typing in Requests and this was a case where there actually appears to be a bug in the code. Note that the response the server is giving you is not a valid response though and should be fixed on their end.

True was chosen for a default in #4442 but doesn't make sense according to the RFC. A parameter is defined to be:

     media-type = type "/" subtype *( OWS ";" OWS parameter )
     parameter = token "=" ( token / quoted-string )

Which means the minimum viable parameter is <token>=. If we read farther in the RFC we find that quoting in parameters is interchangeable without affecting the semantics so <token>= is equivalent to <token>="". Our default in this case should just be an empty string rather than True.

We don't rely on value being boolean anywhere and there's no way to leave the function with the boolean and emit a successful request. The fix we'd proposed on the typing branch was to change the default. This will result in an empty charset= being treated as UTF-8 when decoding rather than an unhelpful ValueError.

[deosrc]

Note that the response the server is giving you is not a valid response though and should be fixed on their end.

Absolutley, this has been fixed.

Your response all makes sense to me and sounds like it's fixed in the typing branch. I'll leave this up to you if you want to merge it in as an fix while the typing is ready, or for those unable to use typing. Happy for it to just be closed if it's not worth it as it's a rather obscure bug.