httpx forces converting json content to ascii

[alleml]

httpx.Request blindly dumps provided json to ascii characters by invoking json.dumps without any parameters. Currently there is no way to provide ensure_ascii=False argument to the Request instance (or even better to the client).

It leads to the following scenario:

>>> httpx.Request(method="POST", url="http://localhost:8000/test", json={"field": "cześć"}).content
b'{"field": "cze\\u015b\\u0107"}'

when I need properly encoded unicode bytes: b"cze\xc5\x9b\xc4\x87"

[alleml]

I guess a bug is here: https://github.com/encode/httpx/blob/master/httpx/_content.py#L177

json_dumps(json).encode("utf-8")

doesn't make sense if you forced conversion to ascii first.
I would recommend changing this line to:

json_dumps(json, ensure_ascii=False).encode("utf-8")

or providing a way to make it configurable to keep backward compatibility.

[tomchristie]

specs... https://www.json.org/json-en.html
I think we're getting this right?

[alleml]

ok, I got your point and indeed it is a valid approach but it is far from optimal. When we force converting unicode characters to ascii for json we have 6 bytes instead of 2 per each converted unicode character. Is there any reason why ascii is forced and there is no way to change this behaviour? Using unicode characters without forcing conversion to ascii is also compliant with JSON spec.

I'm developing audit proxy which captures raw request/response body and finally stores its in BigQuery. I have massive throughput and a lot of text containing non-ascii unicode characters. Forcing conversion from unicode to ascii leads to significant storage cost increase in my case, because we have to store much more bytes.

[alleml]

here is a code example explaining my concerns:

>>> import json
>>> len(json.dumps("ś").encode("utf-8"))
8
>>> len(json.dumps("ś", ensure_ascii=False).encode("utf-8"))
4

so you have extra 4 bytes per each forced to ascii unicode character.