":=" doesn't send non-ascii characters as "raw"

[wataash]

On my shell and terminal, non-ascii arguments are encoded in UTF-8:

$ echo -n '{"a": "あ"}' | xxd
00000000: 7b22 6122 3a20 22e3 8182 227d            {"a": "..."}

The document says := is "Raw JSON fields", so I expected a:='"あ"' to send 0x 7b22 6122 3a20 22e3 8182 227d ({"a": "あ"}) as is, but actually it ensures ascii, sending {"a": "\u3042"}.

I think it's counter-intuitive, and should be warned in stderr.

[jkbrzt]

@wataash HTTPie does not ensure ASCII any longer.

Btw, what is meant by “raw JSON” is just that the value will not be converted to a string value like foo=bar does.

http httpbin.org/post  foo=bar  bar:=true qux:=1

{
    "bar": true,
    "foo": "bar",
    "qux": 1
}

If you’d like to send data without any processing, then you can pass it through STDIN instead:

echo '{"a": "あ"}' | http httpbin.org/post

https://httpie.org/docs#redirected-input

[wataash]

Sorry for the very late reply.

Unfortunately, the problem is not changed. Now it's caused by json.dumps(data[, ensure_ascii=True]) ([, ensure_ascii=True] means the default option of json.dumps()).

Let me explain in detail.

JSON representation of Japanese-a

"あ" is a representative Japanese character with the Unicode code point of U+3042 [1]. Let's call it "Japanese-a" (since it's pronounced "ah").

In UTF-8, U+3042 Japanese-a is encoded as 0xe38182 [2].

[1]: https://unicode.org/charts/PDF/U3040.pdf
[2]: according to https://en.wikipedia.org/wiki/UTF-8#Encoding, U+3042 0x3042 (0b 0011 000001 000010) + 0b 1110xxxx 10xxxxxx 10xxxxxx -> 0b 11100011 10000001 10000010 -> 0xe38182

The JSON specification allows Unicode code points to be represented as both "\u<code point>" and UTF-8 encoded bytes [3].

// Both are valid JSON

// Unicode code point of Japanese-a
// in HTTP packet: 0x 22 5c 75 33 30 34 32 22
//                    ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^
//                     "  \  u  3  0  4  2  "
"\u3042"

// UTF-8 encoded Japanese-a
// in HTTP packet: 0x 22 e3 81 82 22
//                    ^^ ^^^^^^^^ ^^
//                     "       あ  "
"あ"

[3]: RFC 8259 7. Strings, 8.1. Character Encoding, 8.3. String Comparison

:= basically sends string 'as is', but non-ASCII string is exceptional

Btw, what is meant by “raw JSON” is just that the value will not be converted to a string value like foo=bar does.

Yes, so = syntax escapes a string to be a valid JSON string, whereas := does not.

#                          data_field=\n    raw_data_field:="\n"
#                          vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
$ http -v httpbin.org/post data_field='\\n' raw_data_field:='"\\n"'
{
    "data_field": "\\n",
    "raw_data_field": "\n"
}

So I guess data_field=あ might send the escaped representation "\u3042", and expected raw_data_field:='"あ"' to send the non-escaped one "あ" (" + 0xe38182 + "). But both send the escaped one!

http httpbin.org/post data_field=あ raw_data_field:='"あ"'

# Expected: sends {"data_field":"\u3042","raw_data_field":"あ"}
# Actual:   sends {"data_field":"\u3042","raw_data_field":"\u3042"}

Expected:

Actual:

(We cannot inspect this by http -v, so I pasted the Wireshark captures. See #1474.)

The escaping is done here:

json_dict_to_request_body, client.py:312
        data = json.dumps(data)  # RequestJSONDataDict([('a', 'あ')]) -> '{"a": "\u3042"}'
make_request_kwargs, client.py:335
collect_messages, client.py:59
program, core.py:213
raw_main, core.py:100
main, core.py:162
main, __main__.py:9
<module>, __main__.py:19

So := escapes only non-ASCII characters, is this expected behavior?

[wataash]

If you’d like to send data without any processing, then you can pass it through STDIN instead:

echo '{"a": "あ"}' | http httpbin.org/post

This works pretty well. it actually sends {"a","あ"} (あ is 0xe38182).