Add an option to json.tool to bypass non-ASCII characters.

[legnaleurc]

BPO	27413
Nosy	@rhettinger, @bitdancer, @methane, @berkerpeksag, @serhiy-storchaka, @legnaleurc, @qingyunha
PRs	bpo-27413: add --no-ensure-ascii argument to json.tool #201 bpo-30971: Improve code readability of json.tool #2720 bpo-27413: json.tool: Add --no-ensure-ascii option. #17472
Files	json-add-an-option-to-bypass-non-ascii-characters-v4.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-12-06.06:44:23.432>
created_at = <Date 2016-06-29.11:49:29.439>
labels = ['type-feature', 'library', '3.9']
title = 'Add an option to json.tool to bypass non-ASCII characters.'
updated_at = <Date 2019-12-06.06:44:23.427>
user = 'https://github.com/legnaleurc'

bugs.python.org fields:

activity = <Date 2019-12-06.06:44:23.427>
actor = 'methane'
assignee = 'none'
closed = True
closed_date = <Date 2019-12-06.06:44:23.432>
closer = 'methane'
components = ['Library (Lib)']
creation = <Date 2016-06-29.11:49:29.439>
creator = 'legnaleurc'
dependencies = []
files = ['44246']
hgrepos = []
issue_num = 27413
keywords = ['patch']
message_count = 13.0
messages = ['269479', '269498', '269559', '269584', '269641', '269648', '269667', '273803', '273814', '273817', '273819', '273911', '357902']
nosy_count = 7.0
nosy_names = ['rhettinger', 'r.david.murray', 'methane', 'berker.peksag', 'serhiy.storchaka', 'legnaleurc', 'qingyunha']
pr_nums = ['201', '2720', '17472']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue27413'
versions = ['Python 3.9']

[legnaleurc]

This patch adds a command line option "--no-escape" that allows json.tool to display non-ASCII characters.

e.g.:

$ echo '"測試"' | python -m json.tool
"\u6e2c\u8a66"

$ echo '"測試"' | python -m json.tool --no-escape
"測試"

[bitdancer]

Maybe name it --no-insure-ascii? Or --insure-ascii=no/yes. (or true/false).

[legnaleurc]

If the arguments should be aligned with those in dump/load, then maybe "--no-ensure-ascii" is an option?

[bitdancer]

Sorry, yes, that's what I meant. I think it will make it easier to understand and remember if the option uses the same terminology as the function.

[legnaleurc]

Use "--no-ensure-ascii" instead.

[berkerpeksag]

The patch needs tests and documentation.

• parser.add_argument('--no-ensure-ascii', action='store_true', default=False,

I'd go with action='store_false', default=True.

[legnaleurc]

The patch needs tests and documentation.

Ok, I'll update it later.

• parser.add_argument('--no-ensure-ascii', action='store_true', default=False,
  I'd go with action='store_false', default=True.

If I'm not misreading your comment, this will change the original behavior, right? (because options.no_ensure_ascii will be set to True by default)

[legnaleurc]

Added doc and test.

[serhiy-storchaka]

Test fails on non-utf8 locale.

$ LC_ALL=en_US ./python -m test.regrtest -v -m test_no_ensure_ascii_flag test_json
...

======================================================================
FAIL: test_no_ensure_ascii_flag (test.test_json.test_tool.TestTool)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/test_json/test_tool.py", line 115, in test_no_ensure_ascii_flag
    self.assertEqual(out.splitlines(), b'"\\u6e2c\\u8a66"\n'.splitlines())
AssertionError: Lists differ: [b'"\\u00e6\\u00b8\\u00ac\\u00e8\\u00a9\\u00a6"'] != [b'"\\u6e2c\\u8a66"']

First differing element 0:
b'"\\u00e6\\u00b8\\u00ac\\u00e8\\u00a9\\u00a6"'
b'"\\u6e2c\\u8a66"'

• [b'"\\u00e6\\u00b8\\u00ac\\u00e8\\u00a9\\u00a6"']
  + [b'"\\u6e2c\\u8a66"']

----------------------------------------------------------------------

[legnaleurc]

1. Replaced non-ASCII literals to \uXXXX
2. Removed failed assertion

Test passed with LC_ALL=en_US and LC_ALL=en_US.UTF-8 .

I've tried to use locale.getdefaultlocale(), but seems the output string will vary in different locales.

[berkerpeksag]

If I'm not misreading your comment, this will change the original behavior, right?

Assuming you also change

   ensure_ascii = not options.no_ensure_ascii

to

   ensure_ascii = options.no_ensure_ascii

no, it won't change the original behavior. That way you won't need to invert the value of options.no_ensure_ascii in line 37.

[serhiy-storchaka]

The last change just sweeps a problem under a rug.

For now json.tool never fails with valid data. But with the --no-ensure-ascii option it can fail when output a string not encodable with the locale encoding. All can work with common cases on common UTF-8 environment, but unexpectedly fail on nonstandard environment. It would be better to output encodable characters as is and represent unencodable characters with \uXXXX encoding.

[methane]

New changeset efefe25 by Inada Naoki (wim glenn) in branch 'master':
bpo-27413: json.tool: Add --no-ensure-ascii option. (GH-17472)
efefe25