add json.tool option to avoid alphabetic sort of fields

[PavelKazlou]

BPO	21650
Nosy	@rhettinger, @pitrou, @ezio-melotti, @bitdancer, @berkerpeksag, @vadmium, @serhiy-storchaka
Files	issue21650.diff issue21650_v2.diff issue21650_v3.diff issue21650_v4.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/berkerpeksag'
closed_at = <Date 2014-11-10.07:58:05.540>
created_at = <Date 2014-06-03.10:11:31.342>
labels = ['type-feature', 'library']
title = 'add json.tool option to avoid alphabetic sort of fields'
updated_at = <Date 2014-11-10.07:58:05.539>
user = 'https://bugs.python.org/PavelKazlou'

bugs.python.org fields:

activity = <Date 2014-11-10.07:58:05.539>
actor = 'berker.peksag'
assignee = 'berker.peksag'
closed = True
closed_date = <Date 2014-11-10.07:58:05.540>
closer = 'berker.peksag'
components = ['Library (Lib)']
creation = <Date 2014-06-03.10:11:31.342>
creator = 'Pavel.Kazlou'
dependencies = []
files = ['35468', '35631', '37110', '37119']
hgrepos = []
issue_num = 21650
keywords = ['patch']
message_count = 18.0
messages = ['219675', '219677', '219678', '219804', '219834', '219835', '219889', '219891', '219893', '220556', '230431', '230433', '230434', '230511', '230563', '230669', '230939', '230940']
nosy_count = 9.0
nosy_names = ['rhettinger', 'pitrou', 'ezio.melotti', 'r.david.murray', 'python-dev', 'berker.peksag', 'martin.panter', 'serhiy.storchaka', 'Pavel.Kazlou']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue21650'
versions = ['Python 3.5']

[PavelKazlou]

Currently when you use json.tool, fields are reordered alphabetically.
In source code the value of sort_keys is hardcoded to be true.
It should be easy to expose this option as command line parameter.

[PavelKazlou]

This is the line in module I'm talking about:
json.dump(obj, outfile, sort_keys=True, indent=4)

[berkerpeksag]

Here's a patch with a test case.

[pitrou]

I don't really understand the point of this. The "unsorted" output order will be unpredictable for the user (it isn't necessarily the same as the order of fields in the input data).

[bitdancer]

It should be possible to also change the tool to use OrderDicts, though.

[bitdancer]

Or does the data get decoded to a dict *before* it gets passed to the object_hook? Probably, in which case nevermind...

[PavelKazlou]

The idea is to keep the same order as in input.

[bitdancer]

Yes but the input is turned into a dict, and dicts do not preserve order. Further, what is passed to the object_hook is already a dict, so the order is already lost before object_hook is called.

Since the parser (or at least the Python version of the parser) first turns the input into a list of pairs, and a Python OrderedDict can be instantiated from a list of pairs, it would theoretically be possible to extend the object_hook API to allow an OrderedDict to be used on decode. Exactly how to do that so as to retain backward compatibility is a bit of a question.

So, it is *possible* to achieve your aim, but it isn't as simple as allowing sort= to be set False.

IMO being able to preserve the order of the input when desired (ie: use an OrderedDict object_hook) would be a nice feature to have.

[bitdancer]

Wait, I read the code wrong.

You can define object_pairs_hook, and use that to return an OrderedDict. So it should be possible to do this without changing the json module itself. This is actually documented as a place to use OrderedDict. Guess I should have read the docs instead of the code :)

[berkerpeksag]

Updated patch attached based on feedback from David. Thanks!

[rhettinger]

To me, "--unsorted" implies arbitrary ordering such as the order generated by regular dictionaries. I think the option should be "--order-preserving" or some such. The option name needs to make it clear that the output is in the same order an the input.

As side from the option name, I'm +1 on the patch. It is very annoying to look at scrambled JSON when the original presentation order made more logical sense.

[serhiy-storchaka]

May be make sorting keys not default?

[rhettinger]

Maybe make sorting keys not default?

If you mean preserve order by default, then yes that would be a nice default.

[berkerpeksag]

Thanks for the suggestions.

If you mean preserve order by default, then yes that would be a nice default.

issue21650_v3.diff implements this idea.

[serhiy-storchaka]

LGTM.

[rhettinger]

The patch looks good. One nit, the phrase "sorted by their key" has an odd ring to it and is mildly confusing, though technically correct. Perhaps, "sorted alphabetically by key" would be better for most folks.

[python-dev]

New changeset 58a871227e5b by Berker Peksag in branch 'default':
Issue bpo-21650: Add an --sort-keys option to json.tool CLI.
https://hg.python.org/cpython/rev/58a871227e5b

[berkerpeksag]

Thanks for the reviews.