Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/json: no way to preserve the order of map keys #27179

Open
lavalamp opened this issue Aug 23, 2018 · 13 comments

Comments

@lavalamp
Copy link

@lavalamp lavalamp commented Aug 23, 2018

The encoding/json library apparently offers no way to decode data into a interface{} without scrambling the order of the keys in objects. Generally this isn't a correctness issue (per the spec the order isn't important), but it is super annoying to humans who may look at JSON data before and after it's been processed by a go program.

The most popular yaml library solves the problem like this: https://godoc.org/gopkg.in/yaml.v2#MapSlice

@ALTree

This comment has been minimized.

Copy link
Member

@ALTree ALTree commented Aug 23, 2018

This was asked in the past. Last week: #27050, and an old issue: #6244, where brad wrote:

JSON objects are unordered, just like Go maps. If you're depending on the order that a specific implementation serializes your JSON objects in, you have a bug.

While it's unlikely that the current behaviour of the json package will change, a proposal to add some new mechanism that preserves order (similar to the one in the yaml package) may be accepted. It would help, though, to have something more concrete to look at, i.e. a complete, full fledged proposal explaining what the new piece of API would look like.

@lavalamp

This comment has been minimized.

Copy link
Author

@lavalamp lavalamp commented Aug 23, 2018

My search failed to turn up either of those issues, thanks.

I tried to make it clear that this is not about correctness, it's about being nice to human readers. If that's not a relevant concern then we can close this. If there is some chance of a change being accepted then I can make a proposal.

@ALTree

This comment has been minimized.

Copy link
Member

@ALTree ALTree commented Aug 23, 2018

If there is some chance of a change being accepted then I can make a proposal.

I don't know if there is but the package owners can probably tell you: cc @rsc @dsnet @bradfitz

@ALTree ALTree added this to the Go1.12 milestone Aug 23, 2018
@mvdan

This comment has been minimized.

Copy link
Member

@mvdan mvdan commented Aug 23, 2018

There is a bit of precedent to making things nicer for human readers. For example, fmt sorts maps when printing them, to avoid random ordering in the output.

The fmt and json packages are different, and sorting keys isn't the same as keeping the original order, but I think the purpose is similar. Making the output easy to read or modify by humans.

I imagine there's no way to do this under the hood or by simply adding extra API like Encoder.SetIndent, as native maps simply don't keep the order.

Adding a type like MapSlice sounds fine to me - I'd say file a proposal unless someone else comes up with a better idea in the next few days. There's the question of whether the proposal will get past "publish this as a third-party package", but otherwise it seems like a sound idea.

@mvdan

This comment has been minimized.

Copy link
Member

@mvdan mvdan commented Aug 23, 2018

Similar proposal in the past - keeping the order of the headers in net/http: #24375

Just like in this case, the big question was where to store the actual order of the map keys.

@mvdan mvdan changed the title encoding/json: no way to preserve order encoding/json: no way to preserve the order of map keys Aug 23, 2018
@dsnet

This comment has been minimized.

Copy link
Member

@dsnet dsnet commented Aug 23, 2018

Strictly speaking, you can do this today using the raw tokenizer, that json.Decoder provides. That said, I think there will be comprehensive review of all json proposals and issues in the Go1.12 cycle. I can imagine solutions for this issue that also address problems of not being able to detect duplicate keys in objects.

@lavalamp

This comment has been minimized.

Copy link
Author

@lavalamp lavalamp commented Aug 23, 2018

I think the proposal would look like a flag (.SetPreserveOrder()) on the decoder that makes a MapSlice instead of a map[string]interface{}, plus making the encoder handle that type.

@dsnet Yeah, that solves the input half of the problem, but it's very inconvenient.

@dsnet

This comment has been minimized.

Copy link
Member

@dsnet dsnet commented Aug 23, 2018

Alternatively, it could output as [][2]interface{}, in which case you won't need to declare a new type in the json package.

@lavalamp

This comment has been minimized.

Copy link
Author

@lavalamp lavalamp commented Aug 23, 2018

I actually like that a lot. It should probably still be declared in the json package just for documentation purposes.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Sep 26, 2018

For now it seems like the best answer is a custom type (maybe a slice of pairs) with an UnmarshalJSON method that in turn uses the tokenizer.

@rsc rsc modified the milestones: Go1.12, Go1.13 Nov 14, 2018
@rsc rsc added the FeatureRequest label Nov 14, 2018
@Zamiell

This comment has been minimized.

Copy link

@Zamiell Zamiell commented Apr 20, 2019

For the people who stumble upon this issue from Google, the following two libraries (pick one) can help you if you need an ordered JSON map:

https://gitlab.com/c0b/go-ordered-json
https://github.com/iancoleman/orderedmap

Also, for reference, see this common StackOverflow answer:
https://stackoverflow.com/questions/25182923/serialize-a-map-using-a-specific-order

Of course, it would be fantastic if this were eventually part of the standard library, so I'll eagerly await a proposal from someone more proficient than I.

@roshangade

This comment has been minimized.

Copy link

@roshangade roshangade commented Sep 4, 2019

As per JSON specification, order does not matter.
Reference: https://tools.ietf.org/html/rfc7159#section-1

Other technologies, which use JSON, where order matters.
Example: GraphQL
Reference: https://graphql.github.io/graphql-spec/June2018/#sec-Serialized-Map-Ordering

It's a much-needed enhancement.

@astleychen

This comment has been minimized.

Copy link

@astleychen astleychen commented Oct 2, 2019

Feature needed to simplify our implementation on ordered serialization/deserialization.

@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.