Sort keys of objects

[jameschenjav]

The sorting flag from jq is very helpful when dealing with JSON APIs.

  -S               sort keys of objects on output;

There is an example:

❯ echo '{ "foo": 1, "bar": 2, "nested": { "will": 1, "sort": 2, "as": 3, "well": 4 } }' | jq -S .
{
  "bar": 2,
  "foo": 1,
  "nested": {
    "as": 3,
    "sort": 2,
    "well": 4,
    "will": 1
  }
}

Can we have this feature as well? Also, as a new rustacean I really appreciate your work. I had some issue to link lib-jq on my Mac, so this crate will save me a lot of time.

[yamafaktory]

Hi @jameschenjav and thanks for the positive feedback!

Basically jql keeps the insertion order because the preserve_order feature flag was introduced to avoid the default behavior of serde_json (#23, https://github.com/yamafaktory/jql/blob/main/Cargo.toml#L41).

Nobody complained about it so far... and I was the one introducing this (no issue).

So basically, we have two options:

1. remove the flag and then everything will be sorted auto-magically
2. introduce a sort flag

[jameschenjav]

Hi @yamafaktory

Thanks for your response.

It makes sense to keep original order by default. As it will be a breaking change, if someone had already built test-cases without proper comparison, the tests will break.

So imo better to have a new flag.

[yamafaktory]

A follow-up on this topic.

The current issue is that, as already mentioned, serde_json uses a flag to either internally process the data as a BTreeMap or an IndexMap (https://github.com/serde-rs/json/blob/master/src/map.rs#L30-L33). I haven't seen so far a simple way to bypass the flag limitation - basically being to switch from one to another data-structure. Also, to implement a custom sorting from scratch sounds like a poor idea as data can be deeply nested and performance will be pretty bad.

Still trying to investigate a bit. Let's keep this open for now.

[kmaasrud]

I've been investigating this a bit myself, and as you say, there is no way to conditionally toggle a crate feature, since features get applied at build-time.

However, I think Cargo can handle compiling two different versions of serde_json - one with the preserve_order feature and one without (and then simply naming them differently.) I can try this out in a PR if you want, and then we can test if there are any performance or bloat implications.

I personally think the best way is to put optional sorting behind a feature flag in jql. Feels less bloated. Though perhaps we can even hide the sort-flag double-dependency shenanigans behind a feature-flag as well?

[yamafaktory]

@kmaasrud I thought about that too but that would mean two different versions since this is a binary (this can works fine for a the library part of course). Folks that will install it will still get the default one.

[kmaasrud]

@yamafaktory, yes it seems like my idea is not possible.

The thought was to have two versions of serde_json compiled into the binary - one with the preserve_order feature and one without. That would allow us to do e.g. serde_json::from_str::<Value>(json_content) and serde_json_ordered::from_str::<Value>(json_content) based on a condition, where serde_json and serde_json_ordered are the same crate with different features. However, Cargo only allows multiple versions of the same crate if they are indeed DIFFERENT VERSIONS (e.g. 1.0.79 and 1.0.78.) That would be less than ideal - to put it mildly.

I say document the sorting behavior and then people can install their preferred version of jql with cargo install jql --features sorting, for example.

[yamafaktory]

That's an option indeed. My main concern is that it feels a bit awkward and might be even more weird regarding the packages distributed on different OS (not maintained here).

[yamafaktory]

Let's maybe keep this open for a bit longer and see if some trick might help fixing that.

[kmaasrud]

Let's maybe keep this open for a bit longer and see if some trick might help fixing that.

Sounds like a good idea 👍

[j-tesla]

I guess we can 2 have intermediate crates, one with serde json dep and other with serde json (preserved order) dep.

[yamafaktory]

@j-tesla since jql is a workspace now, this is presumably doable. However, this will double the size of this dependency which is not ideal. I'll have a look at #216 again, I've found a crate using it to keep the keys originally ordered but I need to test a bit (and this is using unsafe Rust).