Add explicit sorting to query to assure results

[tomschenkjr]

Per Socrata SoDA documentation, the query should be ordered to deliver consistent results if the dataset is changed while the dataset is downloading.

[geneorama]

Another useful reference:
http://dev.socrata.com/docs/queries.html#the-limit-parameter

[marks]

@tomschenkjr Any thoughts on implementing this? I have run into an issue that appears to be sorting-related. One can add a sort to the URL but it'd be useful, IMHO, to have a default sort applied if one is not specified.

[tomschenkjr]

Yeah, we can target this and make it the top priority. It was always a theoretical issue but now it’s impacting, can push a fix.

/cc @geneorama

From: Mark Silverberg [mailto:notifications@github.com]
Sent: Wednesday, May 18, 2016 7:19 AM
To: Chicago/RSocrata RSocrata@noreply.github.com
Cc: Schenk, Tom Tom.Schenk@cityofchicago.org; Mention mention@noreply.github.com
Subject: Re: [Chicago/RSocrata] Add explicit sorting to query to assure results (#15)

@tomschenkjrhttps://github.com/tomschenkjr Any thoughts on implementing this? I have run into an issue that appears to be sorting-related. One can add a sort to the URL but it'd be useful, IMHO, to have a default sort applied if one is not specified.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com//issues/15#issuecomment-220009097

---

This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail (or the person responsible for delivering this document to the intended recipient), you are hereby notified that any dissemination, distribution, printing or copying of this e-mail, and any attachment thereto, is strictly prohibited. If you have received this e-mail in error, please respond to the individual sending the message, and permanently delete the original and any copy of any e-mail and printout thereof.

[geneorama]

@marks I'm wondering what issue you found.

The only reason I suggested the "order" parameter was to ensure reliability for paged queries (queries that require more than one call). Even for paged queries I thought that the only reason the order was important was if rows were inserted in between calls or if Socrata changed the sort of a data set on the back end, both of which I considered to be unlikely and unreproducible events.

[marks]

@geneorama the issue I found is with a private dataset and I'm having trouble reproducing it anywhere else.. so I dont have much to share but Socrata engineers are telling me a default sort would be a good practice. As you all probably now, there is a "system field" called :id as well as :created_at and :updated_at which may be suitable for default sorts

[geneorama]

Using :id seems like a good idea, and I had forgotten about it. I remember discussing it with @tomschenkjr, and I think I had been using it in my own work. For some reason I didn't end up using it, but I don't remember why.

One issue is that it was only available for .json downloads, maybe that was it?
Maybe it wasn't available for all data sets?

Maybe @tomschenkjr will remember more detail.