-
Notifications
You must be signed in to change notification settings - Fork 67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Handle sorted vs unsorted inputs #501
Comments
For future reference, @nwt added some ideas about the implementation of sortedness analysis here: #869 (comment) |
The analysis mentioned above will likely happen in the query planner, though I don't know if it will happen in this initial epic for it: #1004 |
Closing this since we decided zq doesn't try to take advantage of sorted data in any way and such optimization will all be handled at scale in the lake. Instead, zq just processes the values in order of the files. |
zq
currently has some half-baked pieces for dealing with sorted inputs:This is working well enough right now in the app since
all.bzng
is always sorted by descending timestamps during ingest and the app always passes"dir":-1
in its queries (incidentally, we should remove that from the zqd api since any other value will produce incorrect results).The zql spec includes ordering hints which are directives that can be inserted in a (b)zng file to indicate if/how the contents of the file/stream are sorted. This issue is to generalize the current "Reverse" flag into a more complete solution that handles streams that can be in one of 3 states: sorted ascending, sorted descending, or unsorted. Note that this property might be different at different points in a query graph (e.g., if points are sorted by timestamp and then they enter a
sort foo
proc, they are no longer sorted by timestamp). For now, it is probably sufficient to just implement this for the timestamp field and ignore zng sorting hints if they indicate sorting by any other field.When this was discussed in the past, there was some disagreement about exactly how to implement it. The controversial aspect was how to handle procs that might change the sorting order as described above.
@aswan impelemented a solution in which sorting information was communicated from readers to procs and between procs dynamically at runtime. @nwt, @henridf, and @mccanne all disliked this and proposed a different solution: the introduction of a separate static analysis phase that would analyze an entire proc graph and determine the sorting properties at each point in the graph. It is unclear how this would work when the sortedness of a (b)zng input is unknown until the stream has been read.
The text was updated successfully, but these errors were encountered: