Another specialisation for select - select with datasets. #101
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Ok, why would anyone need this. This is intended to decrease speed to first result in mquery. Instead of doing one big query, mquery will query datasets one by one. Now I can hear you asking: WHY would the logic for this be in mquery? Isn't async querying a pretty generic use-case? So I agree, but implementing this in mquery opens a can of worms that I'm not ready to handle, for example: - Database can change during the query (solution: persistent snapshots?) - Database can't know when it's OK to stop streaming (for example, user cancelled a query, or yara can't catch up). - It makes database performance even more opaque - is it idle? Or are there tasks busy in the background? Because of this, I think that querying DSets one by one is a good solution for now. The proper solution would be something like persistent snapshots + continuable iterators... But yeah, why use a cannon when small PR like this will do the trick. Bonus points: we can run queries in parallel!
msm-code
force-pushed
the
feature/select-with-datasets
branch
from
01a942d
to
b813d0a
Compare
Closed
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ok, why would anyone need this. This is intended to decrease speed to first
result in mquery. Instead of doing one big query, mquery will query datasets
one by one.
Now I can hear you asking: WHY would the logic for this be in mquery? Isn't
async querying a pretty generic use-case? So I agree, but implementing this
in mquery opens a can of worms that I'm not ready to handle, for example:
cancelled a query, or yara can't catch up).
tasks busy in the background?
Because of this, I think that querying DSets one by one is a good solution for
now. The proper solution would be something like persistent snapshots +
continuable iterators... But yeah, why use a cannon when small PR like this
will do the trick.
Bonus points: we can run queries in parallel!