Improve fetching speed by allowing for including more of records#809
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Right now, when records are fetched from the server, only the bare minimum is fetched, with the rest coming on-demand as you access fields of the record. This works in general, however if the user wants to download entire records (+ children), the result is a very large number of small requests.
This PR improves this. There are three main parts:
Being able to say
include=['**'], which will result in the entire record being downloaded, including childrenTo handle fetching children, the children of multiple parent records are fetched together. For example, singlepoint records from the trajectories of multiple optimization records are batched together.
A time-based chunking iterator to help prevent very large requests from choking the server. This also includes a thread pool for background fetching
This requires a few server side changes, so unfortunately this cannot be tested against the running production servers. I am planning on a release in early April, though.
Changelog description
Improve fetching speed by allowing for including more of records
Status