-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve fetching for large datasets #616
Comments
I'd add that, when performing long downloads and computations, the UI should:
|
Our discussion on #632 also gave me an idea: we could make the flattening operation more consistent by encapsulating in the Edit: This was done in #661 It then becomes relevant to this issue as it would be a stepping stone to request the flattening on the back-end. Thus, avoiding another expensive computation in h5web. |
In the providers' |
#635 implements cancellation on the front-end, but it doesn't resolve crashes on Bosquet when attempting to fetch (and cancel the fetch of) extremely large datasets. |
#640 implements retrying after cancelling (including evicting cancellation errors from the value store's cache). |
Just curious - for HSDS, have you tried using HTTP Compression? That should reduce the payload size considerably. |
Unfortunately, the impact will be limited as most of our heavy datasets are not compatible with HSDS due to HDFGroup/hsds#76 😕 But this is something that we still need to try ! |
Binary is now used with H5Grove when getting dataset values: #817 |
The auto-scale-off feature in the Maybe, it is time to review it ? We could
|
#877 implemented an intemediate solution:
|
It seems that h5wasm now (as of v0.4.8) supports lazy loading of arrays. For reference, see this discussion: usnistgov/h5wasm#40 |
Sure, that's relevant also for large datasets. For h5wasm, we have a more specific issue tracking this at #1264 |
Is it also planned to have streaming binary support for hsds? I experienced problems with this while I was experimenting with storing and loading large datasets via hsds and I use h5web in a simple hsds directory browser to view the stored data. However, the hsds server gets stuck on large datasets because h5web requests the data in json format. |
To be honest, we don't really plan to improve the HSDS part since we mostly use h5grove and h5wasm. But you are welcome to contribute and we will be happy to help you doing so. If you have some working code, feel free to open a draft PR to discuss. If something blocks you, you can drop us a line at |
Having played around with large datasets (~10M of points but a slice is ~100k), I find that the long fetch of the data breaks the flow...
Let's make use of this issue to gather possible improvements:
ProviderApi
in its own file and rename providers' API files #657 Abstract values fetching out of mapped vis components #658 Fix fetching waterfall after retrying a NeXus vis #659The text was updated successfully, but these errors were encountered: