Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazy-load compression filters in H5WasmProvider #1522

Merged
merged 1 commit into from
Nov 7, 2023
Merged

Lazy-load compression filters in H5WasmProvider #1522

merged 1 commit into from
Nov 7, 2023

Conversation

axelboc
Copy link
Contributor

@axelboc axelboc commented Nov 3, 2023

Thanks to the new h5wasm-plugins package (kudos @bmaranville!!), I'm able to load compression plugins on demand in H5WasmProvider. 馃帀

I was hoping that everything could be taken care of by H5WasmProvider under the hood, but since @h5web/h5wasm is a library, it's not that simple. The plugin files have to be loaded into Emscripten's file system as array buffers, which means they first have to be fetched lazily somehow ... and only the consumer application knows how to do that:

  • perhaps the application is bundled with Vite or webpack, which means the plugins can be imported as static assets (i.e. to get their file system URLs) ` and then fetched as array buffers;
  • perhaps the application doesn't use a bundler and just serves the plugin files at known URLs;
  • perhaps the application runs in an environment that prevents the use of fetch altogether (e.g. VS Code webview).

The solution I came up with is to expose a getPlugin prop. If the prop is implemented by the consumer app, then it will be called with the name of a known plugin when the time comes to load it into Emscripten. It is then up to the consumer application to return an array buffer (or undefined if the plugin is not supported).

H5WasmProvider holds a mapping of known plugins against their corresponding HDF5 filter IDs. For now, this list will need to be kept up to date as more plugins are added to h5wasm-plugins. In the future, we could add a prop to let consumers declare additional plugins without having to wait for H5Web updates.

This PR brings support for the following 7 compression filters in the h5wasm demo: Blosc, bzip2, LZ4, LZF, SZ, ZFP and Zstandard. You can try it with this file, which comes from the h5wasm-plugins repo: https://www.silx.org/pub/h5web/filters.h5

@axelboc axelboc marked this pull request as draft November 3, 2023 15:58
@axelboc axelboc force-pushed the filters branch 4 times, most recently from 9f82f4e to 02c6233 Compare November 6, 2023 09:13
@axelboc axelboc marked this pull request as ready for review November 6, 2023 09:13
apps/demo/src/h5wasm/H5WasmApp.tsx Show resolved Hide resolved
packages/h5wasm/src/h5wasm-api.ts Show resolved Hide resolved
packages/h5wasm/src/h5wasm-api.ts Show resolved Hide resolved
packages/h5wasm/src/h5wasm-api.ts Show resolved Hide resolved
packages/h5wasm/src/h5wasm-api.ts Outdated Show resolved Hide resolved
Copy link
Member

@loichuder loichuder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing 馃ぉ

packages/h5wasm/README.md Outdated Show resolved Hide resolved
packages/h5wasm/src/h5wasm-api.ts Show resolved Hide resolved
packages/h5wasm/src/h5wasm-api.ts Outdated Show resolved Hide resolved
packages/h5wasm/README.md Show resolved Hide resolved
@axelboc axelboc merged commit 8f38049 into main Nov 7, 2023
8 checks passed
@axelboc axelboc deleted the filters branch November 7, 2023 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants