Big data file

[ManosBrsls]

Is it possible to load a .h5 file of 1gb and plot its data after some modifications on a Heatmap?
I am trying to do it but i receive this type of error {HDF5-DIAG: Error detected in HDF5 (1.14.2) thread 0 hydration error info.ts: 72

#000: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5D.c line 1061 in H5Dread(): can't synchronously read data
major: Dataset
minor: Read failed

Sorry if the question sounds stupid but i am new to programming and i am trying to learn.

Kind regards,

Manos.

[loichuder]

Hello,

I guess you tried to open it via https://myhdf5.hdfgroup.org/ ? A 1Gb file should be displayed with this without issue.

If you get an error, I think it comes from somewhere else. How are you creating this file ?

[ManosBrsls]

Hello again, and thank you for your response.

I found a way to downsample my data before plotting, which resolved the error:
THREE.WebGLRenderer: Image in DataTexture is too big (22000x3000).

However, the main issue still remains — I’m not able to plot the correct data.
I discovered that I'm trying to read from a Float32Array with shape [44000, 3000], which contains a total of 132,000,000 values.

It seems that the h5wasm library can't handle such a large amount of data and ends up filling the array with zeros instead of the actual values. And so the plot is incorrect.

Kind regards,

Manos

[loichuder]

If the problem lies in writing data with h5wasm, I would invite you to create an issue on the h5wasm repository. Brian is usually quick to answer.

Once the writing is fixed, don't hesitate to tell us if there is still something wrong with the display in H5Web

[ManosBrsls]

Thank you very much for your time and for yor help!!
I will create an issue in the h5wasm repository.

Kind regards,

Manos.

[axelboc]

Issue for reference: usnistgov/h5wasm#100 — so it seems to be caused by a very large number of small chunks.

[ManosBrsls]

Yes, that's right. Thank you very much for your help.