Big data file problem

[ManosBrsls]

Dear Sir/Madam,

I am trying to open a .h5 file using the h5wasm library.
The file contains a Float32Array with a shape of [44000, 3000], totaling 132,000,000 values.

It appears that the h5wasm library is unable to handle such a large dataset, as it fills the array with zeros instead of the actual values.

Kind regards,

Manos.

[bmaranville]

Do you know if there is a filter (compression) applied to the data? If so, which one (and was the accompanying plugin loaded to h5wasm before opening the dataset?)

[bmaranville]

Also, I realize it's probably fairly big, but do you have a way of sharing an example file with me that shows the problem? My email address is public on my profile page: https://github.com/bmaranville

[ManosBrsls]

Do you know if there is a filter (compression) applied to the data? If so, which one (and was the accompanying plugin loaded to h5wasm before opening the dataset?)

I’m quite sure that no filter or compression has been applied to the data. I was able to successfully read data from a smaller array with the same structure, and the issue only occurs with the larger array, as I mentioned earlier.

Unfortunately i cannot share an example file with you for confidentiality reasons...

It’s also worth noting that I was able to load the same file successfully using a Python library (h5py and matplotlib), which confirms that the file itself is not corrupted.

Kind regards,

Manos.

[bmaranville]

I am not able to reproduce your error... if I create an HDF5 file with a large array like this:

import h5py
import numpy as np

with h5py.File("big.h5", "w") as output:
    data = np.arange(44000*3000, dtype=np.float32)
    output.create_dataset("data", data=data.reshape((44000,3000)))

I get a file 504 MB in size. If I open it with h5wasm in a browser, I am able to see the contents:

//... after importing the HDF5 bytes and loading the file into an h5wasm.File object named 'f'
f.get('data').shape
// (2) [44000, 3000]
f.get('data').value
// Float32Array(132000000) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, …]

I can load the dataset in myHDF5 in Firefox and visualize it with the line visualization, since the Heatmap has the problem you saw above (texture too big), so the data is loading properly in h5wasm. Do you know how the data gets pulled out of h5wasm and inserted in the View2dFloat32 object in your console output above?

[ManosBrsls]

Thank you for your effort!!
i am sending you the code i am using to read data from the file i have.

` await h5wasmReady;

  const filePath = `/tmp/${filename}`;
  const data = new Uint8Array(buffer);
  FS.writeFile(filePath, data);
  // Now open the file using its path
  const h5File = new File(filePath, "r");

  const pointsDataset = h5File.get("spectrumPoints");
  const spectrumPoints = ndarray(pointsDataset.value, pointsDataset.shape);`

The file where spectrumPoints is stored is over 900 MB in size.

Sorry if my answers aren’t very clear — I’m still new to programming.

[bmaranville]

In your code, do you think you could add this line and post the results from your console, here, to help me diagnose the issue?

// ... your existing code above
const pointsDataset = h5File.get("spectrumPoints");
console.log("metadata:", JSON.stringify(pointsDataset.metadata, null, 2));
// ... the rest of your code

For my test dataset, it looks like this:

metadata: {
  "signed": false,
  "type": 1,
  "vlen": false,
  "littleEndian": true,
  "size": 4,
  "total_size": 132000000,
  "shape": [
    44000,
    3000
  ],
  "maxshape": [
    44000,
    3000
  ],
  "chunks": null
}

[ManosBrsls]

I run the code you suggested. And i got this:

I am also sending you the errors when this line of code is executed :
const spectrumPoints = ndarray(pointsDataset.value, pointsDataset.shape);

[bmaranville]

I can see you're using an older version of h5wasm from the version of HDF5 in the trace: 1.14.2. More recent versions of h5wasm use libhdf5 1.14.6

Can you try upgrading h5wasm to the latest version and repeat the test?

[ManosBrsls]

I update to the latest version. And this is the results after repeating the test.

[bmaranville]

Thank you for the extra info - it was very helpful. I think the issue is related to the chunk size being used - a chunk of [1, 100] is very small for such a large dataset, and it means that more than 1e6 chunks have to be allocated and tracked (there is a hash table involved somewhere).

I tried writing a file with that chunk size and was able to reproduce your error. If I write the same data with an automatic chunk size (using chunks=True in h5py) it uses a chunk size of [688, 47] and I am able to read the values just fine with that chunk size.

The memory limits in the browser are probably lower than in a python application, which is why you were able to open it with h5py. I ran a quick test, in python (using h5py)

import h5py
f = h5py.File("big.h5", "r")
d = f["data"][()]

This python process used about 5.5 GB of RAM for a chunk size of [1, 100]
With the automatic chunk size of [688, 47] the same operation used 0.6 GB of RAM.

So if you can, it would be good to write the data with a different chunk size. Otherwise I think you will exceed the memory allocated to browser javascript/WASM environments (though this varies from browser to browser and OS to OS, you're probably better off using less RAM anyway) and you might run out of memory in Python eventually too.

[bmaranville]

If you are unable to change the chunk size when writing this data (if you are just a consumer of this data) I would like to point out that you could also slice the data, doing smaller reads instead of reading the whole dataset at once. For instance you could iterate over the first dimension by doing

for (let i=0; i<pointsDataset.shape[0]; i++) {
  // leaving the second array empty in the slice call means use everything along that dimension,
  const data_slice = pointsDataset.slice([[i, i+1], []]);
  // to slice along the second dimension you would do .slice([[], [i, i+1]]) with an upper limit of pointsDataset.shape[1]
  // ... do work with data_slice, maybe populating your ndarray
}

[ManosBrsls]

Thank you very much for your help. I will try to apply your suggested solution.

Kind regards,

Manos.