References support

[bmaranville]

Adding support for

• Creating object references (to a Dataset or Group)
• Creating region references to a dataset (similar to a slice)
• Writing object or region references to a Dataset (there are a special dtypes for references)
• Reading object or region references from a Dataset in an HDF5 file (they become instances of Reference or RegionReference)
• Dereferencing a reference, which returns the Group or Dataset it points to
• Retrieving the data from the region defined in a RegionReference (the reference contains information identifying the dataset it refers to, but must be dereferenced from the same File object as the source Dataset)

Examples:

Creating references, and writing references to datasets:

const VALUES = [12,11,10,9,8,7,6,5,4,3,2,1];
const DATA = new Float32Array(VALUES);
const SHAPE = [4,3];
const DATASET_GROUP = "entry";
const DATASET_NAME = "data";
const REFS_GROUP = "refs";
const OBJECT_REF_DATASET_NAME = "object_refs";
const REGION_REF_DATASET_NAME = "dset_region_refs";
const REGION_REF_DATA_0 = [[11.], [ 8.], [ 5.]];
const REGION_REF_DATA_1 = [[12., 11., 10.]];
const write_file = new h5wasm.File(FILEPATH, "w");

write_file.create_group(DATASET_GROUP);
const dataset_group = write_file.get(DATASET_GROUP);
dataset_group.create_dataset({name: DATASET_NAME, data: DATA, shape: SHAPE});

const object_refs = [
  dataset_group.create_reference(),
  dataset_group.get(DATASET_NAME).create_reference(),
];

write_file.create_group(REFS_GROUP);
const refs_group = write_file.get(REFS_GROUP);
refs_group.create_dataset({name: OBJECT_REF_DATASET_NAME, data: object_refs});

const dataset = dataset_group.get(DATASET_NAME);
const region_refs = [
  dataset.create_region_reference([[0,3], [1,2]]),
  dataset.create_region_reference([[0,1], []]),
]
refs_group.create_dataset({name: REGION_REF_DATASET_NAME, data: region_refs});

Reading references from a dataset, and dereferencing them:

const read_file = new h5wasm.File(FILEPATH, "r");

const dataset_group = read_file.get(DATASET_GROUP);
assert(dataset_group instanceof h5wasm.Group);

const refs_group = read_file.get(REFS_GROUP);
assert(refs_group instanceof h5wasm.Group);

const object_refs = refs_group.get(OBJECT_REF_DATASET_NAME).value;
const [obj_0, obj_1] = object_refs.map((ref) => read_file.dereference(ref));
assert(obj_0 instanceof h5wasm.Group);
assert.strictEqual(obj_0.path, `/${DATASET_GROUP}`);
assert(obj_1 instanceof h5wasm.Dataset);
assert.strictEqual(obj_1.path, `/${DATASET_GROUP}/${DATASET_NAME}`);

const region_refs = refs_group.get(REGION_REF_DATASET_NAME).value;
const [region_0, region_1] = region_refs.map((ref) => read_file.dereference(ref));
assert(region_0 instanceof h5wasm.DatasetRegion);
assert.deepEqual(region_0.value, new Float32Array(REGION_REF_DATA_0.flat()));
assert(region_1 instanceof h5wasm.DatasetRegion);
assert.deepEqual(region_1.value, new Float32Array(REGION_REF_DATA_1.flat()));
// assert.deepEqual(hard_link_dataset.value, DATA);

[bmaranville]

@axelboc is this going to mess up the H5WasmProvider? Do you think you could review?

[axelboc]

@axelboc is this going to mess up the H5WasmProvider? Do you think you could review?

I'm not spotting any potential breaking changes. Should be easy enough to at least add basic metadata support in H5WasmProvider 😊