Add scale factors used for MIP to multiscales metadata

[constantinpape]

The multiscales metadata currently does not contain any information about the downscaling factors used for generating the different scale datasets (MIP). This information is necessary for visualisation in a couple of tools, e.g. bigdataviewer. See the bdv.n5 format for an example on how to store it. As a workaround, we have been computing the factors on the fly based on the extent of the scale datasets/arrays.
However, there are some corner cases, e.g. if the extents are not divisible by the scale factors. So I think it would be good to add this as an (optional) field to the metadata.

We could either add it in a similar fashion to BDV and add a field downsamplingFactors = [[1, 1, 1], [2, 2, 2], [4, 4, 4], ...].
Or we could add it to the datasets field: "datasets": [{"path": "0", "downsamplingFactor": [1, 1, 1]}, {"path": "1", "downsamplingFactor": [2, 2, 2]}, ...].

[sbesson]

Thanks @constantinpape, at least from my sidem it definitely feels as useful top-level addition that would reduce the need to compute these value on-the-fly. My initial inclination would be to store these as datasets metadata but the two options you laid out are fairly equivalent.

The BDV downsampling specification makes use of the assumptions that a setup contains 4D arrays (T + XYZ) and downsamplingFactors can always be interpreted as scaling factor alongsize XYZ. Th same assumption does not hold for OME-Zarr especially with your introduction of axes in 0.3. Should the individual elements of downsamplingFactor[s] have the same dimensionality as axes to avoid any ambiguity?

[d-v-b]

Why does bigdataviewer require downsampling factors and not simply the grid spacing of each image? This seems like something that should be fixed on the bigdataviewer side instead of something this spec should support.

[constantinpape]

@sbesson

My initial inclination would be to store these as datasets metadata but the two options you laid out are fairly equivalent.

I agree, these are fairly equivalent and I don't have a strong preference either.

The same assumption does not hold for OME-Zarr especially with your introduction of axes in 0.3. Should the individual elements of downsamplingFactor[s] have the same dimensionality as axes to avoid any ambiguity?

Good point. We should demand that the length is the same as axes.

@d-v-b

Why does bigdataviewer require downsampling factors and not simply the grid spacing of each image? This seems like something that should be fixed on the bigdataviewer side instead of something this spec should support.

bigdataviewer requires the grid spacing of the image at the lowest scale and the downsampling factors for the other images. I assume that it computes the grid spacing for the other scales based on this information internally. There might also be a way to set the grid sampling for all scales without the need for the sampling factors, I am not familiar enough with the details to know.

I think this brings us back to the transformation discussion and shows that are two different options for this:

• Define a transformation for each scale (no need for downsampling factors)
• Define transformation only for lowest scale and give downsampling factors.

[d-v-b]

I think this brings us back to the transformation discussion and shows that are two different options for this:

Define a transformation for each scale (no need for downsampling factors)
Define transformation only for lowest scale and give downsampling factors.

I am 100% opposed to the second option. My mantra is "multiresolution datasets are just collections of datasets", which entails that there is nothing special about the highest or lowest resolution levels. They are all just images, and so they should all have the same kinds of spatial metadata.

Bigdataviewer should deprecate downsamplingFactors and just read the grid spacing for each image directly. The spec should not compensate for a strange decision on the bigdataviewer side.

[m-albert]

Hi guys, just read this thread and wanted to mention an aspect potentially important for designing multiscale metadata: Depending on the method used to calculate the differently scaled images, their translational offset can vary. E.g. a 2x2 binned image has a different offset (with respect to its full resolution image) compared to an image created by subsampling every second pixel.

Therefore it would make sense for this to be reflected in the multiscale metadata (at least optionally).

Also, acknowledging this fact that lower res images can result from a range of operations (binning, subsampling, etc.), indicates that using spacings as opposed to "downsamplingfactors" might be more precise and less ambiguous.

[d-v-b]

@m-albert that's a very good point, and it's another reason why downsamplingFactors is strictly worse than just expressing a spatial transformation. My approach for this is as you suggest -- I put the scale and translation parameters for each dataset in the multiscale metadata, e.g.

[bogovicj]

A little clarification: Bigdataviewer (the viewer) does not require downsamplingFactors.
That particular metadata dialect does require downsamplingFactors, as does N5 Viewer (the metadata dialect, not the viewer)

The fundamental thing for bdv is a list of affines, one per scale level, (see here for example). The first thing it does is to convert to an affine. Other similar implementations such as N5Source are a bit more explicit about this.

I'd rather we didn't use downsamplingFactors and specify scales / offsets in physical space for every scale level. We're probably stuck supporting this for legacy reasons, but I personally never write new datasets with this convention anymore.

[constantinpape]

Thanks for the clarification @bogovicj; I haven't dug into the BDV code so much myself and only looked at the "default" path, which instantiates the viewer from the metadata with downsamplingFactors. Given what you have pointed out it should be easy to instantiate it using a transformation for each scale instead.

I'd rather we didn't use downsamplingFactors and specify scales / offsets in physical space for every scale level. We're probably stuck supporting this for legacy reasons, but I personally never write new datasets with this convention anymore.

Ok, I think there is pretty strong support for not specifying downsamplingFactors and providing a trafo per scale instead. I agree that this solution is cleaner.
I will write an update in the main issue on transformations later and then close this one.