[25.0] Add SpatialData datatype

[nilchia]

I'm trying to add spatialdata as a new data type in Galaxy.
spatialdata is a data framework that comprises a FAIR storage format and a collection of python libraries for performant access, alignment, and processing of uni- and multi-modal spatial omics datasets.

The metadata should be sth like this:

SpatialData object, with associated Zarr store: /Users/macbook/embl/projects/basel/spatialdata-sandbox/mouse_liver/data.zarr
├── Images
│     └── 'raw_image': DataTree[cyx] (1, 6432, 6432), (1, 1608, 1608)
├── Labels
│     └── 'segmentation_mask': DataArray[yx] (6432, 6432)
├── Points
│     └── 'transcripts': DataFrame with shape: (<Delayed>, 3) (2D points)
├── Shapes
│     └── 'nucleus_boundaries': GeoDataFrame shape: (3375, 1) (2D shapes)
└── Tables
      └── 'table': AnnData (3375, 99)
with coordinate systems:
    ▸ 'global', with elements:
        raw_image (Images), segmentation_mask (Labels), transcripts (Points), nucleus_boundaries (Shapes)

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[nilchia]

currently when I upload the file, it is detected as zarr.zip and not spatialdata

[nilchia]

The sniffer works now.
I'm trying to see if it is possible to also show the metadata like this:

SpatialData object, with associated Zarr store: /Users/macbook/embl/projects/basel/spatialdata-sandbox/mouse_liver/data.zarr
├── Images
│     └── 'raw_image': DataTree[cyx] (1, 6432, 6432), (1, 1608, 1608)
├── Labels
│     └── 'segmentation_mask': DataArray[yx] (6432, 6432)
├── Points
│     └── 'transcripts': DataFrame with shape: (<Delayed>, 3) (2D points)
├── Shapes
│     └── 'nucleus_boundaries': GeoDataFrame shape: (3375, 1) (2D shapes)
└── Tables
      └── 'table': AnnData (3375, 99)
with coordinate systems:
    ▸ 'global', with elements:
        raw_image (Images), segmentation_mask (Labels), transcripts (Points), nucleus_boundaries (Shapes)

[nilchia]

It shows a simple metadata like this now:

I don't think the metadata should go deeper

[nilchia]

[nilchia]

FAILED test/integration/test_datatype_upload.py::test_upload_datatype_auto[OMEzarrImages.ome_zarr.zip] - AssertionError: assert 'zarr.zip' == 'ome_zarr.zip'
  - ome_zarr.zip
  ? ----
  + zarr.zip

the ome_zarr is somehow detected as zarr.zip

[davelopez]

Looks good, just a question about metadata usage below:

[davelopez]

Question: Is all this metadata used by any existing or future tool?
I think we want to keep metadata as minimal as possible since this is information that goes into the database for each dataset. So it is important to keep only the essential metadata.

[nilchia]

The metadata is really needed for the users to inspect the info in the data.
I tried to minimize it here:

is it good enough?

[davelopez]

If it is needed, then it is needed :)
I just wanted to make sure.

[github-actions]

This PR was merged without a "kind/" label, please correct.

[LucaMarconato]

Thanks @nilchia and @davelopez!

One comment: since last week's release, spatialdata supports the Zarr v3 format. Mainly in Zarr v3 there is no .zattrs anymore, and instead zarr.json is used. Also, the OME-Zarr metadata is slightly different (see the new "attributes": "ome" keys in the example below):

Example zarr.json for the rasterized image for this merfish dataset:

Reveal the zarr.json

{
  "attributes": {
    "ome": {
      "omero": {
        "channels": [
          {
            "label": 0
          }
        ]
      },
      "version": "0.5-dev-spatialdata",
      "multiscales": [
        {
          "datasets": [
            {
              "path": "0",
              "coordinateTransformations": [
                {
                  "type": "scale",
                  "scale": [
                    1.0,
                    1.0,
                    1.0
                  ]
                }
              ]
            }
          ],
          "name": "/images/rasterized",
          "axes": [
            {
              "name": "c",
              "type": "channel"
            },
            {
              "name": "y",
              "type": "space"
            },
            {
              "name": "x",
              "type": "space"
            }
          ],
          "coordinateTransformations": [
            {
              "type": "sequence",
              "transformations": [
                {
                  "type": "scale",
                  "scale": [
                    1.0,
                    3.863113519548272,
                    3.5077751335606355
                  ],
                  "input": {
                    "name": "cyx",
                    "axes": [
                      {
                        "name": "c",
                        "type": "channel"
                      },
                      {
                        "name": "y",
                        "type": "space",
                        "unit": "unit"
                      },
                      {
                        "name": "x",
                        "type": "space",
                        "unit": "unit"
                      }
                    ]
                  },
                  "output": {
                    "name": "global",
                    "axes": [
                      {
                        "name": "c",
                        "type": "channel"
                      },
                      {
                        "name": "y",
                        "type": "space",
                        "unit": "unit"
                      },
                      {
                        "name": "x",
                        "type": "space",
                        "unit": "unit"
                      }
                    ]
                  }
                },
                {
                  "type": "translation",
                  "translation": [
                    0.0,
                    4548.0,
                    1154.0
                  ],
                  "input": {
                    "name": "cyx",
                    "axes": [
                      {
                        "name": "c",
                        "type": "channel"
                      },
                      {
                        "name": "y",
                        "type": "space",
                        "unit": "unit"
                      },
                      {
                        "name": "x",
                        "type": "space",
                        "unit": "unit"
                      }
                    ]
                  },
                  "output": {
                    "name": "global",
                    "axes": [
                      {
                        "name": "c",
                        "type": "channel"
                      },
                      {
                        "name": "y",
                        "type": "space",
                        "unit": "unit"
                      },
                      {
                        "name": "x",
                        "type": "space",
                        "unit": "unit"
                      }
                    ]
                  }
                }
              ],
              "input": {
                "name": "cyx",
                "axes": [
                  {
                    "name": "c",
                    "type": "channel"
                  },
                  {
                    "name": "y",
                    "type": "space",
                    "unit": "unit"
                  },
                  {
                    "name": "x",
                    "type": "space",
                    "unit": "unit"
                  }
                ]
              },
              "output": {
                "name": "global",
                "axes": [
                  {
                    "name": "c",
                    "type": "channel"
                  },
                  {
                    "name": "y",
                    "type": "space",
                    "unit": "unit"
                  },
                  {
                    "name": "x",
                    "type": "space",
                    "unit": "unit"
                  }
                ]
              }
            }
          ]
        }
      ]
    },
    "spatialdata_attrs": {
      "version": "0.3"
    }
  },
  "zarr_format": 3,
  "consolidated_metadata": null,
  "node_type": "group"
}                                                                                                                           

[nilchia]

Thanks @LucaMarconato for the info, I made a new PR here