Clearly seperate different concerns.

[mpvginde]

Currently mxalign does 4 things

• Definition of Traits along 3 axes (space, time, uncertainty)
• Define specific loaders to conform to the traints
• Provide a python/xarray-accessor api to align different dataset (see example.ipynb)
• Provide a CLI for full alignment pipelines (see advanced.ipynb)

Functionality 1 is now already extracted from mxalign by mlwp-data-specs.

Functionality 2 and 3 form the core of mxalign.

Functionality 4 could potentially also be separated.

[leifdenby]

Thanks for putting these up!

I would reorder steps 1 and 2 in the steps you have listed above. To me it would be natural to:

1. Load data into a single xr.Dataset from source netcdf files, zarr dataset, etc. Here one could provide a user-defined "loader" (as you have now) that restructures xr.Datasets from separate source files (preprocess), combines (with concat_dim) and modifies the resulting xr.Dataset to set global attributes, coords etc (postprocess)
2. Validate that this xr.Dataset confirms to the traits
3. Do transformations on the validated xr.Dataset to "align" on the different traits (e.g. interpolate spatially gridded data to discrete geographical localtions)

To enable the use of mlwp-data-specs on a loaded dataset I would put the spec validation first, which we agreed on in our chat just now :) We also decide that it would be better if the loaders reside in their own package (called mlwp-data-loaders for now), because these are likely to evolve rapidly (because upstream sources of data will evolve quickly) while the trait specs (hopefully) will only evolve slowly (because the structure that mxalign will depend on wont have to change unless we find something is missing in this spec).

The idea would be that people can keep using old revisions of mxalign and mlwp-data-loaders since they will refer to traits that are defined in revisions of mlwp-data-specs. Once we find that in mxalign we want to add functionality that would require new properties of the datasets we can then alter the spec, and thereafter alter the loaders to that they load in the datasets in a way that allows mxalign to carry out the operations for this new functionality.

So the processing steps will be as follows:

Alignment Pipeline Diagram

flowchart TD
    I["anemoi-inference"]
    J["anemoi-datasets"]
    S1[("Source dataset 1")]
    S2[("Source dataset 2")]
    S3[("Source dataset 3")]
    UL["User-provided Loader"]
    subgraph ML["mlwp-data-loaders"]
        subgraph A["1. Use source-specific loader"]
            L1["Loader for anemoi-inference"]
            L2["Loader for anemoi-datasets"]
        end
    end
    U[["xr.Dataset<br/>not yet validated"]]
    subgraph M["mlwp-data-specs"]
        B["2. Validate Traits<br/>Axes: space, time, uncertainty"]
    end
    X[["xr.Dataset<br/>conforming to MLWP data specs"]]
    subgraph MX["mxalign"]
        C["3. Provide a Python / xarray-accessor API<br/>to align different datasets<br/><br/>Example: `examples/introduction.ipynb`"]
        D["4. Provide a CLI for full alignment pipelines<br/><br/>Example: `examples/advanced.ipynb`"]
    end

    I --> S1
    J --> S2
    S3 --> UL
    S1 --> L1
    S2 --> L2
    L1 --> U
    L2 --> U
    UL --> U
    U --> B
    B --> X --> C --> D

Loading

[leifdenby]

Ok, here's an update:

• I've created a proof-of-concept implementation of mlwp-data-loaders: https://github.com/mlwp-tools/mlwp-data-loaders. It would be great to have your thought on the README and general structure to see if you think I've hit the scope correctly
• I've uploaded cerra-rr-an-oper-0001-mars-5p5km-2017-2017-6h-v3-testing.zarr to a S3 bucket on EWC and created a test for this loader in mlwp-data-loaders. I'm tracking this here: Add test for anemoi-datasets loader using CERRA test dataset hosted on EWC mlwp-data-loaders#1. The test is currently failing, which is good, means it is working :)
• I think it would be nice to have a test dataset for the anemoi-inference loader too. Do you have a small (say <10MB) dataset I could upload?
• I've decided not to release v0.1.0 of mlwp-data-specs yet since I think it makes more sense to wait until we have loaders implemented for anemoi-datasets and anemoi-inference that we have tests to ensure actually work. Does that make sense to you? Shouldn't be too much work since most of the functionality is already in mxalign

I am working on getting a tenancy on EWC through my UWC ML Liaison work, for now I am using the MLCast tenancy, but I think that is ok as long as we keep the datasets small :)

What are your plans wrt refactoring mxalign to make use of mlwp-data-specs and mlwp-data-loaders? Should we wait until v0.1.0 of mlwp-data-specs and mlwp-data-loaders are tagged and released?

[mpvginde]

Nice work! The general structure looks good.
I do have some comments (these are all personal flavours, feel free to disagree with them):

• I would prefer if the loader uniquely defines the traits of the returned dataset. You can still allow one loader to load dataset with different traits (with the post_processing). But the user should not decide the final traits of the loaded dataset.
• Relying on xr.open_dataset as core opening mechanism:
  • prevents you from using the dask parallelism in xr.open_mfdataset which can make a big difference if opening a year worth of daily forecasts
  • prevents you from loading more exotic dataformats like sqlite files (see for example here in mxalign)

Let me know what you think @leifdenby

[leifdenby]

• I would prefer if the loader uniquely defines the traits of the returned dataset. You can still allow one loader to load dataset with different traits (with the post_processing). But the user should not decide the final traits of the loaded dataset.

I see, that is a different way of going about it to say that the loader defines the traits of the dataset. I had thought that the same loader might be used for loading datasets on for say gridded or non-gridded data. But your way is simpler I think, and removes the ambiguity of whether the user or the loader should define the traits.

Relying on xr.open_dataset as core opening mechanism:

* prevents you from using the dask parallelism in `xr.open_mfdataset` which can make a big difference if opening a year worth of daily forecasts

* prevents you from loading more exotic dataformats like sqlite files (see for example [here](https://github.com/mlwp-tools/mxalign/blob/main/src/mxalign/loaders/harp_obstable.py) in `mxalign`)

Yes, I see. Ok, in that case we should go with classes the I guess :)

What do you think to my idea of putting test datasets on EWC for each loader we have implemented in mlwp-data-loaders? If you like this idea do you have an idea for where we could source some good sample datasets we can share publicly? I guess you have some on your computer :) Does using that CERRA sample dataset make sense (I can't seem to load it into mxalign though currently, so maybe it was created when anemoi-datasets had a different structure?

• If you like this idea then I will create separate issues on mlwp-data-loaders for each of 1) anemoi-datasets, 2) anemoi-inference and 3) harp_obstable

[mpvginde]

Yes putting test dataset on EWC is a great idea.
I will provide you with some examples and have a look at the cerra one to see what is going wrong (see also my comment on mlwp-tools/mlwp-data-loaders#1 (comment))

[observingClouds]

Following this discussion I wonder where the projection information like the one from CERRA should go to. I could see this as part of mxalign, as this is where the projection information is needed, but I can also very much see this as part of the data-loader as mlwp-data-specs should probably require projection information to be set.

[leifdenby]

Yes, I see. Ok, in that case we should go with classes the I guess :)

@mpvginde I had a look at the harp obstable loader implementation, and I don't think we need a class here actually :) To demonstrate this I have refactored how the loader modules are structure in mlwp-data-loaders, https://github.com/mlwp-tools/mlwp-data-loaders/tree/main/src/mlwp_data_loaders/loaders. Here's the one for harp obstables: https://github.com/mlwp-tools/mlwp-data-loaders/blob/main/src/mlwp_data_loaders/loaders/harp/obstable.py

What do you think? :)

[leifdenby]

Following this discussion I wonder where the projection information like the one from CERRA should go to. I could see this as part of mxalign, as this is where the projection information is needed, but I can also very much see this as part of the data-loader as mlwp-data-specs should probably require projection information to be set.

@observingClouds I would personally prefer that providing projection information should be part of the dataset spec, as-in either people set these in a cf-compliant manner in their source dataset (i.e. in cf-compliant netCDF files) and load directly with eg xr.open_dataset(...) or the loader they use/provide sets the projection information.