Multiple data requirements for a given source

[lewisjared]

The problem

We currently support defining a tuple of data requirements for a metric. Currently if one provides multiple requirements for a given source type, it will silently only use the last one (see snippet below)

    for requirement in metric.data_requirements:
        if requirement.source_type not in data_catalog:
            raise InvalidMetricException(metric, f"No data catalog for source type {requirement.source_type}")

        dataset_groups[requirement.source_type] = extract_covered_datasets(
            data_catalog[requirement.source_type], requirement
        )

Is there ever the need to define multiple data requirements for the same type of data and if so what is the desired behaviour? If not I suggest we raise an exception instead of silently using the last one.

Additional context

After this step in the solve, we do assume that there is a single set of datasets per source type so there would be some cascading changes

[lewisjared]

@nocollier @bouweandela @lee1043 Thoughts? I would have a strong preference for not allowing multiple requirements for a given source, but if there is a need let me know

[nocollier]

I may not completely understand the context of this but it raises the following questions in my mind:

1. I know REF development is specific for FastTrack, but the first thing people will wonder is how things compare to CMIP6 (which we actually have implemented). Does this mean we would have to implement separate metrics (one with CMIP6 requirements and one with CMIP7)?
2. In Exeter we added the possibility of adding in obs4MIPs as a data requirement and I thought it was then creating executions on the product of all possiblities. Does limiting it to a single data requirement mean that I would need separate metric definitions for different obs (gpp vs WECANN and gpp vs Fluxnet for example)?
3. In ILAMB metrics, the user has the option to see primary variable relationships (gpp vs tas for example). We construct these from the observations of which then there could be several. I would define the tuple like (obs4MIP(gpp), obs4MIPs(tas), CMIP6(gpp,tas),).

Am I misunderstanding something?

[lewisjared]

I was intending to add another layer on top of the the data requirements to capture cmip6 + cmip7. So you could define multiple sets of data requirements where each could target different source types of that was wished. Each set of data requirements is solved independently. The question here is if within a set of data requirements does it make sense to have duplicate requirements for a given source type. The other alternative is that we add some additional operators for and/or operations. The extra layer would be simpler but the operators might be easier to understand. I can make another issue with the syntax options when I'm back at my computer.

The option is to merge the concepts of the CMIP eras and try make a single data Catalog for CMIP data. That requires us making up yet another set of naming conventions.

[lee1043]

@lewisjared I think it is not impossible that some metric will need multiple variables from the same source, e.g., need pr and ts from the same dataset or different datasets for each of model and observation to calculate a metric. Will such case need multiple data requirements?

[lewisjared]

The product of the groups calculated from each data source works (see ILAMB metrics). Perhaps we need a construct to be more explicit about what each of the sets of groups that make up the product are. One of set of groups could be cmip6 or cmip7 data and the other might be the different obs groups you mentioned. You want to "or" the data requirements for the CMIP eras I.e. ( (cmip6 | cmip7) X obs) rather than cmip6 X cmip7 X obs.

This might be clearer with a proposed syntax

[lewisjared]

@lewisjared I think it is not impossible that some metric will need multiple variables from the same source, e.g., need pr and ts from the same dataset or different datasets for each of model and observation to calculate a metric. Will such case need multiple data requirements?

But does a single execution need more than one group of datasets from a single source. Or does that result in more than 1 execution. The example @nocollier has is 3 different requirements but each execution would use a group of obs datasets from one of the requirements.

[lewisjared]

One approach could be to us the or operator (|) to allow multiple different data requirements to support the CMIP6 and CMIP7 support (( (cmip6 | cmip7) X obs). We could potentially even use | within the source_type, but that is a more complex soln.

DataRequirement(
    source_type=SourceDatasetType.obs4MIPs,
    filters=(
        FacetFilter(facets={"source_id": (obs_source,), "variable_id": (obs_variable,)}),
    ),
    group_by=("source_id", "variable_id"),
),
(
    DataRequirement(
        source_type=SourceDatasetType.CMIP6,
        filters=tuple(filters),
        group_by=("source_id", "experiment_id", "variant_label", "member_id"),
    )
    | DataRequirement(
        source_type=SourceDatasetType.CMIP7,
        filters=tuple(filters),
        group_by=("source_id", "experiment_id", "variant_label", "member_id"),
    )
)

[lewisjared]

If we needed to, we could also update the type hint for the metric to optionally provide multiple sets of requirements. Each ExecutionDataRequirement is evaluated independently to determine how to calculate the different groups.

ExecutionDataRequirement = tuple[DataRequirement, ...]

class AbstractMetric(Protocol):
    ...
     data_requirements: tuple[ExecutionDataRequirement, ...] | ExecutionDataRequirement

[bouweandela]

Maybe it would be useful to wait with making decisions on this topic until we have integrated more diagnostics (or ran into issues while trying to do that), as we currently have only 6 out of 30 from Diagnostics (view). That should give us a better idea of what is needed.