Establish common data store conventions

[forman]

Is your feature request related to a problem? Please describe.

The xcube data store framework should establish common conventions so we have a consistent behaviour across all store implementations. The current store implementations (SH, CCI, CDS) behave slightly different.

1. Open parameters that constrain the dataset to be opened should all have the same names, have same or compatible type, and should be interpreted in the same way.
2. Stores should share the same idea of what is a coordinate variable and what a data variable.

Describe the solution you'd like

1. Open parameters

For a constraining open parameter P we introduce the following convention:

• If P is None means, parameter not given, hence no constraint applies, hence full containment.
• If not P means, we exclude what would otherwise be fully included.
• Else, the given constraint applies.

Common names are variable_names: List[str], bbox:Union[str,Tuple[float, float, float, float]], crs:str, spatial_res: float, time_range: Tuple[Optional[str], Optional[str]], time_period: str. A store may require all, or individual parameters, or none at all.

E.g. applied to variable_names, this means

• variable_names is None - include all data variables
• variable_names == [] - do not include data variables (schema only)
• variable_names == ["<var_1>", "<var_2>", ...] only include data variables named "<var_1>", "<var_2>", ...

Describe this convention in docstrings.

EDIT

A store implementation should document its available open parameters

1. programmatically in the DataStore.get_open_data_params_schema()method
2. in the docstring of the implementing class. Each parameter description should include its name, Python type and description.

2. Coordinate vs Data Variables

• The open parameter variable_names shall refer to data variables only.
• A store implementation shall add any available coordinate variables at least for every dimension of the included data variables.

Describe this convention in docstrings.

Additional context

We'll create linked issues and PRs in the store code repos where we fix implementations wrt this issue.

EDIT

This issue resolves only with the following:

• Be compliant with data store convention xcube-cci#10
• Be compliant with data store convention xcube-cds#1
• Be compliant with data store convention xcube-sh#12

[dzelge]

Is it possible that a store requires open parameters that are not listed in this convention?

[forman]

Is it possible that a store requires open parameters that are not listed in this convention?

Yes, of course. A store can have any open parameters. This convention is about a set of common parameters.

[pont-us]

Regarding bbox:Union[str,Tuple[float, float, float, float]] and time_range: Tuple[Optional[str], Optional[str]]: I believe that currently all stores use Python lists rather than tuples here, and tuples won't validate against xcube's JsonArraySchema (however, I have a fix in https://github.com/dcs4cop/xcube/tree/pont-334-json-schema-date-limits to remedy this).

I agree that tuples are more suitable here, but we should be careful about not breaking anything with the change. Even if we merge my branch into xcube before updating the stores, we also need to confirm that the generator UI is OK with tuples.

[pont-us]

Also -- in bbox:Union[str,Tuple[float, float, float, float]] , what is the envisaged format of str?

[forman]

I believe that currently all stores use Python lists rather than tuples here

What do you mean by use? Examples?

with the change

Which change? You may confuse the JSON representation of things with the object instances they are deserialized into. A JSON array schema that uses the tuple-item form will usually deserialize a JSON-array into a Python tuple rather than a list.

To shorten discussion I'm also fine with Union[Tuple[float, float, float, float], Sequence[float]].

[forman]

Also -- in bbox:Union[str,Tuple[float, float, float, float]] , what is the envisaged format of str?

WKT. But I agree, we should avoid it for time being.

[pont-us]

What do you mean by use? Examples?

"use" in the sense of "only accept arrays as the bbox argument" (no longer true, but was when I wrote it). e.g. in xcube-sh: JsonArraySchema(items=(JsonNumberSchema(), JsonNumberSchema(), JsonNumberSchema(), JsonNumberSchema())). No longer a problem now that PR 336 is merged, but before that a tuple bbox argument would not have validated.

To shorten discussion I'm also fine with Union[Tuple[float, float, float, float], Sequence[float]].

I think that plain Tuple[float, float, float, float] is preferable now that tuples validate.

[forman]

I think that plain Tuple[float, float, float, float] is preferable now that tuples validate.

No, that is not true. They only validate by JSON schema, that is, only when parameters come as JSON objects. Users can still call our methods without invoking JSON schema validation.

[forman]

To be more pythonic, let's please use Union[Tuple[float, float, float, float], Sequence[float]].