Discussion: Load files for ensemble trial members from multiple directories

[mo-tomosevans]

Describe the bug

Ensemble trial data will typically produce directory structure as follows for a particular run:
20240609T0600Z/enuk_um_003, 20240609T0600Z/enuk_um_004, 20240609T0600Z/enuk_um_005. The number in each folder represents the ensemble member number, each one will have exactly the same filenames in them i.e. enukaa_pd000, enukaa_pd003, ... , enukaa_pd123, where the number represents the forecast lead time.
When passing file globs into the workflow (/enuk_um_0*/enukaa_pd*) it will fail because it sets up one directory and creates symbolic links to each of the files in this directory, using only the final part of the path (i.e. the file name) as the link name. So it will correctly expand the pattern and find all the files, but will only load the first set of enukaa_pd* files it finds. Files for subsequent ensemble members are treated as duplicates, so none of the other members are loaded in.
As a result, the plots generated are only of one realization despite the globbing pointing towards multiple members.

How to reproduce

Steps to reproduce the behaviour:

1. Choose one trial model and point the filepath in rose-suite.conf to a path ending /enuk_um*/enukaa_pd*
2. Run the workflow
3. Look at the job.err log for the fetch_m1 task - it should show it loads the first enukaa_pd* files, but if it finds another file path with the same filename endpoint it will print a duplicate error.

Expected behaviour

This probably requires a discussion, Stephen Gallagher (@SGallagherMet) has suggested this could be a common issue for other meteorological centres producing trial output but may be intentionally coded.

[SGallagherMet]

mo-tomosevans - I've made some tweaks to the title/description, please have a check and make sure I've not got anything wrong.

[SGallagherMet]

This has the potential to produce errors in the verification as well as just missing data. The order of a results returned by the glob method is not guaranteed, so it is hypothetically possible to mix input data from different ensemble members.

[jfrost-mo]

Context

We symlink the files into a directory within CSET to simplify the loading code. We can't directly use the file from there original place as for some sources (such as MASS, HTTP, etc.) it isn't readable directly and copies have to be created.

The globs can be arbitrarily complex, so I don't really want to get into reproducing the directory structure.

Our loading doesn't actually use the filename, so maybe we could just append a bit to it to make the files unique. It is useful to be able to xconv the original files however, so we wouldn't want to replace it wholesale.

Proposal

Add a random suffix to the file names to make them unique. For example, enukaa_pd000.pp would become enukaa_pd000_e329165289aaab09.pp.

Drawbacks

The proposed solution would fix this, but it would make it hard to manually inspect the data, as you don't know what is from where.

[SGallagherMet]

Drawbacks

The proposed solution would fix this, but it would make it hard to manually inspect the data, as you don't know what is from where.

Maybe a halfway house - generate a filename by replacing the path separators with another character and add a random suffix to avoid collisions? It might not be pretty but you'd avoid having to handle directories and at least be able to make a guess at where the files had come from based on the name.

[jfrost-mo]

As long as we use the full path we shouldn't need to add a suffix, as it will have to be unique. We already have the slugify function, so maybe we just save it using that.