Rules for model output filenames/filepaths, especially with respect to round_id identification

[annakrystalli]

Background

An important part of validation consists of validating the submission file name and location within the hub. The file name, it appears, can also be the single source of a round_id in the case of hubs where round_id_from_variable is false.

And given each submission file is targeting a specific round, it is important to be able to accurately determine the round_id of a file, not only to validate it but to determine which schema to use to validate the file itself which may well be round dependant. So by extension, having a solid understanding of the expectations of file names and file paths, especially with respect to round_id identification, is critical.

I think it's a good time to also document this topic in more detail as the current coverage is minimal atm. so apart from clarification, I hope this discussion can become the source for additional documentation content.

Current approaches

From the scan of current hub validation methods, what I've been able to decipher is:

• All current hubs use dates as round_ids which are contained as data within the files.
• These dates are contained in a single model output column in the file (e.g. origin_date, forecast_date, model_projection_date) and the name of the column is hard coded in the validation code.
• These round ID dates are also used as the first part of the file path followed by the team name (although I have seen files where no team exists, possibly because the hub is the model owner?) followed by the model name. all separated by a hyphen (-). Underscores (_) seem to sometimes be used within model/team names.
• Each file is submitted to a particular folder whose name is made up of the team and model name separated by a -.

To summarise in hubverse terms:

• All current hubs use what is effectively in the hubverse a round_id_from_variable set up, where the round_id is contained, as a single unique value, in the column specified in the round_id property of the tasks.json config file.
• The convention for file paths thus seems to be [round_id]-[team_abbr]-[model_abbr] or [round_id]-[model_id] (as model_id = [team_abbr]-[model_abbr]).
• Files by a given team for a given model are submitted to a directory named model_id.
• Underscores (_) are used within parameters like team_abbr etc. Hyphens (-) are used as separators (and therefore should never be used within parameter names) with the exception of date round_iss which DO contain hyphens but follow a very explicit pattern.
• Once the round_id has been identified, it can be used to identify the round and therefore any round specific configuration in the tasks.json config file which will then be used for model output file content (data) validation.

What needs to be validated (with respect to the current discussion):

• That the round_id is valid (i.e. with respect to values specified in the config file).
• If round_id_from_variable is TRUE, that:
  • the round id variable contains a single unique round_id.
  • the single unique round_id in the file matches the round_id in the file name.
• That the model_id portion of the file name matches the folder the file is being submitted to and team/model definitions in the corresponding model metadata file.

Questions/Actions

• 1. How accurate is the above description of the situation? Am I missing something?
• 2. How strict will the expectation in terms of file names/file paths (which include model_output directory structure?) be? Is there a situation where this might deviate? Any sense when and why?
• 3. As mentioned here and in Develop example hub which uses explicit round_ids (not round_id_from_variable: true) #8, all current active hubs and example hubs all follow a round_id_from_variable: true paradigm. So I don't have a realistic use case of a hub where round_id is defined as part of the config only (i.e. not contained in the data file) yet. It would be good to develop one, both for demo and testing purposes. If anyone has ideas of what a realistic example of such a hub would look like, please leave your comments in Develop example hub which uses explicit round_ids (not round_id_from_variable: true) #8.
• 4. As noted, once the overview/details have been agreed upon, it would be good to update hubDocs with them.

[LucieContamin]

To answer your questions from the US Scenario Modeling Hub:

1. The description is correct, however we have some teams that use a mix of _ and -(not common but that happen). So we treat the filename as [YYYY-MM-DD]-[model_id] with model_id being a possible mix of letters, numbers, - and _ containing the team and model abbreviation [team_abbr](_|-)[model_abbr] (for each abbr: [a-zA-Z0-9_-] possible). Also, our ensembles don't have the team information associated, so the model_id for these files is just [model_abbr].
   But we might not want to implement that behavior for the Hubverse.

2. As of now, all our submission where made in the same folder with the same file names/file paths logic. I cannot think of a situation where this might deviate.

3. I think we can do one round with submission files without the origin_date column, but with the target_date. I can add a round in the complex scenario example hub with that information. In this case, the round_id information will only be in the filename.

4. Agreed

[nickreich]

1. In general, i think your description of the current situation is complete and I don't think you are missing anything.

2. I think the strictness of expectation here is an important question and one that I anticipate people will have different feelings about. My general preference here would be to start with a set-up that has a fairly strict expectation of full directory structure and filenames as I suspect that might simplify development. But to also develop with an eye towards being able to accommodate different structures down the road. In partial answer to item 3, I could imagine basically a submission file in a hub where round_id_from_variable: false looking virtually identical to one where round_id_from_variable: true except that it doesn't have the forecast_date column or whichever column has been tagged as the variable from which round_id is taken.

I could see an argument for requiring that the first "part" of a filename be a round_id always. So in the existing hubs that have dates as round_ids, that would mean that we interpret the date that is at the start of the filename as a round_id and basically ignore the model_id that comes after it. It would also allow for simpler labeling in the future, where a file could just be labeled as a round_id which may or may not be a date. There might need to be stricter rules about formatting of the filename and round_id values, though, if we need to end up "searching" for a round_id in a filename that might have other objects in it. As I said earlier, I think it would be ok to be more prescriptive here, at least as we are getting started, about what the structure/names should be.

Mentioning that the conversation

[elray1]

• I think you summarized the situation accurately
• I would be in favor of picking and enforcing some standards for formatting of team/model names. allowing for flexibility in use of _ vs - doesn't seem worth the effort
• I agree with Nick that if it is in any way helpful, to start off with it is fine to pick one rigid standard for what files are called and directory structures. I do think this could be part of a bigger (and potentially separate) issue about allowing hubs to specify a flexible data partitioning scheme, but I'd vote for doing something standard and simple for now and putting options for other data partitions off as an enhancement to maybe look at later on.

[LucieContamin]

I agree with both Nick and Evan

[annakrystalli]

Thanks all! I've made note of the comments so far and fully agree.
I especially agree with @elray1 comment:

I would be in favor of picking and enforcing some standards for formatting of team/model names. allowing for flexibility in use of _ vs - doesn't seem worth the effort.

Having said that, team_abbr and model_abbr will be clearly specified in each models metadata file, correct? That could make it easy to parse as we can just match to fixed text chunks.

One thing I think is important to enforce is adding the model_id to the filename as it is the only info we have to check a file is being submitted to the right directory.

[annakrystalli]

Transferring some pertinent slack conversations here:
nick
quick question: did we set something somewhere that makes round_id_from_variable must always be true? I had some fuzzy recollection of this. (edited)

evan
I think there was a sidebar discussion among some subset of people about whether we could maybe do that, but I don’t think there was a full group conversation or formal decision made.
I think I remember that the original motivation for round_id_from_variable = FALSE was to avoid having duplicate information stored in files (though I note that if file size is a concern, hubs should probably be using parquet format, in which case a column with all values duplicated would be stored very efficiently anyways).

Anna Krystalli
Hmmm, I don't think I had understood what round_id_from_variable was to be used for. This begs a few follow up questions then:

1. If round_id_from_variable: false then what should the value of round_id be? In our schema, here's our current description: "Round identifier. If round_id_from_variable = true, round_id should be the name of a task id variable present in all sets of modeling task specifications"

2. Linked to the above question, I was under the impression that if round_id_from_variable: false , rounds would be identified by a code specified in round_id property, and each round would necessarily need it's own round specification in the config file. If the idea is that we are using round_id_from_variable: false to avoid having an e.g. origin_date column in the data, that would mean having a giant config file with a round specification per round ID, no? How else would we get a list of all valid round_id values?

3. In addition to all this, how would we know from our config what the missing task ID variable that the round_ids refer to mean? For example, if the round_id refers to origin date but there is no origin date task ID and the only place it is found is in the file name, how would one know that we're supposed to use that value in conjunction with horizon to calculate target_date?

evan
Continuing in github discussion sounds fine, but, just to get thoughts down: this is a vague memory, but i think the place this came up originally was thinking about rounds in the scenario hubs, which are currently identified with dates, but where it might be preferable to have a more human-understandable name for the round like “round1” or “round_1_vaccine_efficacy_and_immune_escape”. so the goal would not necessarily be to avoid having whatever necessary columns to identify target dates, etc., but to tie the round id more to a concept or goal of a round rather than the dates when it happened to run. and with a round id like that, maybe having it as a data value in the submission file is not that useful. (edited)

Anna Krystalli

Right! That's what I vaguely remembered too which why the use case of avoiding round id information in file names confused me and threw up all these implementation questions!

[LucieContamin]

It's what I was remembering too. I think the idea was for the scenario hub. I added an example of a round in the tasks.json of the complex scenario repo.

But to follow up on that:
If round_id_from_variable: false, round_id can be a date or a string but should match the associated filename or it's impossible to match the files to the correct rounds.
For example:

     "round_id": "round_X",
      "round_id_from_variable": false,

then the submission files should have at least "round_X" in the filename. Am I correct?

Also, as I think it was mainly thought for scenario hubs, we already have a giant config file with a round specification per round, so I don't think it's going to be an issue.

[annakrystalli]

Thanks @LucieContamin , that's really helpful to illustrate some of my questions.

When someone now submits to round "2022-09-04", there will be no origin_date column in the file.

When we then come to connect to the hub and extract data, for any data from round "2022-09-04" the column origin_date will contain NAs. And even if we wanted to try and populate that column somehow, there is nothing in the configuration of that round to say that "2022-09-04" refers to origin date.