Divide CodeFlag_4_2.csv into multiple files

[amilan17]

things to consider at the same time:

• divide other CodeFlag files too?
• script updates
• divide column with name of discipline and category into two columns

[sebvi]

We already do this in eccodes: we do not maintain a huge code table 4.2 but we break it down by disciplines and categories. The subtables are referenced 4.2.[discipline].[category]
4.2.0.0 is code table 4.2 discipline "meteorology", category "temperatures"

[sebvi]

When are we doing this? it would greatly help when updating that massive table in the branches. Should this be discussed at some point?

[amilan17]

@marijanacrepulja -- can we add this to the agenda tomorrow for discussion if we have time.

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference-10.01.2023 notes:

team agrees to this change and the focus should be on CF 4.2

@SimonElliottEUM  "discipline should come first" and will add a comment with an example of this concept;

[amilan17]

ID	ProductDiscipline_number	ProductDiscipline_name	ParameterCategory_number	ParameterCategory_Name	CodeFlag	Value	MeaningParameterDescription	Note	noteIDs	UnitComments	Status
4.2.0.0	0	Meteorological products	0	temperature	0		Temperature			K	Operational
4.2.0.0	0	Meteorological products	0	temperature	1		Virtual temperature			K	Operational
4.2.0.0	0	Meteorological products	0	temperature	2		Potential temperature			K	Operational
4.2.0.0	0	Meteorological products	0	temperature	3		Pseudo-adiabatic potential temperature or equivalent potential temperature			K	Operational

[amilan17]

related to issue #77

[SimonElliottEUM]

Please see my comment from 10 Jan 2023 above. The discipline should come first. The discipline is given in Section 0 for just this reason - you get to it early on so you can interpret the other sections accordingly. The meaning of CTs 4.1 and 4.2 depend upon which discipline is being used. So you need to know that first.

[amilan17]

@SimonElliottEUM Can you provide an example of how you would like to see discipline first?

[SimonElliottEUM]

@amilan17 In the table above we have ID = 4.2.0.0 (Code table 4.2, Product discipline 0 – Meteorological products, parameter category 0: temperature).
This would be more logical if it were given as ID = 0.4.2.0 (Product discipline 0 – Meteorological products, Code table 4.2, parameter category 0: temperature).
The meaning of CT 4.2 is predicated on the product discipline, so this needs to be known first.

[amilan17]

@SimonElliottEUM I don't understand this statement: "The meaning of CT 4.2 is predicated on the product discipline, so this needs to be known first".

From what I understand, 4.2 is independent of the discipline.

• 4 represents Code tables used in section 4.
• 4.1 Parameter category by product discipline
• 4.2 Parameter number by product discipline and parameter category
• 4.3 Type of generating process
  etc...

[SimonElliottEUM]

@amilan17 The interpretation of CT 4.2 does indeed depend upon the discipline. Consider the definition of Parameter Category 0:

• Discipline 0, CT 4.2, Parameter Category 0 = temperature
• Discipline 1, CT 4.2, Parameter Category 0 = hydrology basic products
• Discipline 2, CT 4.2, Parameter Category 0 = vegetation/biomass
• Discipline 3, CT 4.2, Parameter Category 0 = image format products
• Discipline 4, CT 4.2, Parameter Category 0 = temperature
• Discipline 10, CT 4.2, Parameter Category 0 = waves
• Discipline 20, CT 4.2, Parameter Category 0 = health indicators

[sebvi]

I disagree with @SimonElliottEUM point of view.

We are trying to split the current Code Table 4.2 into smaller chunks in the form of several csv files because it is not manageable at the moment. The majority of the proposals these days is usually to add new entries in Code Table 4.2 . As a result, most branches we create during the fast tracks all touch the same csv file and it is sometimes a nightmare for @amilan17 . Most merging attempts result in conflicts.

I understand the point you make that in the GRIB2 standard, the discipline is defined in section 0, i.e. before section 4 is decoded and before Code Table 4.2 is used. But this mechanism is completely independent of the name of the files we will use. If I follow your reasoning, the category is defined in Code Table 4.1, decoded before Code Table 4.2 so following your reasoning we could also use 0.0.4.2 as well. Note that the mechanism of having the discipline in section 0 is completely broken. The Code table able 0.0, which define the discipline is supposed to be versioned and it is possible to use the range 192-254 for local use ..... except that the version for this table is defined after in section 1, Code Table 1.0 !! So you should start decode the message, ignore the discipline, go to the version of the master Table, find out the version, then go back and read the correct version of Code Table 0.0 to be able to decode it.... This is why, in GRIB3, the discipline was moved in the product section because it did not make any sense to have it in section 0... And in section 0, the first thing you read in the version to be able to decode all the subsequent tables.

But the end, for this issue, it is purely a matter of convention and what we want to do. We could do discipline.category.section.table.csv or any other permutations of these 4 and the content will always be the same in any case. Now for practical reason, since all other tables start with section.table , I find it extremely confusing to have an exception for Code Table 4.2. If you think about it and if we do like you suggest, we will have a huge risks of confusion, particularly for discipline 1 to 3. For instance, 1.4.2.0.csv will be next to 1.4.csv , the first one is a part of Code Table 4.2, the other one is Code Table 1.4 !

Note that in principle, what we try to do here could be applied to Code Table 4.1 which has subtable tables per discipline, hence we could use 4.1. for that table. This is what we do in ecCodes and it is also what we will do in GRIB3 for Table 7.3, which is the mere copy of Code Table 4.2 in GRIB2: GRIB3

[amilan17]

first steps:

1. split tables while keeping history
2. update script: https://github.com/wmo-im/GRIB2/blob/master/scripts/create_master_lists.py

next steps:
determine how to publish to the codes registry

[sebvi]

I am working on the splitting in a branch but I am not done yet.

[sebvi]

since this is done, should we close this issue?

[amilan17]

done.