create new 3-15-023 sequence, to allow for use of correct range of WMO identifiers which cannot be stored using 3-15-013

[jbathegit]

Summary and purpose

Create a new BUFR Table D sequence 3-15-023, beginning with version 39 of the WMO tables.

Stakeholders

• U.S. NOAA/IOOS
• OceanOPS
• @david-i-berry is also aware of the need for this update

Discussions

The BUFR Table D descriptor 3-15-013 is designed for reporting ocean profiles from tagged marine animals. Currently, such data is reported in FM-64 TESAC using 7-digit WMO identifiers beginning with "99". According to the Guide to WIGOS, these identifiers are defined by OceanOPS and can be reported in BUFR using the Table D sequence 3-01-150 for WIGOS identifiers, with "22000" as the "Issuer of Identifier" and the aforementioned 7-digit WMO identifier as the "Local Identifier".

The issue is that this same 7-digit identifier is also intended to be reported as Table B descriptor 0-01-087 within the Table D sequence 3-15-013, especially given that not all users are yet able to process WIGOS identifiers. However, 0-01-087 is defined as a Numeric with a bit width of 23, so the maximum numerical value that can be stored in BUFR using this descriptor is 8388606. In other words, in order to store a 7-digit numerical value beginning with "99", one additional storage bit is required.

Given that recent discussions within the TT-TDCF have indicated a continued concern with the prospect of changing Table D sequences within future versions of the tables, we propose to create a new Table D sequence 3-15-023 which would be identical to 3-15-013, except that it would use operator 2-01-129 to add one additional bit to the 0-01-087 descriptor within the sequence.

Detailed proposal

Create a new Table D sequence 3-15-023 which would be identical to the existing Table D sequence 3-15-013, except for the following difference:

Modify the subsequence:

0-01-087

WMO marine observing platform extended identifier

to:

2-01-129	Change data width
0-01-087	WMO marine observing platform extended identifier
2-01-000	Cancel change data width

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference-16.3.2022: In discussion: revise proposal to only change Table B descriptor to 24 bits @david-i-berry will request feedback from OceanOps on the bit length

The team notes that there are inconsistent approaches between changing existing codes.

[jbathegit]

Yes, I agree that being allowed to change Table B but not Table D is very inconsistent, or, as @SimonElliottEUM put it, "Interesting". And as @david-i-berry noted for this particular case, changing the underlying 0-01-087 descriptor to 24 bits may end up being more impactful to the wider marine community than just changing this one particular 3-15-013 sequence, because 0-01-087 is already being used in other places, whereas 3-15-013 as far as we know is not yet being used by anyone.

Again, I'm happy to discuss this further in a more focused group, but the marine folks would like us to come up with something NLT the November fast-track. And as a reminder, I may not be available to participate during the week of the scheduled April TT-TDCF telecons.

[jbathegit]

Following a more focused group discussion on 6 April, several other TT-TDCF members did not want to modify 3-15-013 to make this change, so instead we are now proposing to add a new Table D descriptor 3-15-023 for FT2022-2. The new 3-15-023 sequence would be identical to the current 3-15-013 sequence, except with the addition of Table C operator 2-01-129 immediately preceding 0-01-087, and the addition of Table C operator 2-01-000 immediately following 0-01-087, as noted above.

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference-18-and-19.5.2022 notes:

Dave Berry and Sergio will validate; Jeff will get sample data from IOOS.

[jbathegit]

@amilan17 @marijanacrepulja

The branch has been updated. Sample data for validation will be provided within the next couple of weeks.

[amilan17]

@jbathegit mismatch of sequence  and element name (this might be an error in 3-15-013 too)

001019

Ship or mobile land station identifier

Is it supposed to be one of these below? 

001019

Long station or site name

or

001011

Ship or mobile land station identifier

[jbathegit]

Yeah, that's an error in 3-15-013 too, which explains why it's now in 3-15-023 as well (because I copied the sequence from 3-15-013 and just edited it to generate 3-15-023 ;-).

It's supposed to be the following in both sequences:

0-01-019

Long station or site name

because otherwise the corresponding Note_en (i.e. Platform ID, e.g. ct145-933-BAT2-18 (max 32 characters)) wouldn't make any sense, given that 0-01-011 only has a width of 9 characters, whereas 0-01-019 has the expected width of 32.

So I guess that's the good news - that the FXY number is correct it's only the element name that's wrong, which in turn means it shouldn't be an impactful correction since the FXY number is the more important and significant identifier here. Do you want me to go ahead and make this edit in both places in the csv file and then push it up as a commit patch?

[jbathegit]

Adding @david-i-berry to this chat for his awareness. David, just FYI that there's a mismatch of FXY number and element name within the existing 3-15-013 sequence, as @amilan17 noted above. The correct element name for 0-01-019 is "Long station or site name", not "Ship or mobile land station identifier".

Practically speaking, I doubt this is really affecting anyone, because most automated software goes by the FXY number, not the element name, and based on the corresponding Note_en the intent here was clearly for this to be a 32-character field, not a 9-character field. But that said, it should be fixed now that it's been discovered.

Good catch Anna, and thanks!

[amilan17]

@jbathegit - yes. please go ahead and update the element names in both places. thx.

[jbathegit]

OK, it's done!

[jbathegit]

@david-i-berry @sergioh-pessoal

Please find attached a zip file containing two sample BUFR messages from IOOS using the new 3-15-023 sequence, along with two corresponding CSV files showing the original values that were encoded into the messages, so that you can confirm you decoded the messages correctly.

If you have any questions or encounter any problems, please let me know. The messages were encoded using the ecCodes software, and I've already decoded them using the NCEPLIBS-bufr software, so technically I think we've already satisfied the criteria for validation, but we would certainly welcome additional eyes on this sample data ;-)

profile_1014355104.zip

[jbathegit]

Hi @david-i-berry and @sergioh-pessoal - is there any chance you could try to validate these sample messages prior to our TT-TDCF team meeting on Wednesday? (Thanks! ;-)

[sergioh-pessoal]

I decoded both files using mbufrtools software and tables in git (version 39) with no problems. The decoded values are the same as the .csv files provided and also looks consistent. So the BUFR template is valid.
Please find the decoded files attached

decoded_profiles.zip

[jbathegit]

Thanks very much Sergio!

[SibylleK]

I had also decoded successfully both BUFR examples with BUFR tables of the branch origin/issue106 with the DWD decoding software.
profiles_decoded_DWD.zip

The values are the same, despite of rounding differences or rather the way of representing. Only the observation sequence numbers in the .csv files are strange. But, I got the same values as Sergio.

[david-i-berry]

Decoded files below, final column in each lists the decoded values. As with @SibylleK, I find agreement with that provided in the examples above (noting rounding) except for the observation sequence number. This appears to be in an error in the sample decoded files provided to and uploaded by @jbathegit rather than in the BUFR.

profile_1014355104.csv
profile_1014380568.csv

Files decoded using eccodes bufr_dump and then merged with the original csv files for comparison.

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference-8.6.2022 notes:

• sequence has been validated
• @marijanacrepulja validate and move to validated column

[marijanacrepulja]

@jbathegit thanks for updating the branch.
@amilan17 I have moved to validated column, please review and move to "Ready for FT"

[amilan17]

@jbathegit @marijanacrepulja 

I think we should add a note to 3 15 013 and this is my suggestion: "Note: It is recommended to use 3 15 023 instead."

Please confirm asap.

[marijanacrepulja]

@amilan17 I am happy with your suggestion