Revision of E-Profile Sequences

[richardweedon]

Branch

https://github.com/wmo-im/BUFR4/tree/issue-21a

Summary and purpose

A windprofiler (WP) is a ground based remote sensing instrument to measure vertical profiles of wind, this can be either a radar, lidar or sodar. Radar based WP networks exist worldwide and the data are encoded in BUFR to be exchanged over the GTS. However, each of these networks uses a different implementation of the BUFR code. In this document we propose a harmonized implementation of BUFR for WP data. The proposal is based on a review of the existing implementations and on data user requirements. The purpose of this template is for the exchange of Level 2 product data for Meteorological Products and not for the distribution of L0 or L1 products.

As part of the COST action ES0702 EG-CLIMET http://www.eg-climet.org , a special working group was formed to look at the “Harmonisation of European automatic (elastic backscatter) Lidar and Ceilometer Network, Calibrations, recording formats and archiving protocols”. As part of these activities it was agreed that a common BUFR template should be agreed upon for this data. After an extensive review the following Table D sequence has been agreed upon by the E-PROFILE ALC Expert Team.

These proposals are an initiative of the E-PROFILE programme of EUMETNET. This was discussed at the E-PROFILE ET in Helsinki 15-17th October.

Action proposed

final proposal from Sep 2020: https://github.com/wmo-im/BUFR4/files/5306953/wind_prof_prp-20200930.docx

Full details of the new (revised 5th june 2016) D sequences and Descriptors are given in the attached document
windproflidar-amend-2_bis+MT.docx

[SibylleK]

@efucile What is the status of this proposal? Do you think it is still possible to get it validated for the fast-track?
The EUMETNET E-PROFILE team has a strong requirement for the new sequences.
The document is worth to be discussed and agreed to by the team.
My comments are, that I do not think that new descriptors are needed to minimize the data width by 1 bit (all "Uncertainty in..") 3 01 032 for "Common header sequence" has to be changed, because it already exists. But, it seems to be a typo and should be 3 01 132. Apart from that, I would agree to the new proposed sequences and descriptors.
If there is no strong objection to the proposal, could we get a branch for the validation?
And if so, @richardweedon could you provide some example for the validation here?

[efucile]

@SibylleK and @richardweedon I think we should try to validate this. There is strong pressure to do it and it is pending for many years. It is time to close it. However, @jitsukoh has to decide if we can push it in this fast track. I think that we are going to be late for other reasons and this could give us time to finalize this.

[jitsukoh]

@efucile, @SibylleK and @richardweedon, thank you for the proposal and my apologies to have not replied earlier. It took some time to refresh my memory on this...
I reviewed our discussion in 2016 and honestly I am a little worried about repeating the same thing. The core issue of the discussion back in 2016 was the representation of uncertainty; on one hand the team was looking for a generic way of expressing it (not only for elements of profiler but for radiosonde) to avoid using too many descriptors and on the other hand, the detailed definition and the way of calculation of proposed uncertainty descriptor for E-EPROFILE (i.e. how to use the proposed descriptors) was not given. I posed same questions over and over again because JMA's profiler experts did not understand how the information would be provided and how they would be used in a meaningful way, but I didn't get any information and I ended up accepting them because blocking the process was not my intention. My question was about the descriptors that are removed in this new proposal. So in short, we wasted 3 descriptors. I think our experts would have the same questions over the uncertainty descriptors (and maybe other questions) and I would like to make sure this point is clarified this time before making a decision.
Can I suggest that we set a deadline, for example by the end of next week for posting technical questions on GitHub and ask the proposer to answer them also on GitHub, and if all the technical issues are solved, we have a short teleconference to conclude? At this moment, I don't think we have time on 22nd to discuss details on this proposal.
I'd appreciate your consideration.

[simonebircher]

@jitsukoh , thank you very much for your efforts spent on the E-PROFILE BUFR template proposal as well as you comments posted here. We appreciate your suggestion to answer technical questions on GitHub by the end of next week and a concluding teleconference.

Could you possibly provide us with a link to the discussion in 2016 that you are referring to?
Thanks a lot in advance and best regards

Simone Bircher-Adrot
E-PROFILE assistant network manager

[efucile]

Dear @simonebircher
we welcome your participation in this forum. I understand that you are working for Eumetnet and you are involved in the E-Profile network, but I don't have your details in the WMO expert database. We can take contributions only from experts registered in the WMO or we accept opinions from invited experts. I would kindly ask you to send an email to me efucile@wmo.int and we can try to formalize your intervention.

[jitsukoh]

Dear team members, @wmo-im/tdcf

We got an urgent proposal from EUMETNET, which was not on the agenda on our teleconference in May, and I would like to propose following arrangements to accommodate the request.
Because of the nature of this proposal which requires an intensive review, I would like to invite all of you to review the proposal and post questions to this forum by June 17, but please let me know if you cannot make it (need more time to review), because this arrangement is an exception of the schedule that the team had already agreed.
I ask @richardweedon and other colleagues to answer posted questions, and when all the technical issues are solved on Github, I will propose a teleconference to conclude.
Thank you for your support in advance.

[SibylleK]

@jitsukoh Dear Jisukoh, I have already made some comments regarding the new descriptors for the uncertainties. I think, they are not needed as they are only minimize the data width by 1 bit of the existing one. I suppose they are only proposed to have the descriptors in a pretty order. The other new descriptors are for changing the reference values to allow negative values. Although I am not at all in favor of changing existing entries, it is also possible to change the reference values for the existing entries in BUFR table version 35. This means of course that the correct version number had to be used by encoding and decoding such BUFR. For one entry (Particle depolarization ratio) the bit width would change, which makes the BUFR unreadable if a wrong BUFR table version is used.
There is a typo at the descriptor for"Common header sequence". It should be 3 01 132.
Within the Common header sequence 0 02 003 -"Type of measuring equipment used" is used instead of the new proposed 0 02 005. It this intended?
The new sequences do not include the Common header sequence. Normally it would be included in such sequences. But of course they could be preceded by the header sequence.

If the requirement is such urgent and the messages should be on GTS as soon as possible, they could be encoded with a pure sequence of descriptors, of course.
Despite of the new entries for the "Upper air remote sensing instrument type" it could be encoded already by using operator descriptors. But, I know, changing the reference value with the usage of 203YYY causes trouble in some decoding software....

These are my more technical comments to the proposal and I am now going on holiday until the 6th of July, therefore I am sorry, that I am not able to respond or support the validation process till then.

[marijanacrepulja]

I agree with @SibylleK comments on the new descriptors for the uncertainties, as I believe we can use existing descriptors without introducing new elements for reducing data width. When comes to new descriptors for introducing new reference value, we can use operator 203YYY and existing descriptors for this purpose. However, if there is an issue that 203 operator can cause trouble in some decoding software we can address this with new descriptors.

@richardweedon I have a question regarding 0 33 002.
Does quality information (0 33 002) refer to u, v, w component respectively in the new sequence 3 09 024?
Similar in the sequence 3 09 025, quality information (0 33 002) refers to virtual temperature and w component respectively?
When comes to sequence 3 09 026, does quality information (0 33 002) refer to entire message or?

[jitsukoh]

This is the question that I posed in 2016, but could you provide us with any reference document(s) about the calculation of measurement uncertainties that you are using in producing the wind profiler and lidar products?

[jitsukoh]

Dear team members, @wmo-im/tdcf

To accommodate the request for approval of E-Profile sequences by the next fast-track procedure, a separate teleconference will be held on July 8 at 12UTC. Enrico already sent an invitation to those who already posted comments, but if there is anyone interested in this discussion and decision making, s/he is welcome to join, so please let me know.

[jbathegit]

@jitsukoh , @efucile , @SibylleK , @marijanacrepulja , @richardweedon

I'm interested in this topic; however, I'm not available on July 8th at 12 UTC because I have a separate telecon (for GODEX-NWP) from 11-14 UTC on that same day. Could I just ask you to please keep me in the loop on any developments or final decisions, because we ingest wind profiler data at NCEP, so we'll want to begin preparing our software for the new wind profiler sequence at whatever time it is planned to start being used on the GTS. (Thanks!)

[jitsukoh]

@jbathegit sorry for having scheduled the meeting without notice. It was an urgent request and we needed to adjust our procedure. I will make sure all the questions are answered on Github and also discussions at the teleconference will be recorded here to keep you in the loop.

[richardweedon]

The Revised versions of the proposal along with a summary of the questions asked in the intial review of the proposal with the corresponding answers is attached

[richardweedon]

BUFR responses.docx
windproflidar-amend-20200624.docx

[richardweedon]

see attached

[sergioh-pessoal]

I am interested in this topic and would like to attend the meeting on the 8th at 12 UTC.
Thanks

[jitsukoh]

@jbathegit A summary of the discussion today is here. The sequence will be amended and presented to the team as part of validation process.

[jitsukoh]

@chenxiaoxia2019 attached is the revised version of the BUFR sequence provided by @richardweedon based on our discussion on Wednesday. Could you create a branch for this proposal and pose questions if there are anything unclear?
wind_prof_prp_MT_AH.docx

[chenxiaoxia2019]

@jitsukoh Thanks. I have already created the branch for this issue. I will make changes according to the new version which is more clear. I will get you updated about the progress. @richardweedon

[SibylleK]

In the updated proposal the existing descriptors were changed in reference value and data width. But I thought it was "preferred to create new elements". What should be the finale decision here?
In addition the definition of the new descriptor 0 02 006 - "Upper Air Remote Sensing Instrument Type" is missing.

[marijanacrepulja]

I believe we decided to create new elements with reference to windproflidar-amend-20200624.docx
We also wanted to add 0 08 021 Time significance and set it to 2 with regards to date and time repeated twice in the header sequence.

[jitsukoh]

Thank you @SibylleK and @marijanacrepulja , yes, you're right. We decided to introduce new descriptors instead of changing existing ones and also use 0 08 021 time significance in the common header sequence. @richardweedon , could you update the document before producing sample data?

[richardweedon]

Marijana Apologies for the misunderstanding in the original proposal we will re-issue with the new Table B descriptors as originally discussed. We were less clear on the inclusion of the Significance qualifier , if we change the test data at this moment in time it will require more work than originally envisioned. The document put out by Enrico makes no mention of this. Seehttps://github.com/wmo-im/CCT/wiki/Teleconference-8.7.2020 Regards Richard From: Marijana Crepulja <notifications@github.com> Sent: 15 July 2020 17:25 To: wmo-im/BUFR4 <BUFR4@noreply.github.com> Cc: Weedon, Richard <richard.weedon@metoffice.gov.uk>; Mention <mention@noreply.github.com> Subject: Re: [wmo-im/BUFR4] Revision of E-Profile Sequences (#21) I believe we decided to create new elements with reference to windproflidar-amend-20200624.docx We also wanted to add 0 08 021 Time significance and set it to 2 with regards to date and time repeated twice in the header sequence. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#21 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AO5TCSP3SWNZPPQB4OURI73R3XJ4PANCNFSM4MZMWKYQ>.

[richardweedon]

Jitsukoh , Sibylle , Marijana I will amend the original proposal as below – Changes to Table B Descriptors BUFR CREX TABLE REFERENCE DATA DATA ELEMENT NAME UNIT SCALE REFERENCE WIDTH UNIT SCALE WIDTH F* X Y VALUE (Bits) Characters 0 15 073 replaces 0 15 063 Attenuated Backscatter m –1 Sr -1 8 -524288 20 m –1 Sr -1 8 7 0 15 074 replaces 0 15 065 Particle Backscatter Coefficient m –1 Sr -1 8 -524288 20 m –1 Sr -1 8 7 0 15 075 replaces 0 15 067 Particle Extinction Coefficient m1 8 -524288 20 m1 8 7 0 15 076 replaces 0 15 069 Particle LIDAR Ratio Sr 2 -2048 13 Sr 1 5 0 15 077 replaces 0 15 070 Uncertainty in LIDAR Ratio Sr 2 0 12 Sr 1 5 0 15 078 replaces 0 15 071 Particle Depolarization Ratio % 2 -8192 15 % 2 5 The changes above will require new D sequences as below. The proposal will be changed accordingly 309024 to replace 309021 309026 to replace 309023 I am in the process of amending the original proposal will re-issue this afternoon. Regards Richard From: jitsukoh <notifications@github.com> Sent: 16 July 2020 06:46 To: wmo-im/BUFR4 <BUFR4@noreply.github.com> Cc: Weedon, Richard <richard.weedon@metoffice.gov.uk>; Mention <mention@noreply.github.com> Subject: Re: [wmo-im/BUFR4] Revision of E-Profile Sequences (#21) Thank you @SibylleK<https://github.com/SibylleK> and @marijanacrepulja<https://github.com/marijanacrepulja> , yes, you're right. We decided to introduce new descriptors instead of changing existing ones and also use 0 08 021 time significance in the common header sequence. @richardweedon<https://github.com/richardweedon> , could you update the document before producing sample data? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#21 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AO5TCSITPHKZXUE5CTEFCTTR32HYFANCNFSM4MZMWKYQ>.

[amilan17]

@jitsukoh  -- Is this targeted for FT-2021-1?

[jitsukoh]

@amilan17 yes, it is. The team has already concluded to include this (table D entries) in FT2021-1.

[jitsukoh]

@richardweedon (cc @SibylleK @marijanacrepulja ) could you quickly confirm what was the final decision about the note given to the new Common header sequence? It appears:

(Common header sequence) (See Note 12)
(12) The common header sequence 3 01 132 should be incorporated into E-PROFILE Table D data sequences as detailed below.

but this is not appropriate because this sequence is already included in 3 09 024 - 3 09 026. I said on Sep. 28

I'd suggest updating the note to something like "The common header sequence 3 01 132 cannot be used on its own and must be followed by one of the sequences for windprofiler product data."

What would you say?

[SibylleK]

I would suggest to delete Note 12, as it is not needed anymore.

[jitsukoh]

@SibylleK Thank you for your suggestion. I agree! If there is not objection from @richardweedon I will proceed.

My apologies, can I ask if we need a Note for existing sequences 3 09 021, 3 09 022, and 3 09 023, which will be "deprecated" by introducing the 3 new sequences. The names are similar for old and new ones and I'm afraid that users would be confused.
My suggestion would be to add a Note reading "The sequences 3 09 021, 3 09 022, and 3 09 023 should not be used because they are outdated. The sequence 3 09 024, 3 09 025 and 3 09 026 respectively, should be used instead." to 3 09 021, 3 09 022, and 3 09 023.

[richardweedon]

Sybylle & Jitsuko
I have spoken to Simone and Myles. We are all in agreement that both notes should be withdrawn as suggested. Many thanks for your help in this matter

Richard

[jitsukoh]

@richardweedon thank you for your confirmation.

What about the note for existing sequences 3 09 021, 3 09 022, and 3 09 023?

[amilan17]

@jitsukoh -- please advise on the way to move forward with the notes for sequences 3 09 021, 3 09 022, and 3 09 023 - if we don't hear from @richardweedon by the end of the day. Thanks!

[richardweedon]

Anna / Jitsuko. I am trying to contact development team to finalise their response. In their absence I will have to say that we are happy to go with the following ,

The replacement of Note 12 with -

‘The common header sequence 3 01 132 cannot be used on its own and must be followed by one of the sequences used for E-Profile data'

In addition an extra note which relates to the deprecation of the class 9 sequences -

"The sequences 3 09 021, 3 09 022, and 3 09 023 should not be used because they are outdated. The sequence 3 09 024, 3 09 025 and 3 09 026 respectively, should be used instead."

[jitsukoh]

@richardweedon I understood that Note 12 to sequence 3 01 132 should be deleted. Is that correct?

[richardweedon]

To clarify the current Note 12 to 3 01 132 should be deleted and replaced with the entry

‘The common header sequence 3 01 132 cannot be used on its own and must be followed by one of the sequences used for E-Profile data'

[jitsukoh]

I don't believe so... We don't need this note, because this common header sequence can be used by other sequences.

[richardweedon]

Sorry Jitsuko, Myles has just corrected me. Please disregard my comments about note 12 and delete it completely. My apologies for the confusion

[jitsukoh]

Thank you @richardweedon for your confirmation.
I post the reply from @simonebircher for the record.

1. The note to 3 01 132 ("The common header sequence 3 01 132 should be incorporated into E-PROFILE Table D data sequences.")
   Our team's suggestion would be to remove this note completely, because now the product data sequences include this, and this common header sequence can be used in other sequences than E-Profile ones (as many of class 1 sequences).
   -> We fully agree that this note can be removed completely

2. An additional note to 3 09 021, 3 09 022, 3 09 023
   Can I understand that these existing sequences will not be used, once the new ones (3 09 024, 3 09 025 and 3 09 026) are introduced?
   If this is the case, I would suggest adding a note to clarify that these ones should not be used, because otherwise users would be confused about which they should use.
   My suggestion would be to add a Note, which reads "The sequences 3 09 021, 3 09 022, and 3 09 023 should not be used because they are outdated. The sequence 3 09 024, 3 09 025 and 3 09 026 respectively, should be used instead." to 3 09 021, 3 09 022, and 3 09 023.
   -> We fully agree with you that adding such a note to the sequences 3 09 021, 3 09 022 and 3 09 023 would be very useful in order to avoid confusion

[jitsukoh]

@amilan17 please go forward with the conclusion above.

For the second one, Note 13 "The sequences 3 09 021, 3 09 022, and 3 09 023 should not be used because they are outdated. The sequence 3 09 024, 3 09 025 and 3 09 026 respectively, should be used instead." will be added to Category 09, associated with 3 09 021, 3 09 022, and 3 09 023.

[amilan17]

Thank you.

[amilan17]

I added Note 20 to sequences 3 09 021, 3 09 022 and 3 09 023.

@jitsukoh - Please take a look at BUFR_TableD_en_09.csv

https://github.com/wmo-im/BUFR4/compare/FT21-1-validation#diff-10293517c4fbcdc25c51ebcb9a033a10b37d868d4782db6de1b4c375124136d5

[jitsukoh]

@amilan17 I think the concept is correct.
Sorry I don't fully understand how the notes are handled in Github tables, but how you added Note 20 seems a little different from e.g. Note 1:
09,Vertical sounding sequences (conventional data),309030,(Ozone sonde flight data) (see Note 1),,015004,Ozone sounding correction factor (CF),,,Operational

09,Vertical sounding sequences (conventional data),309021,(Single wavelength wind profiler wind data (product data)),,301001,WMO block and station numbers,,"(20) The sequences 3 09 021, 3 09 022, and 3 09 023 should not be used because they are outdated. The sequence 3 09 024, 3 09 025 and 3 09 026 respectively, should be used instead.",Operational
09,Vertical sounding sequences (conventional data),309022,(RASS virtual temperature (product data)),,301001,WMO block and station numbers,,See Note 20,Operational

[amilan17]

@jitsukoh Unfortunately, all of the notes in GitHub tables are placed inconsistently. Right now, it doesn't break anything, but it's something we'd like to fix in the future. For the moment, we only need to ensure that the content of the note is correct and they are displayed in the amendment doc and the manual correctly.

[jitsukoh]

@amilan17 thank you for the explanation. Then we're good to go! I'll check again in the amendment doc.