Table D Class 40: New BUFR template for cloud and aerosol observations from satellite lidar

[marijanacrepulja]

Branch

https://github.com/wmo-im/BUFR4/tree/issue91

Summary and purpose

ECMWF is proposing a new BUFR template for representation of satellite lidar observations for clouds and aerosols.

Stakeholders

Marijana Crepulja, Mark Fielding, Marta Janiskova, Angela Benedetti , Julie Letertre-Danczak (all from ECMWF)

Action proposed

The team is kindly asked to review and approve the contents for inclusion within the next update to the WMO Manual on Codes.

Discussions

ECMWF is preparing for monitoring and operational assimilation of cloud and aerosol observations from EarthCARE . Therefore a new BUFR template is needed.

Detailed proposal

1. Add a new entries in code table 0 05 069 Receiver Chanel

CODE TABLE 0-05-069
0-05-069 Receiver Chanel

Code Figure	Receiver Chanel
2	Cross polar

2. Add a new entry in the Processing information class of table B (0-25-130) and a new code table to describe the Processing level
(also in Issue #90)

F X Y	ELEMENT NAME	UNIT	SCALE	REFERENCE VALUE	DATA WITH (bits)
0-25-139	Processing level	Code table	0	0	5

3. Add new CODE TABLE 0-25-139 for Processing level
(also in Issue #90)

Code Figure	Processing level
0	L1B
1	L2A
2-30	Reserved
31	Missing

4. Add a new BUFR template for lidar observations from satellite

Table reference	Table references	Element name	Description / comments
3-40-026		Lidar observations from satellite
	0 01 007	Satellite Identifier
	0 02 019	Satellite Instruments
	0 01 033	Originating Centre
	0 01 034	Originating Sub Centre
	3 01 011	Year, Month, Day
	3 01 013	Hour, Minute, Second
	3 01 021	Latitude/longitude (high accuracy)
	0 10 033	Altitude (Platform To Ellipsoid)
	0 08 043	Atmospheric chemical or physical constituent type
	0 25 139	Processing level
	0 02 155	Satellite channel wavelength
	1 12 000	Delayed replication of 12 elements
	0 31 002	Extended Delayed replication factor
	0 07 071	Height
	0 33 003	Quality information
	2 01 136	Increase width
	0 15 074	Particle Backscatter Coefficient
	0 15 066	Uncertainty In Particle Backscatter Coefficient
	0 15 075	Particle Extinction Coefficient
	0 15 068	Uncertainty In Particle Extinction Coefficient
	2 01 000	Cancel increase width
	0 15 076	Particle lidar Ratio
	0 15 070	Uncertainty in lidar ratio
	0 15 078	Particle Depolarization Ratio
	0 15 072	Uncertainty in depolarization ratio
	1 07 003	Delayed replication of 7 elements
	0 05 069	Receiver Chanel
	1 04 000	Delayed replication of 4 elements
	0 31 002	Extended Delayed replication factor
	2 01 135	Increase width
	0 15 073	Attenuated Backscatter
	0 15 064	Uncertainty in Attenuated Backscatter
	2 01 000	Cancel Increase width

[amilan17]

@amilan17 - research issues with CODE TABLE 0-05-069
0-05-069 Receiver Chanel (PDF shows a different table name: Averaging period for following value)

[amilan17]

@amilan17 - error in title of CODE TABLE 0-05-069
0-05-069 Receiver Chanel

[marijanacrepulja]

I have updated the proposal following discussions

1. Rename new descriptor 'Processing level flag' to 'Processing level' and increase width, thanks to @SimonElliottEUM
2. Instead of new descriptor 'Height of observations' in class 10, propose new descriptor 'Height' in class 7, thanks to @jbathegit
3. New sequence for Lidar satellite observations to be in class 40, thanks to @jbathegit

[amilan17]

@marijanacrepulja talk with colleagues about the negative height in 0-07-073. What is it relative to?

[amilan17]

Is it "legal" to move/change the Code figures for Reserved and Missing in Code Table 0-05-069? 

[amilan17]

@marijanacrepulja BUFR table is not in branch yet.

[marijanacrepulja]

I have updated branch with new entries.
Sample data and corresponding output of decoded BUFR file is attached.

@marianmajan-ibl and @sergioh-pessoal could you please help with validation. Many thanks.

[marijanacrepulja]

@marijanacrepulja talk with colleagues about the negative height in 0-07-073. What is it relative to?

Following a review of a new set of test data and consultation with colleagues from research, we decided to use existing BUFR descriptor 007071

I have removed proposed descriptor 007073

[marijanacrepulja]

Is it "legal" to move/change the Code figures for Reserved and Missing in Code Table 0-05-069?

@amilan17 it is legal, missing value is remain the same as per definition. Reserve means that number should be used for a new entries.

[sergioh-pessoal]

Hello @marijanacrepulja and @marianmajan-ibl

In the BUFR sample, the descriptor 0-15-068 was, apparently encoded with 28 bits
On the other hand, this part of the template above (.. 2-01-132, 0-15-068,) indicates that 0-15-068 has 24-bit

I don't remember if the correctly would be (.. 2-01-132, 0-15-068,.. ) or (.. 2-01-136, 0-15-068,.. )

Anyway, I decoded the BUFR sample , by changing 2-01-132 to 2-01-136 in my table D

The BUFR sample was decoded correctly with mbufrtools.
The text format of the files is different , but I compared the values and all values are Ok

The result is attached
EarthCARE-lidar.bufr.decoded_mbufr.txt.gz

[marijanacrepulja]

Hello @sergioh-pessoal,

Many thanks for your feedback. Yes, you are right, the descriptor 0-15-068 was encoded with 28 bits and operator 2-01-136 should be used.

Because my local definition of 0-15-068 was incorrect, with a width of 24 instead of 20, I made the error of using operator 2-01-132.
In the branch, I've changed the proposal and tables.

I am uploading a new test data samples.

EarthCARE-lidar_test.gz

Could you please decode samples once more to ensure everything is in order?
Once again, thank you for your time and please accept my apologies.

[sergioh-pessoal]

Hello @marijanacrepulja

Yes of course

I've use the table in the branch and decoding was successful.
The decoding file is attached.

best regards

Sergio

EarthCARE-lidar_test.bufr.decoded-100.txt.gz

[marianmajan-ibl]

Hello @marijanacrepulja, I was able to decode data using the tables from the branch. It is quite complicated to compare data with your samples as you do have it decoded in array form, however the manual check of few messages showed the same results. The decoded file's size is 3,5 GB and even the zip with the highest compression is too big to be attached here (7z was small enough but is not allowed to be attached). I have uploaded the zip archive to our GDrive -- https://drive.google.com/file/d/160iKNpLtsX2X_N_e8OkLckFtcBjuUA6t/view?usp=sharing

[marijanacrepulja]

@marianmajan-ibl and @sergioh-pessoal Thanks very much for your help to validate this proposal!

[jitsukoh]

@marijanacrepulja @amilan17 I confirm that the branch is updated and this issue is moved to the validated status.