new entries in BUFR flag table 0-33-066

[jbathegit]

Branch

https://github.com/wmo-im/BUFR4/tree/issue-7

Summary and purpose

This document proposes 2 new entries in BUFR flag table 0-33-066.

Action proposed

The team is requested to review and approve the new entries for inclusion in the November 2020 fast-track updates.

Discussions

As part of their ongoing efforts to mitigate the negative effects of the loop-heat pipe failure and provide more useable data from the advanced baseline imager (ABI) on board the GOES-17 satellite, NESDIS is retooling some of their algorithms used to calculate derived-motion winds. Accordingly, they request 2 new entries in BUFR flag table 0-33-066 in order to help end-users better understand any nuances associated with these data values.

Detailed proposal

Add new entries to flag table 0-33-066 "AMV Quality Flag"

Bit	Description
20	Good wind, but an alternative channel used for feature tracking
21	Good wind, but an alternative set of channels used for the determination of cloud-top height/AMV height assignment

[SimonElliottEUM]

I appreciate the allocation of the high order bits first - this will help with the subsequent compression, just as we have discussed

[marijanacrepulja]

Thanks @jbathegit for proposing additional AMV Quality Flag. I have asked my colleagues in research who are working with the data about the wording and they would appreciated if it can be reformulated as bellow.

1. The use of the phrase “mitigated band” seems unclear. The impression it gives is that a normally used channel has been altered in some way (subject to the mitigation strategy) rather than being totally replaced by another channel which I believe is what is actually happening. Perhaps something like “Good wind but modified channel selection used for…” or “alternative band for feature tracking” could be clearer?

2.The inclusion of the first listed flag “Good wind. Passes all quality control internal to the AMV algorithm.” seems unnecessary. At present, if the AMV has not passed the tests, then my understanding is that it is not disseminated. By default, if we’re receiving a wind, then its already deemed “good”. All producers would essentially have to fill this bit in the flag. Maybe there is a reason why NOAA wanted to include this? In future, information on failed internal tests might be useful if the data are being disseminated but a more specific description might be desirable at that point.

[jbathegit]

Thanks Marijana for the review by you and your colleagues. I've forwarded your comments to my NESDIS colleagues and will let you know what they say.

[jbathegit]

@marijanacrepulja, just wanted to give you a heads-up that NESDIS has reformulated their request based on ECMWF's comments. The proposed bit 18 entry has now been removed, and the other entries further clarified, including what was really meant by "mitigated band". Please let us know if there are still any concerns or confusion, and thanks again for your careful review of our proposal!

cc @SimonElliottEUM

[marijanacrepulja]

@jbathegit many thanks for taking the feedback on-board!

There are a few more questions.
Does this mean a different channel frequency will be given in the BUFR field for the channel frequency as well when 19 (or 21) are set? Or it could be that the "intended" frequency is given (remaining the same all the time), and the flag 19 (or 21) is set to indicate that a different one is used when there is degradation in the intended channel.

cc @SimonElliottEUM

[jbathegit]

@marijanacrepulja - yes, a different channel frequency will be given in the BUFR field for the channel frequency when bit 19 (or 21) are set. So the frequency you see will be what was actually used.

cc @SimonElliottEUM

[jitsukoh]

@jbathegit I add a note based on Sibylle's comment at yesterday's teleconference.
Bit 21 might not be necessary because it can be represented by using 19 and 20 together.
This will be confirmed and necessary changes will be made as part of the validation.

[jbathegit]

@jitsukoh - thanks, and rest assured we haven't forgotten about this. I agree that this makes sense, and I've already been in touch with my NESDIS colleagues and they agree as well. In fact, we're all surprised (and a bit embarrassed ;-) that we didn't catch this ourselves, so much thanks to @SibylleK for her careful review and comments!

For now, I've gone ahead and updated the proposal in the above block, but I'm still waiting for some updated samples from NESDIS, and I'll post those for validation as soon as I have them.

[jbathegit]

@wmo-im/tdcf - as requested for validation, NESDIS has posted samples of GOES-17 wind data utilizing the proposed new bits in BUFR flag table 0-33-066. You can find them at ftp://ftp.star.nesdis.noaa.gov/pub/smcd/jpss-ait/GOES17_Winds_Algorithm_Mitigations/ as filenames "NB-GOES17_ABI_2KM_FD_YYYYJJJ_HHMM_SS_WINDS_AMV_EN-14-CT.bufr", along with corresponding "NB-GOES17_ABI_2KM_FD_YYYYJJJ_HHMM_SS_WINDS_AMV_EN-14-CT.bufr_encoded" files showing what the decoded output values should be.

There are 4 samples in total, where the YYYYJJJ_HHMM_SS values are as follows:

1. Good wind: 2019242_0800
2. Good wind, but an alternative channel used for feature tracking (bit 20): 2019242_1000
3. Good wind, but an alternative set of channels used for the determination of cloud-top height/AMV height assignment (bit 21): 2019242_1030
4. Good wind, but an alternative channel used for feature tracking (bit 20) AND an alternative set of channels used for the determination of cloud-top height/AMV height assignment (bit 21): 2019242_1100

[chenxiaoxia2019]

@jbathegit Hi, Jeff, I have already created a branch for this issue. Could you please check it? Thanks.

[jbathegit]

Thanks @chenxiaoxia2019 The branch looks fine to me. In fact, it doesn't seem as though the latest commit 2f3ee16 was really even necessary, because I think the parent commit 07bf06a (which is the same text content but without the surrounding quotation marks) would still have worked fine within the .csv format. But again, the new branch looks fine to me, and thanks!

[jbathegit]

Hello again @chenxiaoxia2019, and please disregard my last comment as I see now why the quotation marks were needed in commit 2f3ee16 - because the text content already had embedded commas! So I agree now that the quotation marks were needed, and again everything checks out OK.

[marijanacrepulja]

Hello @jbathegit Jeff,
I have decode samples provided using ecCodes and tables from the branch.
Here is the output of the decoding.
AMV_qualityFlag.tar.gz

[jbathegit]

@marijanacrepulja Thanks very much for your help to validate this proposal! I spot-checked a few subsets in your output, and it looks like your output values are correct.

[amilan17]

Approved by FT-2020-2.