ENRAM meeting Norrkoping

#Meeting minutes ENRAM workshop meeting at SMHI, Norrköping, 12-15 July 2016

Participants:

• AD = Adriaan Dokter
• AH = Anders Henja
• GU = Günther Haase
• CN = Cecilia Nilsson
• LV = Liesbeth Verlinden

These minutes provide on overview of discussion points, findings and actions taken during the meeting. In the meeting we combined short discussion sessions and actions to directly solve identified problems.

Main aim of the meeting was to discuss specifications and immediately implement a production pipeline that:

1. extracts bird profiles for the entire OPERA network
2. makes these bird products available on a server
3. stores the bird products in a long-term repository for further analysis.

We reached the milestone of set up and testing this production pipeline. Pending final permission from OPERA to run the production pipeline, it is currently running only for the data from Belgian and the Netherlands - which are the countries that already publish bird profile data via http://www.flysafe-birdtam.eu.

###1. Discussing / setting up a production pipeline of bird profiles

1. Bird algorithm (vol2bird) was installed both in the Baltrad system (nordic countries), as well to act on the OPERA data stream (these are two separate data streams at SMHI)
2. A Python wrapper for vol2bird and python scripts was installed on the production server at SMHI to generate bird profiles for newly incoming radar data (AH, AD)
3. An FTP service was set up to make bird data available to the outside world (AH). We decided to have the data available there for two days, after that they are overwritten.
4. We decided on a prototype service to grab the data from the SMHI FTP server. AD has set up a repository at beehub, and installed a script at a server at the university of Amsterdam to grab these data, and store them on beehub. The pipeline was tested for 3 days for a total of 100 radars as a proof of concept. The data stream has then be stopped, pending official OPERA permission to use the data. The pipeline will be continued to be tested for data from the Netherlands and Belgium, for which the permission is already obtained.
5. A Baltrad node has been set up at the Swedish national supercomputing centre (NSC). At the NSC also the historical data of OPERA are archived. The bird algorithm has been installed on the node, such that archived data can be processed much more easily.
6. A critical technical issue in the Baltrad system was identified, which causes OPERA data to be overwritten after entering SMHI. As a result, currently only for 3 countries complete data from OPERA has been properly stored.
7. We discussed definition of nominal time in OPERA. Currently the nominal time is the time at which the polar volume was created at Baltrad, i.e. the time of receiving the data, not the time of collecting the data. As a solution, AD added an option to vol2bird that reads start and end time attributes of all scans contained in the polar volume, and stores their extrema in the bird profile output.
8. AD implemented dialiasing of radial winds by the method of Haase et al., as available in the Baltrad toolbox. We found that many countries still provide only aliased data (e.g. Denmark, Finland), therefore proper dealiasing is a critical feature needed for large-scale analyses.
9. ACTION Evaluate the quality of dealiased VRAD in the case of bird migration (AD/LV)
10. ACTION As an intermediate solution, GH will attempt before 1 Aug to apply a fix that stores the incoming data, to secure data for the upcoming migration season.
11. ACTION A ticket has been made for the Swedish radar network to fix this issue, as it is more general than ENRAM alone (e.g., also the Numerical Weather Prediction community)
12. ACTION store nominal time instead of time of receiving the data (GH)

###2. Discussing / evaluating bird profile quality

1. Two 2x 1 week of data was selected (1 week spring and 1 week autumn) as a test (CN/GH/LV). Only for three countries (SE, NL, FR) both radial velocity and reflectivity data was available, for which bird profiles could be extracted
2. 2 weeks test case data was processed and visualised (LV/CN). We found that for many countries (meta)data was missing, and that radial velocity data was regularly folded (e.g. Denmark, Finland).

###3. Kullaberg validation campaign / other papers

1. Cecilia provided an update of analyses so far. Correspondence between weather radar and tracking radar looks good, but weather radar suffers from a few short but intense rain contaminations.
2. Decided on rules to filter out residual rain in Kullaberg data. We applied an additional post-processing filtering step for the weather radar data, in which altitude profiles are removed when at least 80% of the altitude layers in the 0-2 km band contains rain. This improved the nightly accumulations of bird density as compared to the Birdscan and tracking radars.
3. ACTION LV will look at weather radar speed biases with altitude and density. Results will be shared with CN so that they can become part of the paper.
4. ACTION CN will share tracks and balloon winds with LV
5. ACTION Günther will check whether HIRLAM wind profiles for the Kullaberg site are readily available, and if so make them available for the project.
6. AD/CN brainstormed on swift paper, and set a deadline of 1 jan 2017 for a finished draft.

###4. Progress license OPERA - ENRAM

• ACTION AD has sent email to Elena Saltikoff with results of this meeting, and requested an update on the licensing process. Licenses are decided by the general assembly of EUMETNET, i.e. a decision level above OPERA. AD will discuss with Elena whether a license should include an option to give permissions for profiles, volumes, and composites separately.

###5. North Sea / western Europe dataset We defined the critical actions needed to compile a large-scale migration case, being a main output / goal of the ENRAM COST action.

Highest priority

• signing a data use license with OPERA
• Fix problem overwriting of DBZ/VRAD files (see actions under 1 above, SWERAD)

Second priority

1. ACTION AD, GH will report to OPERA on findings of the 2 weeks of test data, and data quality issues. This document can also be sent to the OPERA user group. Deadline is 1 Aug, for it to be ready before next OPERA meeting In this note we should list:
   • Push OPERA to store Nyquist velocity
   • Push OPERA to store wavelengths
   • Move to 2 weeks test case data to full autumn 2016, due to lack of data
   • Data quality/filtering
2. ACTION Make configuration options settable per radar (AD)
3. ACTION Add de-aliased VRAD to polar volumes generated in Baltrad by default (SVERAD)
4. Evaluate de-aliased wind data for bird migration cases.

The current biggest open technical issue is the overwriting of the DBZ/VRAD files within the OPERA data stream within SMHI. BALTRAD does not have the overwrite issue, so in principle it would be possible to get workable data from Nordic countries (Norway and Finland) via BALTRAD. We will have to ask these two countries explicitly if we can access their data via BALTRAD and not via OPERA, as the two are different data streams.

We decided we will update vol2bird on BALTRAD and start to run it already there. Once we got the permissions, we can access the data then. At BALTRAD data is only stored for limited (unknown) time, though this time period will be increased to make sure we will have data for the autumn migration of 2016.

GH will do his best to fix the overwrite issue before the start of autumn migration, at the earliest some time in September, but no hard guarantees on the timeline are possible. It depends on when people (Anders and colleagues at SMHI) will be back from holiday and their workload ob other higher priority projects.

End of August, beginning of September we will evaluate the progress of achieving the license. In case it seems this could take quite some time, we will approach the countries around the North sea and ask them individual permission to use data of a limited time interval, e.g. 2 months.

Holidays:

• AH until 2nd week August
• GH 1/8 - 4/9
• LV 4/8 - 18/8
• CN not
• AD starting in US 29/7