Add support for forecast information

[amotl]

Introduction

We would like to add support to acquire forecast information from DWD, for both MOSMIX-L and MOSMIX-S.

Background

@drmrbrewer and @onlygecko already asked for this within panodata/dwdweather2#2.

Thoughts

Together with @jlewis91, we already planned to merge dwdweather2 with dwdbulk somehow, but now we should consider to add missing functionality here.

@jlewis91' convert_xml_to_pandas method from forecasts.py might be used 1:1 already. Note that this one has originally been written to read MOSMIX-S only.

[meteoDaniel]

@amotl I have built a parser but it is not advanced in the moment. I can provide the corresponding ode and we can build up a more powerful parser.

[amotl]

@FL550 just added another implementation to grok MOSMIX-L through dwdforecast.py.

[amotl]

Comparison

What I like from @jlewis91' implementation forecasts.py is that it already uses Pandas. However, we will have to expand it to also parse MOSMIX-L, currently it only does MOSMIX-S. Also, it extracts the list of .kml files into a local directory. We should do that in-memory.

What I like from @FL550's implementation dwdforecast.py is that is does everything in-memory already and that it maps the technical meteorological element names to more meaningful field names (see #54). Also, it features a mapping to resolve the weather condition codes to meaningful textual labels.

Both implementations use lxml.etree for parsing the XML.

Resources

• The KML format description is available at Data Format of MOSMIX Forecasts.
• A number of elements from the MOSMIX-Elemente table is already accessible in machine-readable format through MetElementDefinition.xml.
• Mapping information for the nu­mer­i­cal weath­er pre­dic­tion mod­els - the "significant weather condition" code from field ww - is available at Weather interpretation for the ICON/ICON-EU and COSMO-D2 models as well as Used weather condition codes in the MOSMIX-forecasts. This topic is also discussed within Weather condition jdemaeyer/brightsky#85.

[amotl]

Hi there,

within #114, I've added a preliminary spike the other day, mostly based on the original dwdbulk::forecasts implementation by @jlewis91 (thanks again!).

I believe we should use this issue here to discuss the future evolution of this feature, even beyond the lifetime of #114.

Backlog

General

• Add software tests.
• Update documentation.
• Adapt get_nearby_stations for MOSMIX.
• Integrate into command line interface.
• Integrate into HTTP API.
• Add docstrings.

Metadata

Adding parameters, column names and such into the metadata/knowledgebase subsystem at wetterdienst/constants would also be awesome if that actually makes sense. I see these details here:

• The list of obtained parameters are only strings for now and not Enum-based yet. We might want to map this for both MOSMIX-L and MOSMIX-S, see mosmix.py#L303 vs. mosmix.py#L310.

• Output column names should be mapped to sensible Enum-based symbols as well as meaningful humanized English names, see Using meaningful column/field names #54. When implementing this, we might want to revisit Using meaningful column/field names #54 (comment), where I referenced some work from @meteoDaniel (thanks!) which might help us here.

  Borrowing from GribMagic

  @meteoDaniel also added some humanized labels through unified_forecast_variables.py, thanks already! Wetterdienst might also draw some inspiration from this.

  Remark: For observation data, we added retrieving metadata about column names from PDF files on the DWD data repository as outlined within Obtaining metadata about specific data sets #112 through DWDObservationMetadata API #199.

• Things like coerce_columns are still in their infancy.

New feature: Station concordance

Now, @gutzbenj asked at #114 (comment):

Also as we have talked about the station ids, it would be awesome to have a translating system from regular DWD station ids to MOSMIX station ids so that one could use the same id for historical observations and MOSMIX forecasts.

That would definitively be awesome. I believe @wetterfrosch did this concordance thing in a hands-on manner while he implemented the Panodata DWD Meteogram. He might be interested in helping us to bring this subsystem into Python land.

With kind regards,
Andreas.