Weather radar

[jdemaeyer]

With v2.0.2, a new /radar endpoint landed in Bright Sky. 🎉

This endpoint contains

• rainfall radar data
• on a 1200 km (height) x 1100 km (width) grid
• with each pixel corresponding to 1 km²
• in 5-minute intervals
• including a forecast for the next two hours.

This data comes from the DWD's RV radar composite product.

Please feel very welcome to add feedback and suggestions to this issue!

Quickstart

This request will get you radar data near Münster, reaching 200 km to the East/West/North/South, as a two-dimensional grid of integers:

https://api.brightsky.dev/radar?lat=52&lon=7.6&format=plain

API structure

• The new endpoint is available at https://api.brightsky.dev/radar

• The response looks roughly like:

  {
    "radar": [
      {
        "timestamp": "2023-05-09T11:05:00+00:00",
        "source": "RADOLAN::RV::2023-05-09T11:05:00+00:00",
        "precipitation_5": "... base64-encoded bytestring or array of integers, see below ..."
      },
      ...
    ],
    "geometry": {
      "type": "Geometry",
      "coordinates": [
        [1.4633,55.86209],
        [3.56699,45.69643],
        [16.58087,45.68461],
        [18.73162,55.84544],
      ]
    }
  }
  

• Allowed parameters (all of them optional):

  • format: determines how the precipitation data is encoded into the precipitation_5 field (default: compressed):
    • compressed: base64-encoded, zlib-compressed bytestring of 2-byte integers
    • bytes: base64-encoded bytestring of 2-byte integers
    • plain: Nested array of integers
  • bbox: bounding box top, left, bottom, right in pixels, edges are inclusive (default: full 1200x1100 grid)
  • lat / lon / distance: alternative way to set a bounding box, where lat / lon will lay in the center pixel of the returned radar data, which will reach distance meters to each side of this pixel (see example request below). distance can be omitted and defaults to 200000 (i.e. 200 km). The exact x-y-position of the supplied lat/lon will be returned as latlon_position
  • date: ISO8601 timestamp of first record (default: 1 hour before latest measurement)
  • last_date: ISO8601 timestamp of last record (default: 2 hours after date)
  • tz: timezone to be used for timestamps (default: UTC)

Parsing examples

The radar data is quite big (naively unpacking the default 25-frames response into Python integer arrays will eat roughly 125 MB of memory), so use bbox whenever you can.

compressed format

With Python using numpy:

import base64
import zlib

import numpy as np
import requests


resp = requests.get('https://api.brightsky.dev/radar')
raw = resp.json()['radar'][0]['precipitation_5']
raw_bytes = zlib.decompress(base64.b64decode(raw))

data = np.frombuffer(
    raw_bytes,
    dtype='i2',
).reshape(
    # Adjust this to the height/width of your bbox
    (1200, 1100),
)

With Python using the standard library's array:

import array

# [... load raw_bytes as above ...]

data = array.array('H')
data.frombytes(raw_bytes)
data = [
  # Adjust `1200` and `1100` to the height/width of your bbox
  data[row*1100:(row+1)*1100]
  for row in range(1200)
]

Simple plot using matplotlib:

import matplotlib.pyplot as plt

# [... load data as above ...]

plt.imshow(data, vmax=50)
plt.show()

bytes format

Same as for compressed, but add ?format=bytes to the URL and remove the call to zlib.decompress, using just raw_bytes = base64.b64decode(raw) instead.

plain format

This is obviously a lot simpler than the compressed format. It is, however, also a lot slower. Nonetheless, if you have a small-ish bbox the performance difference becomes manageable, so just using the plain format and not having to deal with unpacking logic can be a good option in this case.

With Python:

import requests


resp = requests.get('https://api.brightsky.dev/radar?format=plain')
data = resp.json()['radar'][0]['precipitation_5']

Content

• The grid is a polar stereographic projection of Germany and the regions bordering it. This is different from the mercator projection used for most consumer-facing maps like OpenStreetMap or Google Maps, and overlaying the radar data onto such a map without conversion will be inaccurate!
• The DWD data does not cover the whole grid! Many areas near the edges will always be 0.
• Values represent 0.01 mm / 5 min. I.e., if a pixel has a value of 45, then 0.45 mm of precipitation fell in the corresponding square kilometer in the past five minutes.
• The four corners of the grid are as follows:
  • Northwest: Latitude 55.86208711, Longitude 1.463301510
  • Northeast: Latitude 55.84543856, Longitude 18.73161645
  • Southeast: Latitude 45.68460578, Longitude 16.58086935
  • Southwest: Latitude 45.69642538, Longitude 3.566994635
• The latitude/longitude for the center of any grid cell (pixel) can be found in this file.

You can find details and more information in the DWD's RV product info (German only). Below is an example visualization of the rainfall radar data taken from this document, using the correct projection and showing the radar coverage:

Notes and Questions for Feedback

• This is an experimental feature and the interface is very much subject to change. Please regularly check this issue if you're using it.
• Use a bbox and the default compressed format if possible -- this'll get you the fastest response times by far and reduce load on the server. If you have a small-ish bounding box (e.g. 250 x 250 pixels), using the plain format should be fine.
• I'm not sure how useful the bytes format is. The idea is to save the hassle of parsing a giant JSON array on the client side, but maybe that's actually not too bad?
• The polar-stereographic projection that this data uses makes a lot of sense from a data perspective (as each pixel represents the same area) but is rarely used in maps. If you're looking for tiles you can overlay on a map, take a look at the dwd:RV-Produkt layer on the DWD's open GeoServer.
• The conversion between pixel x/y coordinates and lat/lon coordinates is quite unwieldy. Many users will probably just want to know "what's the rainfall like close to my location", so maybe alternative parameters for setting the bounding box by supplying the lat/lon of its center and a maximum distance could be a simple interface for this?

Resources

• Coordinates for the center of every grid cell
• Script that generated the above coordinates
• Raw data on the Open Data Server
• Details on the RV product (German)
• Visualization of current rainfall radar
• General info on DWD radar products (German)
• Radar status (German)
• DWD notifications for radar products (German)

[poetaster]

As you suggest in your last comment on conversion, I'm thinking of a first simple bounding box from the current location (or a user supplied location). I'm rendering in QT and suppose I could render directly to canvas (the QML type derived from javascript). If I understand correctly, that would be a trivial task of just taking the matrix for the area (say 512x512 pixels) and directly rendering the floats as 'my favourite color pallette'?

[ntadej]

Note that DWD also has a GeoServer instance which is much nicer to be used (and if I understand correctly can be used directly by clients). Then supported mapping engines can render it more efficiently (although it's still bitmaps). I use that in my Qt-based app.

[poetaster]

Then supported mapping engines can render it more efficiently (although it's still bitmaps).

I'm currently using a webview using leaflet and the data from rainviewer.com (which consumes the DWD data) but it's both slow and, obviously, not native. I guess I could go to rendering something like the Composite openlayer view: https://maps.dwd.de/geoserver/dwd/wms?service=WMS&version=1.1.0&request=GetMap&layers=dwd%3AWN-Produkt&bbox=-543.462%2C-4808.645%2C556.538%2C-3608.645&width=703&height=768&srs=EPSG%3A1000001&styles=&format=application/openlayers#toggle

Is that what you mean? I don't actually want to use a mapping engine since I'm a mobile user and want to conserve resources. For directly rendering, say, via leaflet.js I think you're obviously correct, but what if I just want a simple/naive representation?

[jdemaeyer]

If I understand correctly, that would be a trivial task of just taking the matrix for the area (say 512x512 pixels) and directly rendering the floats as 'my favourite color pallette'?

That's how I would do it, yeah. I am however not very experienced in handling map data, unfortunately. My intuition is this: If the canvas already contains a map that you render on top of, and you aligned the center of the map (i.e. the user location) to the center of the radar data matrix, the location mismatch should be negligible for the center pixels but will become increasingly bigger for locations away from the center—I don't know by how much, though. However, for a naive "I just wanna know if there's rain coming up at my location" that's probably Good Enough™?

[jdemaeyer]

Before it gets lost in the wall of text I posted above: Here is a file containing the center coordinates for each pixel.

I will probably use those (or rather, the mathematical conversion outlined in this document on page 13) to implement center_lat/center_lon and width/height parameters if those requests turn out to be a major use case.

[poetaster]

However, for a naive "I just wanna know if there's rain coming up at my location" that's probably Good Enough™?

I think I'll need to compare the actual data. It could well be that overlay data from DWD is smaller, but the additional map js/tile data again competes. I'll have to try it to see how it looks. Thanks!

[jdemaeyer]

Sounds great, let me know how it goes! :)

[jdemaeyer]

I've added lat, lon, and distance parameters as an easier way to set a bounding box. This request will return a pixel grid with Münster in its center, reaching 200 km in every direction:

https://api.brightsky.dev/radar?lat=52&lon=7.6&format=plain

(The distance in each direction can be adjusted by supplying distance in meters, the default is 200000)

[AndiLeni]

I really like to see this feature.
However, I can not see how I may use this to easily show a map to the user.

What I really would like to see is this Radardata as a geojson or SVG image, since the official dwd map proxy only provides them as png/jpg and not in a scalable format.

But thanks nonetheless for this awesome feature!

[jdemaeyer]

Thanks for the feedback @AndiLeni. I was afraid this feature will not see much use as it's not easy to integrate into map applications in its current form (I've had a heated discussion with @ubergesundheit over lunch about this as well 👋). Bright Sky is very much a service for numerical weather data, and I fear leaving that core competence and making it into a "numerical weather data but then also some vector map tiles service" will open the gate to a whole range of development/maintenance nightmares.

That being said, providing the radar data as GeoJSON seems close enough to what we do currently that we could sneak it in. What would radar data look like in this format, a giant list of Polygon features, with one polygon for each grid cell (pixel) of the radar data?

[AndiLeni]

@jdemaeyer I am honestly not sure what might be the best.
Aggregating several polygons with the same data-information to a larger one would be the most ideal way to keep the geojson simple and lightweight.
But this would probably require a lot of preprocessing (?)

It might be good to try it with leaflet or openlayers, so see how performant or good it is.

Would it make you a lot of work to deliver geojson for evaluating this?