API responses very slow since about 3 pm CEST today

[BeyondVertical]

Hey there,
I am using your API and am running into timeouts since about 3 pm today.
Your Grafana dashboards shows this too. Can you have a look at the server, when you have the time?
Thanks for this great API and your work!

[tawissus]

True. For me, the last time access worked was around 16:00.

[BeyondVertical]

Yeah. Access works again now - shortly after midnight there were no timeouts. Thanks for fixing! :)

[jdemaeyer]

Hi guys, thanks for the report!

As you said, response times stabilized shortly after midnight. That's when most users changed their queries from date=2023-08-20 to date=2023-08-21 -- however, it doesn't seem that there were any changes in the underlying data in the middle of Sunday that would massively impact database query time like this. So I guess it was a hiccup that got reset by some daily maintenance task or rotation.

I'm swamped with work right now so can't investigate deeply, but I'll keep a closer look on the server throughout the day (my monitoring tools actually did pick up the outage yesterday, I just wasn't reachable 🙈)

[smilingchemist]

The problem seems to be back. Last record is from yesterday, 14:30 UTC...

[BeyondVertical]

The problem seems to be back. Last record is from yesterday, 14:30 UTC...

Yeah. Was gonna say this too.
But it is a little different than last time. This time the response time seems fine, but the data is missing.

[jdemaeyer]

Thanks for reporting! New data is currently being parsed and should be available in the next 30 minutes.

This issue originated from a prolonged outage of dwd.de (not the open data server but their official website, from which we load a station list as a prerequisite for parsing) that we should build resilience against. I'll open a new issue in a bit

[jdemaeyer]

#154