TimeMachine data only available before May 2023

[NJAldwin]

Describe the bug

The documentation says that the time machine cannot be used for historic data within the last 5 days, and to use the forecast API instead.

Because "the last five days" isn't exactly defined, I want to make sure I can build in error handling.

If I try to hit the forecast API with a date that's too far in the past, it gives me a 400 Bad Request with the body "Requested time too early, please use https://timemachine.pirateweather.net". Great!

However, if I try to hit the time machine API with a date that's too recent, it gives me a 502 with the body

{
	"message": "Bad Gateway",
	"error": "could not JSON decode Lambda function response: statusCode validation failed"
}

Expected behavior

I would expect hitting the time machine API with a date that's too recent would return a 400 Bad Request, ideally with the latest timestamp that would be available on the time machine, but just a 400 would be fine too.

Actual behavior

Instead, I see an opaque 502 that suggests that an upstream service is returning an error.

API Endpoint

TimeMachine

Location

42.794697,-71.357407

Other details

Sample requests (requests made today, 24 December, around noon (1700UTC)

• forecast call that correctly 400s (1 Dec 23:59 ET is before "5 days ago") https://api.pirateweather.net/forecast/REDACTED/42.794697,-71.357407,2023-12-02T04:59:59Z
• time machine call that I believe should 400, but instead 502s (since 22 Dec 23:59 ET is within 5 days) https://timemachine.pirateweather.net/forecast/REDACTED/42.794697,-71.357407,2023-12-23T04:59:59Z
• time machine call that works normally, since it's far enough in the past https://timemachine.pirateweather.net/forecast/REDACTED/42.794697,-71.357407,2023-12-02T04:59:59Z

Also, would it be possible to clarify when exactly the cutoff for "last 5 days" is? Is it based on UTC midnight? Or the location midnight? Or exactly (current date - 5*24h)?

Troubleshooting steps

• I have searched this repository and Home Assistant Repository to see if the issue has already been reported.
• I have read through the API documentation before opening this issue.
• I have written an informative title.

[timeisapear]

I also see issues when trying to bridge the Forecast API and Time Machine API over the past week or so. I am not able to successfully query data from either API for a day that occurred T-4 or T-5 days ago.

[cloneofghosts]

Thank you both for the reports. I'll ping @alexander0042 to take a look at updating the error message wording and to clarify the number of past days stored. I remember seeing somewhere that stated it was five which is what was used in the docs but it looks like it only stores the past three days. I did a quick documentation change to update the number of days stored to three which matches with the current behaviour.

[alexander0042]

Hi, thanks for opening this issue and using this API- sometimes the implementation of these features changes between when I start building something to when I finish, which leads to these sports of hiccups. I started out at 5 days, but reduced it down in order to optimize the storage speeds somewhat.

Also, great "documentation" tag!

[alexander0042]

Ok, so this ended up turning into a different issue, but I've got it fixed now!

AWS removed the underlying ERA-5 dataset last week, which took down timemachine; however, since this is all open data, I was able to flip it over to use Google's dataset instead, and we're back online now!

Also updated the docs to clarify the time period when the api end point should be used instead of time machine

[juste97]

@alexander0042 I think something might not be right again regarding this.

With /52.520008,13.404954,1707951600 and api.pirateweather.net I get "Requested time too early, please use https://timemachine.pirateweather.net" and with timemachine I get "Latest date available is May 2023".

Or could it be that there really is no data in between? Data before May 2023 was correctly returned. 

[cloneofghosts]

There really is no data between May 2023 and now. AWS took down the ERA5 dataset and it was replaced with Googles dataset which only offers data until May 2023. @alexander0042 said he has met with AWS to see if they can bring back the ERA5 dataset so we'll see if they do.

[juste97]

Thanks for the fast feedback! Fingers crossed that they can bring it back 😃