Observations returns data out of order chronologically #474
Replies: 6 comments 2 replies
-
I've been noticing as well for the past two weeks on multiple observation stations such as: KMDW, KDPA and KLOT. It is also intermittent and returns data in the proper order sometimes. When the data is out of order, I've implemented sorting and noticed that there are gaps in the typically hourly data that I was used to seeing before this out-of-order glitch started showing up. |
Beta Was this translation helpful? Give feedback.
-
I actually have not looked closely to see if it is multiple per hour, or maybe even two identical timestamps or null data. I'm just looking at my graph output and seeing a lot of gaps that aren't normally there. I'll have to dig into the actual numbers this weekend and see exactly what the timestamps look like sorted. Here's an example output for temperature, dewpoint and "feels like". Previously these graphs would have been continuous. |
Beta Was this translation helpful? Give feedback.
-
This is super weird, because I know the code is doing a sort. (Multiple, in fact, because of some quirks in how we receive the data.) I'll have to take a look. |
Beta Was this translation helpful? Give feedback.
-
I studied a bit of data from this endpoint https://api.weather.gov/stations/KDPA/observations?start=2021-10-01T00:00:00-05:00 for a little while. And couldn't find any rhyme or reason for the unsorted data. Note that I tried with various start parameters including no start parameter and saw the same data. I was hoping to see something like there was always a jump immediately before a null value, or that it was sorted by hour then day but nothing like that stuck out at me. Here's just timestamps and temperature excerpts from the above endpoint in the order that the features array was received. Note: temperatures shown below in original units: C
Maybe someone else can notice something. I also sorted the above data by timestamp to see if the nulls also had a value at the same timestamp and it turns out this was not the case. All the null values had a unique timestamp and were not on the 52 minute mark that this station reports at. But some data on the non 52 minute timestamps was also present. Here's temperature and timestamps sorted for your perusal. I did intentionally pick some suspect data, such as the multiple temperatures exactly the same over several hours and in other cases several minutes. Note: temperatures have been converted to F in this data. The extra decimal places are an artifact of my conversion code.
I hope my research helps share a bit of time off of @StephenClouse looking into this. |
Beta Was this translation helpful? Give feedback.
-
@netbymatt during significant weather change events (and other situations) additional METARs get generated in addition to the normal hourly (or semi-hourly) ones, and those off-cycle SPECI METARS do not always contain all of the datapoints, which would likely explain the missing temps seen with the non-hourly readings. |
Beta Was this translation helpful? Give feedback.
-
I have confirmed this is a bug with the recent upgrade. I'll see if we can get a hotfix out tomorrow, maybe. |
Beta Was this translation helpful? Give feedback.
-
The calls I'm making to the api for observations from stations are returning data in unordered/incorrect timeframes.
Example my call is to the following endpoint on 2021-10-08T15:40 :
https://api.weather.gov/stations/KBTV/observations
results in data returned with timestamps from all over (see image).
Perhaps I'm missing something and this data is sorted in a method other than descending by date?
Beta Was this translation helpful? Give feedback.
All reactions