FYI - /gridpoints outage until January 1, 2024

[wsamoht]

It looks like NWS just announced an unplanned outage of the /gridpoints endpoint until January 1, 2024.

From https://www.nco.ncep.noaa.gov/status/messages/:

SENIOR DUTY METEOROLOGIST NWS ALERT ADMINISTRATIVE MESSAGE
NWS NCEP CENTRAL OPERATIONS COLLEGE PARK MD
1837Z FRI DEC 22 2023


...API FORECAST GRIDPOINT OUTAGE UNTIL 1 JAN 2024...

Due to a known software issue, the API gridpoint forecast
service will be unavailable until Monday 1 January 2024. 


Kempisty/SDM/NCO/NCEP

[treddinwater]

Seriously?
NWS Mission
Provide weather, water and climate data, forecasts, warnings, and impact-based decision support services for the protection of life and property and enhancement of the national economy.
Or not. Now I remember why I had written my apps to use the openweathermap API.

[wsamoht]

Yeah, pretty crazy. Luckily I only use the NWS API for additional details/accuracy. If NWS calls fail, my primary data source is solely used and things go on their merry way. I am sure others, though, are scrambling 😳🤯

[jah1007]

This was an especially bad week last year for the folks who were tasked with keeping the site up and functioning. Know that this was the least impactful solution available to us. Unfortunately the web service behind the scenes that serves data to gridpoints endpoint is in desperate need of replacement as evidenced by the many known issues with it. If you could email nco.ops@noaa.gov it will help with getting a permanent fix prioritized.

[dangelsaurus]

Email sent.

[lbush04]

I sent an email, too.
Everyone here knows how nice to have the National Weather Service as a free and (hopefully) reliable option for the public.

[kmpeterson]

I've also sent an email. I get the impression that either it is actually being worked this next week or that there is some lack of resources that prevents this, and it's not a happy time. Thanks for letting us know.

[paulnadler]

Is this responsible for observations not updating since the 27th? It seems like a different issue since its not based on gridpoint data.

[JohnArochaDev]

Is this why the below endpoint keeps getting an error?

https://api.weather.gov/gridpoints/LBF/19,98/forecast

[jah1007]

Yes.

[wsamoht]

Yup! 😓

[JohnArochaDev]

Thanks for the help! I've been scavenging the web fore answers haha

[nbenford]

Looks like I picked a bad week to start messing with the weather.gov API!

Is this affecting other aspects of the API?

For example, looking at the API METAR for KCNK, it lists the wind as 17018G26KT. The API wind data has no direction, speed, or gust listed.

The weather.gov website, however, does have wind speed, direction, and gusts listed in their present conditions section. Is the website itself getting data from a different source than we have access to? And is the lack of wind info a part of this outage? Trying to parse the METAR string would be an impossible headache. I'm just trying to plan how to use the API.

Thank you.

[treddinwater]

@nbenford Don't think these errors will go away when the gridpoint service comes back online Jan 1. The gridpoint service has a history of long lag times, unreliable responses, no responses, etc.. Perhaps this is why there are so many commercial services offering what is the domain of this domain. Perhaps fostering the commercial services is the real mission of weather dot gov. I too sent an email per @jah1007 -- against my better judgement.

[scadergit]

@treddinwater fwiw, that history and this issue is all related, and NCO support continues to perform CPR on the upstream service until a formal replacement can be developed. We all can't wait. We're not funded life private companies, but there has been an recent increase in resources that will help champion some long overdue issues and improvements.

[scadergit]

I would double-down on what @jah1007 noted...the entire architecture that supports dissemintation of weather data has been overhauled using modern design principles save one: the upstream solution that digests netcdfs and makes them available to the API and other services. If there's a silver lining perhaps this will get attention that it deserves to allow NCO support to get the rest they deserve.

[dwhitemv25]

Out of curiosity ... are those NetCDFs available anywhere for download?

[t1m0thy]

This was super painful for my young startup. I was quite stunned. What confuses me is why it was necessary to completely take down the end point for so long. It's pretty common practice to standup an alternative and then do a switchover when it's ready. Was this a miscommunication between parts of the agency where the team supporting the end point was also surprised and it just collapsed? I've now switched away to an alternative provider, but remain confused about what underlying problems cause something like this to happen.

[jah1007]

As noted elsewhere in this thread it was the least bad option available in keeping the rest of the service available. The backing service for the endpoint is in need of an overhaul. Emailing nco.ops@noaa.gov is the best way to get your voice heard.

[keithnicholson]

For those of you who have had options to pay services, what do you recommend?

[wsamoht]

I use Visual Crossing (moved there from DarkSky after it was shutdown by Apple). I found it to be more accurate than OpenWeather. I know you can use WeatherKit by Apple now (DarkSky's replacement) but back when I was looking (January/February of 2023), the Apple Forums were full of people having issues with reliability.

In my quest to find a replacement for DarkSky, I looked at 13 different services and Visual Crossing was the winner balancing features and affordability for a small business.

[dangelsaurus]

I haven't heard of Visual Crossing, but the licensing looks very favorable compared to WeatherKit and OpenWeather. I briefly looked at their website, but I didn't see anything mentioning how forecast areas are arranged. The "hyper local" aspect is important to my needs, how precise is Visual Crossing in that regard?

[wsamoht]

I am not sure. I do use their current weather but can't speak on hyper local accuracy as my use case is by a geographical area (typically a zip code). This might help - https://www.visualcrossing.com/resources/documentation/how-do-we-create-our-weather-forecast/ or reach out to them directly. I found their support to be better than OpenWeather too :)

[wyatt8740]

This is probably minor compared to everyone else's complaints, but this is preventing me from finishing and demoing a project I've been working on for months before my last day at this internship (tomorrow).

It's a bummer. Hope what I'm setting up now works properly when it comes back online.

[haveard]

You should always mock external APIs for this reason. Never trust an external API to be accessible and without a broken API contract. Over time you will find that this instance is not unique, and it is a bummer.

[aeharding]

Important! Only the following endpoints are considered operational. Changes to operational endpoints are subject to PNS and SCN notices. All other endpoints are subject to change without notice.

/alerts/*

https://www.weather.gov/documentation/services-web-api

All other endpoint have no uptime guarantees. I recommend using a fail-over service. That's what I do.

If you are not familiar with the SLA of external API providers you use, that's on you.

[haveard]

Still down for me. Maybe they are rolling out a release/fix.

An error occurred.
The information you are requesting is unavailable until 01/01/2024 due to a known technical issue.

[jah1007]

No release or fix. The issue is centered around having data in two different years. On 1/1 that will no longer be the case. As is noted elsewhere, please email nco.ops@noaa.gov so that we can show the impact of this outage.

[wsamoht]

I sent an email and they pointed me to this discussion 😆

[aloh86]

@jah1007 Can you explain the issue in a little more detail (just engineer's curiosity)? Is this issue having data in two different years something that happens every year around December? If not, why this particular year? Also, it sounds like no fix/release is needed and the issue will correct itself on 1/1/24?

[jah1007]

The obs issue is most certainly not related to gridpoints. The issue cropped up last year around the same time and unfortunately the backing service could not be replaced this year. Knowing what we learned from last year we took a more active approach in order to keep the other endpoints functioning.

[chaunceygardiner]

And gridpoints are back (at least for the two I tried):
https://api.weather.gov/gridpoints/MTR/92,88/forecast
https://api.weather.gov/gridpoints/OKX/34,37/forecast

[sean-user]

Yes, it's back and working without 500 errors. Good job NCO team! Happy New Year.

[ktrue]

working for me too using
https://api.weather.gov/gridpoints/MTR/94,80/forecast

Thanks, and Happy New Year!

[mrydz]

FYI - This may happen again this year, looks like an entire year wasn't enough to fix it
https://www.weather.gov/media/notification/pdf_2023_24/pns24-66_potential_api_outage.pdf

[jah1007]

The first half of the year was spent ensuring that many other vital web services would not go dark on June 30th. One of those required much more effort than anticipated.