Getting outdated data on fetch(), but current data on browser

[fuzzfactory0]

The issue I have is the following: when requesting the endpoint https://api.weather.gov/gridpoints/TOP/31,80/forecast directly from chrome, everything works fine and I get the current data with valid timestamps.
However, when I hit the same URL using fetch() in JavaScript, I get outdated weather data, with an invalid last-modified header.

This is the response I get from chrome:

access-control-expose-headers: X-Correlation-Id, X-Request-Id, X-Server-Id
cache-control: public, max-age=476, s-maxage=3600
content-encoding: gzip
content-length: 1548
content-type: application/geo+json
date: Thu, 28 Oct 2021 13:20:23 GMT
expires: Thu, 28 Oct 2021 13:28:19 GMT
last-modified: Thu, 28 Oct 2021 12:10:03 GMT
server: nginx/1.16.1
strict-transport-security: max-age=31536000 ; includeSubDomains ; preload
vary: Accept-Encoding
vary: Accept,Feature-Flags,Accept-Language

And this is the response fetch('https://api.weather.gov/gridpoints/TOP/31,80/forecast') gets me:

access-control-expose-headers: X-Correlation-Id, X-Request-Id, X-Server-Id
cache-control: public, max-age=370, s-maxage=3600
content-encoding: gzip
content-length: 1445
content-type: application/geo+json
date: Thu, 28 Oct 2021 13:19:56 GMT
expires: Thu, 28 Oct 2021 13:26:06 GMT
last-modified: Tue, 02 Nov 2021 20:00:00 GMT
server: nginx/1.16.1
strict-transport-security: max-age=31536000 ; includeSubDomains ; preload
vary: Accept-Encoding
vary: Accept,Feature-Flags,Accept-Language

As you can see, the last-modified header in the fetch() response is in the future.

The server also blocks me from using if-modified-since and cache-control headers, returning a CORS error because said headers are not allowed; but their official FAQ says you should send that particular header to get new data.
But somehow chrome is able to send the header when doing the request...

Request headers from chrome:

:method: GET
:path: /gridpoints/TOP/31,80/forecast
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cache-control: max-age=0
if-modified-since: Thu, 28 Oct 2021 12:10:03 GMT

What's going on?

EDIT: Workaround (kind of)

Adding the header 'Feature-Flags' to the request seems to fix the issue, no matter the content of the header:
fetch('https://api.weather.gov/gridpoints/TOP/31,80/forecast', {headers: {'Feature-Flags': ''}});
I have no real explanation for this behavior, so the issue persists

[Pantheon1]

I am having the same problem...data fetched with my webcore piston (https GET) is outdated. Browser request yields current data.

[scadergit]

It's important to open a support ticket for operational issues, that is the only way we can be assigned to work any fix.

[Pantheon1]

I have opened a support ticket. Response...
"Your incident has been resolved and will automatically close in 10 days."
That was several days ago. I responded with more erroneous data logs. I have not heard back.

[scadergit]

@Pantheon1 what was the incident number?

[Pantheon1]

I think it was INC0308783

[lektrikpuke]

Same issue here. Response header last modified is future dated. What gives?

[wburliso]

The Feature_Flags workaround seems to work ... the first time. Then that gets cached and you have the same problem. So then if I change the Feature-Flag value each time, would that work? Hmmm ... seems to. I simply input a random number for the Feature-Flag:

pp = requests.get(url,headers={'Feature-Flags': str(random.randint(1,1000))})

[Pantheon1]

How would I add that Feature Flags to my GET request with "https://api.weather.gov/gridpoints/AFG/382,360/forecast"?

[lektrikpuke]

Depends on how you're making the request. I would suggest googling "how to send custom headers with GET request." You can refine the query with the language name (or tool) you're using.

[wburliso]

url = "https://api.weather.gov/gridpoints/AFG/382,360/forecast"
page = requests.get(url,headers={'Feature-Flags': str(random.randint(1,1000))})

[Brosilio]

not that I have any real clue what could be happening, but it seems like their cache server is caching when it shouldn't be. the feature-flags workaround seems to work for me, but without it, the API seemingly sometimes returns the correct data and sometimes returns old data, and has occasionally returned "internal server error".

basically, issue still relevant.

[dmunevar]

passing time() for feature_flags worked for me. good workaround until they fix the issue. Thx

[Pantheon1]

Absolutely agree! I just can't figure out the syntax for the flags workaround. Sent from Yahoo Mail on Android On Fri, Dec 10, 2021 at 8:21 AM, ***@***.***> wrote: not that I have any real clue what could be happening, but it seems like their cache server is caching when it shouldn't be. the feature-flags workaround seems to work for me, but without it, the API seemingly sometimes returns the correct data and sometimes returns old data, and has occasionally returned "internal server error". basically, issue still relevant. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[lektrikpuke]

I sent an e-mail to nco.ops (at noaa) per their website. I included a link to this discussion. Maybe they'll fix???

[wburliso]

I suspect web servers are out of sync.

…

On Fri, Dec 10, 2021 at 8:58 AM lektrikpuke ***@***.***> wrote: I sent an e-mail to nco.ops (at noaa) per their website. I included a link to this discussion. Maybe they'll fix??? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#492 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH6QREF6R5UGKG5KB2TSECTUQIWVBANCNFSM5G44ZXVQ> .

[nkocodes]

Been having this problem: Did the feature flag thing to no success:

import requests
 
update = {'Feature-Flags': ''}   
response = requests.get ("https://api.weather.gov/points/45.5117,-122.6756", headers = update)
data = response.json()
forecast = data["properties"]["forecast"]
response = requests.get(forecast)
data = response.json()
time = data["properties"]["periods"][1]["startTime"][:10]
time_cut = time[5:]
time_final_cut = time_cut[5:]
print(time_cut)

[wburliso]

The example I provided a week or so ago uses a value for feature flags. A value that changes with each call. This causes the header to be different each time. You will get a fresh page from the server. This bandaid is an alternative to the good old tactic of using random stuff appended to the URL. In this case the URL's are strict so you cannot append anything to the url. Hope that helps.

…

On Sat, Dec 11, 2021, 2:54 PM gunraidan ***@***.***> wrote: Been having this problem: Did the feature flag thing to no success: `#Scratch Pad import requests response = requests.get ("https://api.weather.gov/points/45.5117,-122.6756", headers = {'Feature-Flags': ''}); data = response.json() forecast = data["properties"]["forecast"] response = requests.get(forecast) data = response.json() time = data["properties"]["periods"][1]["startTime"][:10] time_cut = time[5:] time_final_cut = time_cut[5:] print(time_cut)` — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#492 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH6QREA5EZZS7SRKCWOH5VDUQPJB5ANCNFSM5G44ZXVQ> .

[nkocodes]

Did this and nothing changed:

`import requests
import random

update = {'Feature-Flags': str(random.randint(1,1000))}
response = requests.get ("https://api.weather.gov/points/45.5117,-122.6756", headers = update)
data = response.json()
forecast = data["properties"]["forecast"]
response = requests.get(forecast)
data = response.json()
time = data["properties"]["periods"][1]["startTime"][:10]
time_cut = time[5:]
time_final_cut = time_cut[5:]
print(time_cut)`

[chaunceygardiner]

Passing current time as the value for the Features-Flag header is currently working for me. Thank you for that.

[wburliso]

Looking at the value for Feature flags in your code, it shows an empty string. The value should contain characters. And the characters should change each time. That is why I used a random number. I could very well be missing something in your question. Hope that helps.

…

On Sat, Dec 11, 2021, 7:28 PM gunraidan ***@***.***> wrote: Isn't that what my code shows? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#492 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH6QREAZ2IINNZCSHA2ZQNDUQQJFRANCNFSM5G44ZXVQ> .

[nkocodes]

I have this and it still doesn't change:

update = {'Feature-Flags': str(random.randint(1,1000))}
response = requests.get ("https://api.weather.gov/points/45.5117,-122.6756", headers = update)
data = response.json()
forecast = data["properties"]["forecast"]
response = requests.get(forecast)
data = response.json()
time = data["properties"]["periods"][1]["startTime"][:10]
time_cut = time[5:]
time_final_cut = time_cut[5:]
print(time_cut)`

[nkocodes]

Apparently when you change the coordinates to literally any other town it works fine. It's either the exact town messing up on the api's end or something with the cache.

[Pantheon1]

I have been using whole numbers the entire time. In fact, I have been doing two requests one minute apart. I make a request with coordinates in Alaska (that will obviously be incorrect for me in Tennessee). Then a request for my area. I did this thinking that the data I was getting was being cached and was resulting in incorrect reports. Nope. Spend some time with Jesus every day! Todd Mumpower On Saturday, December 11, 2021, 11:40:02 PM EST, gunraidan ***@***.***> wrote: Possible solution: When I round up the coordinates in the link in URL by replacing this: response = requests.get ("https://api.weather.gov/points/45.5117,-122.6756") with this: response = requests.get ("https://api.weather.gov/points/46,-123") It works fine. Really as long as either the longitude or latitude is a whole number. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[wburliso]

Give that new URL a while and you will eventually start getting old cached copies again. The reason a browser never has a problem is that it sends ever changing information in the request header. Therefore each request is different, fetching a fresh copy every time. I think that is about all I can think of. Hope that helped. THE HORRIBLE THING ABOUT THIS is that we are adding little by little strain on the web servers. It invalidates the reason for having a server farm. This is nuts!!!!!!!!!!!!!!!!!!!!!!!!!!!

…

On Sun, Dec 12, 2021 at 1:52 AM Pantheon ***@***.***> wrote: I have been using whole numbers the entire time. In fact, I have been doing two requests one minute apart. I make a request with coordinates in Alaska (that will obviously be incorrect for me in Tennessee). Then a request for my area. I did this thinking that the data I was getting was being cached and was resulting in incorrect reports. Nope. Spend some time with Jesus every day! Todd Mumpower On Saturday, December 11, 2021, 11:40:02 PM EST, gunraidan ***@***.***> wrote: Possible solution: When I round up the coordinates in the link in URL by replacing this: response = requests.get ("https://api.weather.gov/points/45.5117,-122.6756 ") with this: response = requests.get ("https://api.weather.gov/points/46,-123") It works fine. Really as long as either the longitude or latitude is a whole number. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#492 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH6QREE27B6LHINWRJVUJMDUQRWHTANCNFSM5G44ZXVQ> .

[Pantheon1]

I am getting invalid data in webcore often....usually several days behind. Spend some time with Jesus every day! Todd Mumpower On Monday, December 13, 2021, 9:30:18 AM EST, wburliso ***@***.***> wrote: Give that new URL a while and you will eventually start getting old cached copies again. The reason a browser never has a problem is that it sends ever changing information in the request header. Therefore each request is different, fetching a fresh copy every time. I think that is about all I can think of. Hope that helped. THE HORRIBLE THING ABOUT THIS is that we are adding little by little strain on the web servers. It invalidates the reason for having a server farm. This is nuts!!!!!!!!!!!!!!!!!!!!!!!!!!!

On Sun, Dec 12, 2021 at 1:52 AM Pantheon ***@***.***> wrote: I have been using whole numbers the entire time. In fact, I have been doing two requests one minute apart. I make a request with coordinates in Alaska (that will obviously be incorrect for me in Tennessee). Then a request for my area. I did this thinking that the data I was getting was being cached and was resulting in incorrect reports. Nope. Spend some time with Jesus every day! Todd Mumpower On Saturday, December 11, 2021, 11:40:02 PM EST, gunraidan ***@***.***> wrote: Possible solution: When I round up the coordinates in the link in URL by replacing this: response = requests.get ("https://api.weather.gov/points/45.5117,-122.6756 ") with this: response = requests.get ("https://api.weather.gov/points/46,-123") It works fine. Really as long as either the longitude or latitude is a whole number. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#492 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH6QREE27B6LHINWRJVUJMDUQRWHTANCNFSM5G44ZXVQ> .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[Pantheon1]

This is from another expert webcore programmer who did an evaluation of this API....I hate to say… but I suspect that even if we can force “fresh” data, the current API layout might frustrate more than illuminate. For example, the numbers for each dataPoint do NOT line up. Currently, here is what I see: 112 slots in temperature (hourly for approx 4.6 days) 118 slots in dewpoint (4.9 days) ..8 slots in maxTemperature (approx a week) ..9 slots in minTemperature (approx a week) .81 slots in relativeHumidity ..1 slots in heatIndex 133 slots in windChill .34 slots in skyCover (every 3+ hours?) .24 slots in windDirection (every 3 hours?) Wow. This is a coding nightmare. You would have to pull (and convert) the validTime for every single data point needed… IE: No more simple coding like: IF temp[1] is greater than temp[2] (because 1 or 2 could mean anything) Sent from Yahoo Mail on Android On Fri, Dec 10, 2021 at 8:21 AM, ***@***.***> wrote: not that I have any real clue what could be happening, but it seems like their cache server is caching when it shouldn't be. the feature-flags workaround seems to work for me, but without it, the API seemingly sometimes returns the correct data and sometimes returns old data, and has occasionally returned "internal server error". basically, issue still relevant. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[efishinsea]

I posted this on another thread for this problem, but removing the headers in the curl request seems to give the appropriate data for the request. If I include the header array ("Accept: application/json",) it fails most times.

Maybe one of their servers is having issues or out of sync with another? It is baffling.

[nkocodes]

I'm confused. Could you explain in more detail where
("Accept: application/json",)

was?

[efishinsea]

in the PHP code making the call

$headers = array( "Accept: application/json", "Cache-Control: max-age=360", );

[nkocodes]

So I'd put it like this:

        contact = {'User-Agent': '("Accept: application/json", "Cache-Control: max-age=360",)'}
        response = requests.get (f'https://api.weather.gov/points/{self.geolocation}', headers = contact)
        self.data = response.json()
        forecast = self.data["properties"][specify]
        response = requests.get(forecast)
        self.data = response.json()```

(Using Python btw)

[JJKraw]

Similar to Ken, getting some stale data.

https://api.weather.gov/gridpoints/BOX/64,79/forecast/hourly

Fetch from my 1&1 Ionos service returns this. If I use my browser (via RCN), I get fresh data.

HTTP/2 200 
server: nginx/1.20.1
content-type: application/geo+json
last-modified: Thu, 29 Dec 2022 22:53:01 GMT
access-control-allow-origin: *
access-control-expose-headers: X-Correlation-Id, X-Request-Id, X-Server-Id
x-request-id: 1eb2660b-2f38-435e-851e-7d67e8693d64
x-correlation-id: 24d5f357
x-server-id: vm-bldr-nids-apiapp7.ncep.noaa.gov
x-akamai-request-id: 82575c50.1b534fa
cache-control: public, max-age=3223, s-maxage=3600
expires: Tue, 08 Feb 2022 02:59:25 GMT
date: Tue, 08 Feb 2022 02:05:42 GMT
x-cache: TCP_REFRESH_MISS from a104-70-164-93.deploy.akamaitechnologies.com (AkamaiGHost/10.7.1-38761473) (S)
x-edge-request-id: 1b534fa
vary: Accept,Feature-Flags,Accept-Language
strict-transport-security: max-age=31536000 ; includeSubDomains ; preload

[JJKraw]

Getting old data for both forecast and hourly (BOX, 64,79).

<!-- point forecast more than 18hrs old (age h:m:s is 981:28:06) updated:'2021-12-29T22:53:01+00:00'-->
<!-- headers from the gridpoint request
HTTP/2 200 
server: nginx/1.20.1
content-type: application/geo+json
last-modified: Thu, 29 Dec 2022 22:53:01 GMT
access-control-allow-origin: *
access-control-expose-headers: X-Correlation-Id, X-Request-Id, X-Server-Id
x-request-id: bfdf8d06-1441-4bf2-aa48-b378e74ecec0
x-correlation-id: 24d5ef3c
x-server-id: vm-bldr-nids-apiapp7.ncep.noaa.gov
cache-control: public, max-age=1436, s-maxage=3600
expires: Tue, 08 Feb 2022 20:45:03 GMT
date: Tue, 08 Feb 2022 20:21:07 GMT
content-length: 11858
x-cache: TCP_MEM_HIT from a104-70-164-135.deploy.akamaitechnologies.com (AkamaiGHost/10.7.1-38761473) (-)
x-edge-request-id: 34f4b7df
vary: Accept,Feature-Flags,Accept-Language
strict-transport-security: max-age=31536000 ; includeSubDomains ; preload
x-akamai-request-id: 34f4b7df
 -->

<!-- hourly data more than 18hrs old (age h:m:s is 981:28:06) updated:'2021-12-29T22:53:01+00:00' -->
<!-- headers from the hourly request
HTTP/2 200 
server: nginx/1.20.1
content-type: application/geo+json
last-modified: Thu, 29 Dec 2022 22:53:01 GMT
access-control-allow-origin: *
access-control-expose-headers: X-Correlation-Id, X-Request-Id, X-Server-Id
x-request-id: 1eb2660b-2f38-435e-851e-7d67e8693d64
x-correlation-id: 24d5f357
x-server-id: vm-bldr-nids-apiapp7.ncep.noaa.gov
cache-control: public, max-age=3237, s-maxage=3600
expires: Tue, 08 Feb 2022 21:15:04 GMT
date: Tue, 08 Feb 2022 20:21:07 GMT
x-cache: TCP_MEM_HIT from a104-70-164-135.deploy.akamaitechnologies.com (AkamaiGHost/10.7.1-38761473) (-)
x-edge-request-id: 34f4b8dc
vary: Accept,Feature-Flags,Accept-Language
strict-transport-security: max-age=31536000 ; includeSubDomains ; preload
x-akamai-request-id: 34f4b8dc
 -->

[chaunceygardiner]

Now that @jah1007 has alerted us to the expiration date…
expires: Tue, 08 Feb 2022 20:45:03 GMT
expires: Tue, 08 Feb 2022 21:15:04 GMT

It looks like you’ll see current 12-hour forecasts in about 16 minutes and hourly in 46 minutes.

[JJKraw]

Seeing fresh data now.

I'm on the NCO OPS email thread for this so I assume I should use that if I see further issues?

[chaunceygardiner]

It’s really looking like we’re in the final throes of the old forecasts issue. Thank you, @jah1007, and everyone who worked on this.

[jah1007]

@JJKraw We have one remaining step to complete on internal support of the backing application but look for an email asking if the issue has been resolved. @chaunceygardiner This one was a tough one but hopefully we can limp along until we get an update from the development team.

[ktrue]

I'm still seeing Stale also:

<!-- curl fetching 'https://api.weather.gov/gridpoints/MTR/94,80/forecast' -->
<!-- HTTP stats:  RC=200 dest=184.87.164.14 port=443 (from sce=74.208.252.86)
      Times: dns=0.061 conn=0.072 pxfer=0.217 get=0.015 total=0.231 secs -->
<!-- loaded point-forecast https://api.weather.gov/gridpoints/MTR/94,80/forecast - 11623 bytes -->
<!-- wrote cache file ./cache/forecast-CAZ513-0-geojson.txt -->
<!-- point forecast more than 18hrs old (age h:m:s is 119:54:05) updated:'2022-02-03T20:45:00+00:00' .. use Zone forecast instead -->
<!-- headers from the gridpoint request
HTTP/1.1 200 OK
Server: nginx/1.20.1
Content-Type: application/geo+json
Last-Modified: Wed, 09 Feb 2022 06:00:00 GMT
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: X-Correlation-Id, X-Request-Id, X-Server-Id
X-Request-ID: 8d7ba7ef-ffe9-4992-a1e0-595b0e07d37c
X-Correlation-ID: b3f667fa
X-Server-ID: vm-lnx-nids-apiapp7.ncep.noaa.gov
Cache-Control: public, max-age=3101, s-maxage=3600
Expires: Tue, 08 Feb 2022 21:30:46 GMT
Date: Tue, 08 Feb 2022 20:39:05 GMT
Content-Length: 10803
X-Cache: TCP_MEM_HIT from a104-70-164-87.deploy.akamaitechnologies.com (AkamaiGHost/10.7.1-38761473) (-)
Connection: keep-alive
X-Edge-Request-ID: 31f49079
Vary: Accept,Feature-Flags,Accept-Language
Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload
X-Akamai-Request-ID: 31f49079
 -->
<!-- logged 'Stale' to './cache/advforecast2-log-2022-02-08.txt' -->

[JJKraw]

Latest status via email from NCO Ops:

NCO-OBT is aware that cached data calls are still hanging around in api.weather.gov/gridpoints on the external web.
-- We are able to see the same issues that customers are seeing through our internal testing and are continuing to troubleshoot the issue.
--- Additional updates will come forward as soon as more is determined during Dev system testing / simulations.

[jah1007]

This is accurate. We have an understanding of what is occurring and are working to resolve it.

[dmunevar]

I tried the same request from a few different servers around the country and as when the server ID matches (I.e. X-Server-ID: vm-lnx-nids-apiapp7.ncep.noaa.gov), I get the same response. But a server that returns an outdated response for one api call, may return good data for a different api call. Seems like a CDN is caching data for too long. If it helps, here is the response header for a call I made today but returned data from 12/31/2021. The vm-lnx-nids-apiapp5.ncep.noaa.gov server returned the correct information.

HTTP/1.1 200 OK
Server: nginx/1.20.1
Content-Type: application/geo+json
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: X-Correlation-Id, X-Request-Id, X-Server-Id
X-Request-ID: 1e405ff0-0fff-4a8d-b25b-543f9b3ffa26
X-Correlation-ID: bbd8729
X-Server-ID: vm-lnx-nids-apiapp7.ncep.noaa.gov
Cache-Control: public, max-age=3600, s-maxage=3600
Expires: Wed, 09 Feb 2022 18:20:45 GMT
Date: Wed, 09 Feb 2022 17:20:45 GMT
Connection: keep-alive
X-Edge-Request-ID: 1624f8d9
Vary: Accept,Feature-Flags,Accept-Language
Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload

[Brosilio]

I'm not sure if this is related, but I pretty frequently get 500 responses from the API (seemingly randomly). the weather data is also usually inconsistent between calls when this happens, though I haven't seen data as old as others in the thread are saying.

[JJKraw]

Prompting me to ask: should we document the 500s (I see fairly frequently) as a separate problem, or is it possibly related?

[jah1007]

@dmunevar The fact that you are getting old data but not the last-modified header is about what I would expect there to be sometimes. The purpose of the change is to make it hurt less when it happens. What we would see is that old data would be served with a Last-Modified time in the future which caused it to be cached for long periods of time. Removing the header and upping the cache time to 1 hr was a trade off of more reliability while we keep the origin protected from an unknown amount of added calls. Could you provide the URL you were seeing old data for?

@Brosilio @JJKraw the 500s are yet another symptom of garbage data being returned from the underlying service. While it is a known issue it does require the underlying service be re-written which is coming. I unfortunately don't have any further tricks, band-aids, or bubble gum available to prevent that from happening.

[nkocodes]

Okay, update. The data is recent for some zipcodes but not others.

Glad to know you are on the case, but is there anything I can do to help?

[chaunceygardiner]

The workaround to provide a random value for Feature-Flags (suggested in this thread towards the end of last year) has been working flawlessly for me for a day now.
Edited to correctly show back ticks in second command.

The following is consistently returning a 2021-12-27 forecast:
curl --include https://api.weather.gov/gridpoints/MTR/91,87/forecast/hourly

And with the Feature-Flags header using date as the value, consistently returning the current forecast:
curl --header "Feature-Flags:`date`" --include https://api.weather.gov/gridpoints/MTR/91,87/forecast/hourly

[chaunceygardiner]

Post above edited to correctly show the backticks in the second curl command.

[nkocodes]

Could you provide the list of URLs that you are using, both good and bad?

Sorry for responding so late.

It still isn't working sometimes.

Here is the url list of outdated info:

https://api.weather.gov/gridpoints/PQR/108,102/forecast
https://api.weather.gov/gridpoints/PQR/108,102/forecast/hourly

https://api.weather.gov/gridpoints/PQR/108,102/forecast
https://api.weather.gov/gridpoints/PQR/108,102/forecast/hourly

https://api.weather.gov/gridpoints/MKX/81,42/forecast
https://api.weather.gov/gridpoints/MKX/81,42/forecast/hourly

https://api.weather.gov/gridpoints/LOX/148,47/forecast
https://api.weather.gov/gridpoints/LOX/148,47/forecast/hourly

Honestly everything I entered is working wrong. It's never been this bad. :(

[jah1007]

@gunraidan Please try again.

[nkocodes]

@gunraidan Please try again.

These don't work:

https://api.weather.gov/gridpoints/OKX/32,36/forecast
https://api.weather.gov/gridpoints/OKX/32,36/forecast/hourly

https://api.weather.gov/gridpoints/PQR/107,99/forecast
https://api.weather.gov/gridpoints/PQR/107,99/forecast/hourly

[JJKraw]

Update I just received:

At 1355z, current status for the issue of old/stale data during API call, is:

• NWS/NCO/OBT - has identified what appears to be a common Akamai edge server still serving a very old cache despite our efforts to clear it.

--- We have opened a support ticket with Akamai for assistance in clearing the stubborn cache. AKAM-CASE #F-CS-5740418.

Status Update pending outcome of vendor assistance.

[wburliso]

Please forgive my questions that may sound critical but that I don't mean to sound personal. I hope people will chime in to let me know if I am off base or not: Can we help effectively escalate an issue? It seems this issue has been going on for some time. Is there a priority level on this public service? I think it could be helpful to know what we should expect. Is there a particular link to more comprehensive documentation? I wonder if a forum might be a better way for us to communicate as an alternative to email lists. Is there a way to request enhancements like this? Please let me know if I am in the right place to ask these questions. Thank you!

…

On Fri, Feb 11, 2022, 6:13 AM JJKraw ***@***.***> wrote: Update I just received: At 1355z, current status for the issue of old/stale data during API call, is: - NWS/NCO/OBT - has identified what appears to be a common Akamai edge server still serving a very old cache despite our efforts to clear it. --- We have opened a support ticket with Akamai for assistance in clearing the stubborn cache. AKAM-CASE #F-CS-5740418. Status Update pending outcome of vendor assistance. — Reply to this email directly, view it on GitHub <#492 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH6QREAI45JCUYZG3HQYOHDU2UKSHANCNFSM5G44ZXVQ> . You are receiving this because you commented.Message ID: ***@***.***>

[jah1007]

More thorough documentation of the API can be found here. This issue has been assigned to a release but that release continues to be pushed back for other higher priority tasks assigned to the development team. We have identified what we believe to be the last change we need to make to mask the issue until the backing service can be fully upgraded. Our understanding of the problem differed slightly from what is occurring. We are seeking approval for the change to be made next week.

[mbof]

Thanks

A couple of documentation suggestions

• This issue should be listed in the Updates tab's list of known issues: the gridpoints forecast served to some clients can be weeks or (months?) stale, persistently across fetches.
• Up to you whether to document the workaround of sending random garbage in Feature-Flags header — maybe there's a better workaround? The fact that browsers are unaffected and command line clients on the same machine are affected suggests that maybe there's another way to work around the issue (browsers get fresh data but I assume not by sending bogus Feature-Flags headers)
• On the main documentation tab, the header for feature flags is documented as Feature-Flag, but it's apparently Feature-Flags.

[jah1007]

We have just pushed what should be the final change required for this. Please let us know.

[mbof]

Seems to work for me now!

$ curl -s https://api.weather.gov/gridpoints/LOX/148,44/forecast | grep generatedAt
        "generatedAt": "2022-02-25T20:22:54+00:00",

[sweinreich]

I'm still seeing stale data. This is from 2/26 10:23am EST and the forecast is from Dec 31 2021:
"{'Server': 'nginx/1.20.1', 'Content-Type': 'application/geo+json', 'Last-Modified': 'Sat, 31 Dec 2022 11:47:31 GMT', 'Access-Control-Allow-Origin': '*', 'Access-Control-Expose-Headers': 'X-Correlation-Id, X-Request-Id, X-Server-Id', 'X-Request-ID': '0c0e5908-9444-425c-a219-b1c9e268e766', 'X-Correlation-ID': 'ab4b9c90', 'X-Server-ID': 'vm-bldr-nids-apiapp12.ncep.noaa.gov', 'Content-Encoding': 'gzip', 'Content-Length': '1706', 'X-Akamai-Request-ID': '51e36d3b.32e29673.139cff0b.16095209', 'Cache-Control': 'public, max-age=1721, s-maxage=3600', 'Expires': 'Sat, 26 Feb 2022 15:52:30 GMT', 'Date': 'Sat, 26 Feb 2022 15:23:49 GMT', 'X-Cache': 'TCP_REFRESH_MISS from a23-53-34-21.deploy.akamaitechnologies.com (AkamaiGHost/10.7.2-39280538) (S)', 'Connection': 'close', 'Vary': 'Accept-Encoding, Accept,Feature-Flags,Accept-Language', 'X-Edge-Request-ID': '16095209', 'Strict-Transport-Security': 'max-age=31536000 ; includeSubDomains ; preload'}"

Here's another from 6:39pm EST yesterday, where I got a stale forecast from Dec 24 2021. I see that this is from a different x-server-id than the example I showed above.
"{'Server': 'nginx/1.20.1', 'Content-Type': 'application/geo+json', 'Last-Modified': 'Sat, 24 Dec 2022 08:27:31 GMT', 'Access-Control-Allow-Origin': '*', 'Access-Control-Expose-Headers': 'X-Correlation-Id, X-Request-Id, X-Server-Id', 'X-Request-ID': '438bb9f4-7fed-4e53-b6e8-2321e01b5fef', 'X-Correlation-ID': '44272bce', 'X-Server-ID': 'vm-bldr-nids-apiapp11.ncep.noaa.gov', 'Content-Encoding': 'gzip', 'Content-Length': '1717', 'X-Akamai-Request-ID': '12b344f2.19b7d8e2.58ce6eee', 'Cache-Control': 'public, max-age=1872, s-maxage=3600', 'Expires': 'Fri, 25 Feb 2022 23:40:19 GMT', 'Date': 'Fri, 25 Feb 2022 23:09:07 GMT', 'X-Cache': 'TCP_REFRESH_MISS from a104-88-207-213.deploy.akamaitechnologies.com (AkamaiGHost/10.7.2-39280538) (S)', 'Connection': 'close', 'Vary': 'Accept-Encoding, Accept,Feature-Flags,Accept-Language', 'X-Edge-Request-ID': '58ce6eee', 'Strict-Transport-Security': 'max-age=31536000 ; includeSubDomains ; preload'}"

[jah1007]

@sweinreich If you are still seeing the issue please provide the URLs.

[Sudo-Rob]

Here's one to check out: https://api.weather.gov/gridpoints/LWX/100,87/forecast/hourly

[sweinreich]

@jah1007
Here's one from about 30 minutes ago:
https://api.weather.gov/gridpoints/BOX/52,72/forecast
{'Server': 'nginx/1.20.1', 'Content-Type': 'application/geo+json', 'Last-Modified': 'Sat, 31 Dec 2022 11:47:31 GMT', 'Access-Control-Allow-Origin': '*', 'Access-Control-Expose-Headers': 'X-Correlation-Id, X-Request-Id, X-Server-Id', 'X-Request-ID': '0c0e5908-9444-425c-a219-b1c9e268e766', 'X-Correlation-ID': 'ab4b9c90', 'X-Server-ID': 'vm-bldr-nids-apiapp12.ncep.noaa.gov', 'Content-Encoding': 'gzip', 'Content-Length': '1706', 'Cache-Control': 'public, max-age=1734, s-maxage=3600', 'Expires': 'Wed, 02 Mar 2022 17:37:50 GMT', 'Date': 'Wed, 02 Mar 2022 17:08:56 GMT', 'X-Cache': 'TCP_MEM_HIT from a23-34-83-84.deploy.akamaitechnologies.com (AkamaiGHost/10.7.2-39280538) (-)', 'Connection': 'close', 'Vary': 'Accept-Encoding, Accept,Feature-Flags,Accept-Language', 'X-Edge-Request-ID': 'e13ba66', 'Strict-Transport-Security': 'max-age=31536000 ; includeSubDomains ; preload', 'X-Akamai-Request-ID': 'e13ba66'}

[mtl010957]

This query is still giving old results as of right now (15:25 EST):
curl -i -s -H "Accept: application/geo+json" https://api.weather.gov/gridpoints/DTX/51,48/forecast
The bad responses all seem to be coming from one virtual server:
x-server-id: vm-bldr-nids-apiapp12.ncep.noaa.gov

[jah1007]

Try again please.

[Pantheon1]

I have been getting the right day for a week or so. But getting lots of 500's (about every other). Spend some time with Jesus every day! Todd Mumpower On Friday, February 25, 2022, 3:28:08 PM EST, mbof ***@***.***> wrote: Seems to work for me now! $ curl -s https://api.weather.gov/gridpoints/LOX/148,44/forecast | grep generatedAt "generatedAt": "2022-02-25T20:22:54+00:00", — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: ***@***.***>

[jah1007]

Cannot do much about the 500 errors, unfortunately.

[mtl010957]

The query I am using works correctly now, it's hitting a different virtual server now (x-server-id: vm-bldr-nids-apiapp8.ncep.noaa.gov).

[brunerd]

Looking good here! Getting up-to-date forecast data. No Feature-Flag workaround needed! 🤞

% curl -I https://api.weather.gov/gridpoints/LOT/69,72/forecast                         
HTTP/2 200 
server: nginx/1.20.1
content-type: application/geo+json
access-control-allow-origin: *
access-control-expose-headers: X-Correlation-Id, X-Request-Id, X-Server-Id
x-request-id: f2899352-c567-40f0-a455-c5dbc234476d
x-correlation-id: 1d29f75c
x-server-id: vm-bldr-nids-apiapp9.ncep.noaa.gov
cache-control: public, max-age=3536, s-maxage=3600
expires: Sat, 26 Feb 2022 20:51:02 GMT
date: Sat, 26 Feb 2022 19:52:06 GMT
x-edge-request-id: 8d1c6e1
vary: Accept,Feature-Flags,Accept-Language
strict-transport-security: max-age=31536000 ; includeSubDomains ; preload

% curl -I -H "Feature-Flags: $(date)" https://api.weather.gov/gridpoints/LOT/69,72/forecast
HTTP/2 200 
server: nginx/1.20.1
content-type: application/geo+json
access-control-allow-origin: *
access-control-expose-headers: X-Correlation-Id, X-Request-Id, X-Server-Id
x-request-id: 077259d4-9eac-4ae9-978a-0f3c3e2d7bf3
x-correlation-id: 7c2aafc
x-server-id: vm-bldr-nids-apiapp2.ncep.noaa.gov
cache-control: public, max-age=3527, s-maxage=3600
expires: Sat, 26 Feb 2022 20:50:56 GMT
date: Sat, 26 Feb 2022 19:52:09 GMT
x-edge-request-id: 4904674a
vary: Accept,Feature-Flags,Accept-Language
strict-transport-security: max-age=31536000 ; includeSubDomains ; preload

[chaunceygardiner]

Looking good here! Getting up-to-date forecast data. No Feature-Flag workaround needed!

Don’t be hasty. I had 18 hours of good forecasts before getting December forecasts again.

[Pantheon1]

It appears that I am getting good data. No Feature_Flag workaround for me either. But I am a little confused by the nomenclature. I live in the Eastern time zone and I get the following: So is my "Today" startTime adjusted for my Eastern time zone, ie -05:00?Is the detailedForecast for an actual time of 6 am (for me) as indicated by the "startTime" above?Or do I need to make the -05:00 adjustment myself? Spend some time with Jesus every day! Todd Mumpower Spend some time with Jesus every day! Todd Mumpower On Saturday, February 26, 2022, 2:57:26 PM EST, John Kline ***@***.***> wrote: Looking good here! Getting up-to-date forecast data. No Feature-Flag workaround needed! Don’t be hasty. I had 18 hours of good forecasts before getting December forecasts again. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: ***@***.***>

[JJKraw]

All of the timestamps and any textual times in detailedForecast or other fields have been adjusted to local time. No need to convert anything unless you wanted to present it in the context of another time zone. Not sure why you'd do that though.

[jah1007]

One thing to try if you are still getting responses with Last-Modified headers is to flush the DNS on the client side.