outputcache max-age incorrect #330

Closed
mofee opened this Issue Oct 26, 2016 · 30 comments

Comments

Projects
None yet
@mofee

mofee commented Oct 26, 2016

a simple controller like below:
public class DetailController : Controller { [OutputCache(Duration = 300, VaryByParam = "id")] public ActionResult Index(int id) { return View(); } }

and the view
`@{
Layout = null;
}

<title>Index</title>
@DateTime.Now.ToString()
`

when i first request the url http://localhost:80/Detail/Index?id=3, the max-age in response headers is correct (max-age=300), then i refresh the page (ctrl + f5), the respones state is 304, but the max-age in response headers is incorrect (max-age=63613066544) , and this cause the url link click request read the response from the browser's disk cache ( 200 ok from disk cache).

the issue only happen in .net framework 4.6.2, not happen in 4.6.1 or early version.

@albigi

This comment has been minimized.

Show comment
Hide comment
@albigi

albigi Oct 28, 2016

Same issue occurring to a customer of mine.
I can reproduce the issue very easily by setting the Mvc OutputCache attribute in a boilerplate ASP.NET 4 app.
From some initial investigation the problem doesn't seem to be related to 4.6.2 only, at least in my case, since downgrading the framework is not helping.
Apparently, the wrong value is only applied when the System.Web.Mvc.OutputCacheAttribute overridden methods are not invoked, that is when the response is coming from the cache.
On top of that I could verify from some live debug the HttpCachePolicy is properly populated:
image
Will run some further investigation on this.
Additionally, there's also a SO thread related to this: http://stackoverflow.com/questions/40218428/iis7-5-max-age-issueasp-net-mvc-output-cache

albigi commented Oct 28, 2016

Same issue occurring to a customer of mine.
I can reproduce the issue very easily by setting the Mvc OutputCache attribute in a boilerplate ASP.NET 4 app.
From some initial investigation the problem doesn't seem to be related to 4.6.2 only, at least in my case, since downgrading the framework is not helping.
Apparently, the wrong value is only applied when the System.Web.Mvc.OutputCacheAttribute overridden methods are not invoked, that is when the response is coming from the cache.
On top of that I could verify from some live debug the HttpCachePolicy is properly populated:
image
Will run some further investigation on this.
Additionally, there's also a SO thread related to this: http://stackoverflow.com/questions/40218428/iis7-5-max-age-issueasp-net-mvc-output-cache

@albigi

This comment has been minimized.

Show comment
Hide comment
@albigi

albigi Oct 28, 2016

Where I could get so far is that apparently the issue is occurring in System.Web.HttpCachePolicy, more specifically in the method
image

In this method we do:

image

the issue comes when age is set to a negative value due to _utcTimestampRequest being null
image

I couldn't pinpoint why it comes that's null as I assumed it was taken by HttpResponse.Context.UtcTimestamp

Perhaps we are calling a Reset() too early?
It seems this happens all the time we call System.Web.UI.Page.InitOutputCache and in the response the HttpCachePolicy object "Cache" is instantiated for the first time.

albigi commented Oct 28, 2016

Where I could get so far is that apparently the issue is occurring in System.Web.HttpCachePolicy, more specifically in the method
image

In this method we do:

image

the issue comes when age is set to a negative value due to _utcTimestampRequest being null
image

I couldn't pinpoint why it comes that's null as I assumed it was taken by HttpResponse.Context.UtcTimestamp

Perhaps we are calling a Reset() too early?
It seems this happens all the time we call System.Web.UI.Page.InitOutputCache and in the response the HttpCachePolicy object "Cache" is instantiated for the first time.

@mofee

This comment has been minimized.

Show comment
Hide comment
@mofee

mofee Oct 29, 2016

in our production env, we only uninstall 4.6.2 and install 4.6.1, and the issue not occur now.
it is not effact to modiffy targetFramework in web.config if you install 4.6.2

mofee commented Oct 29, 2016

in our production env, we only uninstall 4.6.2 and install 4.6.1, and the issue not occur now.
it is not effact to modiffy targetFramework in web.config if you install 4.6.2

@DazedLead

This comment has been minimized.

Show comment
Hide comment
@DazedLead

DazedLead Nov 1, 2016

We are also getting this issue.
on Microsoft Azure Web App

DazedLead commented Nov 1, 2016

We are also getting this issue.
on Microsoft Azure Web App

@albigi

This comment has been minimized.

Show comment
Hide comment
@albigi

albigi Nov 3, 2016

The issue has been identified.

We are not setting any more the _utcTimestampRequest value in HttpCachePolicy.ResetFromHttpCachePolicySettings

In previous versions apparently we used to set it tot he actual timestamp of the inbound request.

As a result, we are calling HttpCachePolicy.GetHeaders with a _utcTimestampRequest object with a dateData value set to 0.

Hence when we do age = _utcTimestampRequest - _utcTimestampCreated; we get an absurd value such as the ones in the repro.

I cannot think about any immediate code workaround since the issue appears to happen all within System.Web

albigi commented Nov 3, 2016

The issue has been identified.

We are not setting any more the _utcTimestampRequest value in HttpCachePolicy.ResetFromHttpCachePolicySettings

In previous versions apparently we used to set it tot he actual timestamp of the inbound request.

As a result, we are calling HttpCachePolicy.GetHeaders with a _utcTimestampRequest object with a dateData value set to 0.

Hence when we do age = _utcTimestampRequest - _utcTimestampCreated; we get an absurd value such as the ones in the repro.

I cannot think about any immediate code workaround since the issue appears to happen all within System.Web

@albigi

This comment has been minimized.

Show comment
Hide comment
@albigi

albigi Nov 7, 2016

Just to provide an update on this: a fix is being tested and it is bound to be released with the 4.6.3 update or earlier.

albigi commented Nov 7, 2016

Just to provide an update on this: a fix is being tested and it is bound to be released with the 4.6.3 update or earlier.

@OnTheMike

This comment has been minimized.

Show comment
Hide comment
@OnTheMike

OnTheMike Nov 15, 2016

Is there already news on a date for the release of the fix?

Is there already news on a date for the release of the fix?

@mortb

This comment has been minimized.

Show comment
Hide comment
@mortb

mortb Nov 24, 2016

This issue is quite severe, we're about to release an application using asp.net mvc 4.6.2
We rely on cache with a time set to reduce network traffic, we'd rather not remove the OutputCache attribute as that would lead to unnecessary increases in response time / traffic.
Is there anything we can do to correct this before the fix?

mortb commented Nov 24, 2016

This issue is quite severe, we're about to release an application using asp.net mvc 4.6.2
We rely on cache with a time set to reduce network traffic, we'd rather not remove the OutputCache attribute as that would lead to unnecessary increases in response time / traffic.
Is there anything we can do to correct this before the fix?

@albigi

This comment has been minimized.

Show comment
Hide comment
@albigi

albigi Nov 24, 2016

Unfortunately, there are no updates on the official release date for the fix yet. This is still being tested and it's yet to agree on whether shipping this with 4.6.3 or earlier.

Please note that the bug is affecting the way the max-age field is populated only. This can have an impact mainly for caches or CDNs that are reading the max-age attribute in order to manage the content freshness and distribution.
The actual caching is still working as expected and the webserver will not send stale content after the refresh interval has expired.

Currently, the only recommended workaround is to downgrade to 4.6.1 by removing KB151864.

If you are unable to, there might be one hack possible which would use reflection to change the internal private field _utcTimestampRequest and populate it with a proper TimeDate value. This should happen when the EndRequest event is called. This hasn't been tested and it is NOT RECOMMENDED.

Further updates will be made available on this thread.

albigi commented Nov 24, 2016

Unfortunately, there are no updates on the official release date for the fix yet. This is still being tested and it's yet to agree on whether shipping this with 4.6.3 or earlier.

Please note that the bug is affecting the way the max-age field is populated only. This can have an impact mainly for caches or CDNs that are reading the max-age attribute in order to manage the content freshness and distribution.
The actual caching is still working as expected and the webserver will not send stale content after the refresh interval has expired.

Currently, the only recommended workaround is to downgrade to 4.6.1 by removing KB151864.

If you are unable to, there might be one hack possible which would use reflection to change the internal private field _utcTimestampRequest and populate it with a proper TimeDate value. This should happen when the EndRequest event is called. This hasn't been tested and it is NOT RECOMMENDED.

Further updates will be made available on this thread.

@kria

This comment has been minimized.

Show comment
Hide comment
@kria

kria Nov 24, 2016

Are the Azure people looped in? This is getting to be a big problem on there.

kria commented Nov 24, 2016

Are the Azure people looped in? This is getting to be a big problem on there.

@Wesley-GONG

This comment has been minimized.

Show comment
Hide comment
@Wesley-GONG

Wesley-GONG Nov 25, 2016

This bug is very very very serious, please fix it ASAP, thanks!

This bug is very very very serious, please fix it ASAP, thanks!

@mortb

This comment has been minimized.

Show comment
Hide comment
@mortb

mortb Nov 25, 2016

No, it is not only an issue for CDN:s.
We set e.g. [OutputCache(Duration=30)] on a asp.net mvc controller method that via ajax gets a list of news items in our application.
First time this resource is requested we get the 200 OK Cache-Control:maxage=30 but if we re request again the url we get a 304 NotModfied with maxage=3615676478 (roughly 110 years...) In some cases I've seen the high maxage on 200 OK responses as well.
Chrome seems to add this to the cache and subsequent calls in Chrome seems to all be resolved by the client's disk cache (you can see this in Developer tools/F12) I've waited for ten minutes and calls are still resolved by the cache (the timeout should be 30 seconds as stated above). I have not yet waited for 110 years to see if the cache is released...

Behavior between Chrome and IE really isn't the same when it comes to caching.

Now our application is for intranet usage only, so damage is limited if we release it with the cache bug, but would this be a public site that would seem superbad actually...

mortb commented Nov 25, 2016

No, it is not only an issue for CDN:s.
We set e.g. [OutputCache(Duration=30)] on a asp.net mvc controller method that via ajax gets a list of news items in our application.
First time this resource is requested we get the 200 OK Cache-Control:maxage=30 but if we re request again the url we get a 304 NotModfied with maxage=3615676478 (roughly 110 years...) In some cases I've seen the high maxage on 200 OK responses as well.
Chrome seems to add this to the cache and subsequent calls in Chrome seems to all be resolved by the client's disk cache (you can see this in Developer tools/F12) I've waited for ten minutes and calls are still resolved by the cache (the timeout should be 30 seconds as stated above). I have not yet waited for 110 years to see if the cache is released...

Behavior between Chrome and IE really isn't the same when it comes to caching.

Now our application is for intranet usage only, so damage is limited if we release it with the cache bug, but would this be a public site that would seem superbad actually...

@ttqaserver

This comment has been minimized.

Show comment
Hide comment
@ttqaserver

ttqaserver Dec 5, 2016

Can't use CDN's with such bug. How to downgrade to 4.6.1 for Azure WebApp`s?

ttqaserver commented Dec 5, 2016

Can't use CDN's with such bug. How to downgrade to 4.6.1 for Azure WebApp`s?

@Slowacki

This comment has been minimized.

Show comment
Hide comment
@Slowacki

Slowacki Jan 2, 2017

Is there any progress on this one?

Slowacki commented Jan 2, 2017

Is there any progress on this one?

@agrinei

This comment has been minimized.

Show comment
Hide comment
@agrinei

agrinei Jan 20, 2017

This is critical! A bugfix should be released even before .Net 4.6.3

agrinei commented Jan 20, 2017

This is critical! A bugfix should be released even before .Net 4.6.3

@skimonkey

This comment has been minimized.

Show comment
Hide comment
@skimonkey

skimonkey Feb 23, 2017

I am getting this exact same issue on a .Net 4.5.2 website hosted on Azure (IIS8).
Setting the outputcache on the website main page controller Index method as
[OutputCache(Duration = 86400 )]

On first request
Cache-Control: public, max-age=86400
on all subsequent requests
Cache-Control: public, max-age=63623538558

On subsequent requests
Chrome sends an If-Modified-Since header so I get a 304 and html is loaded from browser cache so less of an issue here
IE and Firefox dont send any such header so I get a 200 and full response.

Dosent look like this is a 4.6.2 only issue...

Any update on a fix..? Any advice..?
this is costing money as Azure charges on output bandwidth

skimonkey commented Feb 23, 2017

I am getting this exact same issue on a .Net 4.5.2 website hosted on Azure (IIS8).
Setting the outputcache on the website main page controller Index method as
[OutputCache(Duration = 86400 )]

On first request
Cache-Control: public, max-age=86400
on all subsequent requests
Cache-Control: public, max-age=63623538558

On subsequent requests
Chrome sends an If-Modified-Since header so I get a 304 and html is loaded from browser cache so less of an issue here
IE and Firefox dont send any such header so I get a 200 and full response.

Dosent look like this is a 4.6.2 only issue...

Any update on a fix..? Any advice..?
this is costing money as Azure charges on output bandwidth

@kria

This comment has been minimized.

Show comment
Hide comment
@kria

kria Feb 26, 2017

@skimonkey it doesn't matter what version you target, it's still going to run using the current Azure .NET Framework version (4.6.2).

kria commented Feb 26, 2017

@skimonkey it doesn't matter what version you target, it's still going to run using the current Azure .NET Framework version (4.6.2).

@richlander

This comment has been minimized.

Show comment
Hide comment
Member

richlander commented Mar 10, 2017

/cc @glennc

@OnTheMike

This comment has been minimized.

Show comment
Hide comment
@NickCraver

This comment has been minimized.

Show comment
Hide comment
@NickCraver

NickCraver Apr 7, 2017

We discovered this at Stack Overflow as the root cause to many problems. This should absolutely be a hotfix to existing .NET 4.6.2 installs. For many .NET 4.7 will not be an option for some time, and this is a critical bug. I'm honestly shocked that this went unsolved for months, we would never had expected such a known issue to stay open for this long. So many people on 4.6.2 will not ever even realize this is the root cause of some of their issues. We didn't until it was too late. The cache poisoning downstream has already caused severe harm.

As a result of this bug, we have tens of millions of downstream requests that are cached in proxies we don't control for over 2000 years. Only manual purges or volatile cache competition will purge them, which results in impossible to debug or remotely solve scenarios as our users report stale content. This is a huge, HUGE bug. It needs a fix for all ASAP. Though we're willing to deploy .NET 4.7 on a whim, very few people are in that position. They need a hotfix, ASAP.

To work around this at all we have effectively disabled all OutputCache directives by stripping cache control at our proxy layer (Fastly) and removed it from anything not behind Fastly. This costs us in bandwidth, latency, and overall slower loads from uncached local resources for users. No workaround is desirable since this bug is so deep in the framework where you cannot safely solve it through user code. This is costing us dollars and performance every minute we have the workaround active.

We discovered this at Stack Overflow as the root cause to many problems. This should absolutely be a hotfix to existing .NET 4.6.2 installs. For many .NET 4.7 will not be an option for some time, and this is a critical bug. I'm honestly shocked that this went unsolved for months, we would never had expected such a known issue to stay open for this long. So many people on 4.6.2 will not ever even realize this is the root cause of some of their issues. We didn't until it was too late. The cache poisoning downstream has already caused severe harm.

As a result of this bug, we have tens of millions of downstream requests that are cached in proxies we don't control for over 2000 years. Only manual purges or volatile cache competition will purge them, which results in impossible to debug or remotely solve scenarios as our users report stale content. This is a huge, HUGE bug. It needs a fix for all ASAP. Though we're willing to deploy .NET 4.7 on a whim, very few people are in that position. They need a hotfix, ASAP.

To work around this at all we have effectively disabled all OutputCache directives by stripping cache control at our proxy layer (Fastly) and removed it from anything not behind Fastly. This costs us in bandwidth, latency, and overall slower loads from uncached local resources for users. No workaround is desirable since this bug is so deep in the framework where you cannot safely solve it through user code. This is costing us dollars and performance every minute we have the workaround active.

@MichalGrzegorzak

This comment has been minimized.

Show comment
Hide comment
@MichalGrzegorzak

MichalGrzegorzak Apr 10, 2017

@NickCraver - I couldn't agree more with you. This is really huge issue, and it`s disappointing that MS seems to not realizing it.

I'm the (un)lucky one on Azure, anyone knows when 4.7 will be on Azure?

@NickCraver - I couldn't agree more with you. This is really huge issue, and it`s disappointing that MS seems to not realizing it.

I'm the (un)lucky one on Azure, anyone knows when 4.7 will be on Azure?

@richlander

This comment has been minimized.

Show comment
Hide comment
@richlander

richlander Apr 11, 2017

Member

Update on this issue:

Apologies that the fixes are not coming sooner. We're currently testing the 4.6.2 fix right now. We're working with @NickCraver to get some testing in his environment before going broader.

@MichalGrzegorzak Will get back to you on when either 4.7 or this fix for 4.6.2 will be in Azure.

Member

richlander commented Apr 11, 2017

Update on this issue:

Apologies that the fixes are not coming sooner. We're currently testing the 4.6.2 fix right now. We're working with @NickCraver to get some testing in his environment before going broader.

@MichalGrzegorzak Will get back to you on when either 4.7 or this fix for 4.6.2 will be in Azure.

@richlander

This comment has been minimized.

Show comment
Hide comment
@richlander

richlander Apr 12, 2017

Member

@MichalGrzegorzak can you tell which Azure service you use? Azure app services deploys .NET Framework differently than Azure VMs, for example.

Member

richlander commented Apr 12, 2017

@MichalGrzegorzak can you tell which Azure service you use? Azure app services deploys .NET Framework differently than Azure VMs, for example.

@richlander

This comment has been minimized.

Show comment
Hide comment
@richlander

richlander Apr 21, 2017

Member

Fix is coming, so closing.

Member

richlander commented Apr 21, 2017

Fix is coming, so closing.

@richlander richlander closed this Apr 21, 2017

@mortb

This comment has been minimized.

Show comment
Hide comment
@mortb

mortb Apr 27, 2017

Few things are so out of your control as a cache you cannot purge as a developer. Maybe it would be time for MS to put some effort into writing serious unit tests that detect a problem like this before anything is released?

mortb commented Apr 27, 2017

Few things are so out of your control as a cache you cannot purge as a developer. Maybe it would be time for MS to put some effort into writing serious unit tests that detect a problem like this before anything is released?

@NickCraver

This comment has been minimized.

Show comment
Hide comment
@NickCraver

NickCraver Apr 27, 2017

Since I hit this several times when verifying changes, posting here: curl -I (capital I) to view headers will not reproduce it. Because that makes a HEAD request, it's never a full/cached response to trigger the output cache. You need to make a GET request to verify it's fixed when patches land, like this:

curl -XGET -I http://myserver/test-route

Since I hit this several times when verifying changes, posting here: curl -I (capital I) to view headers will not reproduce it. Because that makes a HEAD request, it's never a full/cached response to trigger the output cache. You need to make a GET request to verify it's fixed when patches land, like this:

curl -XGET -I http://myserver/test-route
@NickCraver

This comment has been minimized.

Show comment
Hide comment
@NickCraver

NickCraver Jun 11, 2017

For others ^ we've deployed this across the Stack Overflow web tiers and haven't found any ill effects. The issue is now resolved here.

For others ^ we've deployed this across the Stack Overflow web tiers and haven't found any ill effects. The issue is now resolved here.

@MichalGrzegorzak

This comment has been minimized.

Show comment
Hide comment
@MichalGrzegorzak

MichalGrzegorzak Jun 12, 2017

@richlander We are using WebApp's. Still can`t find information when Azure will be patched. Any idea?

MichalGrzegorzak commented Jun 12, 2017

@richlander We are using WebApp's. Still can`t find information when Azure will be patched. Any idea?

@offirpeer

This comment has been minimized.

Show comment
Hide comment
@offirpeer

offirpeer Jul 19, 2017

@richlander

This release is no longer recommended / available.
For the link you provided.

@NickCraver we encountered the same problem with our site(eCommerce) and customers were asking for special offers/sales that are no longer exists because they saw them online and we must provide it to them or else they may have a reason to sue us. This bug cost us a lot of money!

offirpeer commented Jul 19, 2017

@richlander

This release is no longer recommended / available.
For the link you provided.

@NickCraver we encountered the same problem with our site(eCommerce) and customers were asking for special offers/sales that are no longer exists because they saw them online and we must provide it to them or else they may have a reason to sue us. This bug cost us a lot of money!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment