abort() when trying to revalidate expired output resource #1553
Comments
|
So far no luck hitting this code path with vanilla mod_pagespeed. It looks like somehow |
|
I've tried to modify pagespeed to force return expired for every metadata cache lookup: This way both my server and This happens with both vanilla and patched pagespeed for me. I do have some modifications for PSOL and obviously there is integration step but changes do not seem to be related. I didn't touch rewriting at all, only cache/system stuff. Can you reproduce above steps to confirm the issue on your side? What happens in that test case I think is that cache extend filter requires already rewritten resource as input. It seems that
It seems that |
|
This triggers the issue with vanilla pagespeed without modification above. |
|
I should point out that CacheHtmlFlow was never part of mod_pagespeed or
ngx_pagespeed; it was in support of a feature in PageSpeed Service (
https://developers.google.com/speed/pagespeed/service/CachePartialHtml).
The test you modified is no longer part of the source trunk on github.
This might be not be related to where you decided to put the repro-test,
though.
Can you explain more about your PSOL integration? As you may be aware
there are a few entry-points in the system to support various server
architectures (threaded, async, multi-process) and network setups (proxy,
origin-modifying). I suspect it's the edge of the system that you've
diverged from the paths that have a ton of mileage.
…On Mon, Jun 5, 2017 at 8:16 AM, cellscape ***@***.***> wrote:
This triggers the issue with vanilla pagespeed without modification above.
TEST_F(CacheHtmlFlowTest, TestCacheHtmlRevalidateInputs) {
GoogleString text;
RequestHeaders request_headers;
ResponseHeaders response_headers;
FetchFromProxy("http://test.com/flush_subresources.html"
"?PageSpeedFilters=+extend_cache_css,-inline_css", true,
request_headers, &text, &response_headers, NULL, false);
SetTimeMs(MockTimer::kApr_5_2010_ms + 2 * Timer::kYearMs);
FetchFromProxy("http://test.com/flush_subresources.html"
"?PageSpeedFilters=+extend_cache_css,-inline_css", true,
request_headers, &text, &response_headers, NULL, false);
}
% ./pagespeed_automatic_test --gtest_filter=CacheHtmlFlowTest.TestCacheHtmlRevalidateInputs [src/tags/1.11.33.4]
Running main() from gtest_main.cc
Note: Google Test filter = CacheHtmlFlowTest.TestCacheHtmlRevalidateInputs
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from CacheHtmlFlowTest
[ RUN ] CacheHtmlFlowTest.TestCacheHtmlRevalidateInputs
[0605/191504:FATAL:output_resource.cc(223)] Output resources shouldn't be loaded via LoadAsync, but rather through FetchResource
./pagespeed_automatic_test() [0x11c22da]
./pagespeed_automatic_test() [0x11c36cf]
./pagespeed_automatic_test() [0x1314398]
./pagespeed_automatic_test() [0x1316dfe]
./pagespeed_automatic_test() [0x10074e7]
./pagespeed_automatic_test() [0x1005db7]
./pagespeed_automatic_test() [0x101bdb1]
./pagespeed_automatic_test() [0x136e2c4]
./pagespeed_automatic_test() [0x13bcefa]
./pagespeed_automatic_test() [0x13c31f8]
./pagespeed_automatic_test() [0x136e2c4]
./pagespeed_automatic_test() [0x13c9862]
./pagespeed_automatic_test() [0xefe108]
/lib64/libpthread.so.0(+0x7374) [0x7fb43e66d374]
/lib64/libc.so.6(clone+0x5f) [0x7fb43e3b860f]
zsh: abort (core dumped) ./pagespeed_automatic_test
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#1553 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AB2kPaZmRXuroeUHvOjpv23nFiT6EH3Sks5sA_GggaJpZM4NvmuS>
.
|
|
CacheHtmlFlow was just the first place where I observed the crash in tests with modification above. I just modified it to be able to reproduce quickly. It's in no way specific and I believe the issue is general one. Basically this test does rewrite (with image compression to cache extender filter chain), expires stored metadata for As for integration, I already described it some time ago on mailing list: https://groups.google.com/forum/#!msg/mod-pagespeed-discuss/_LJJ0UmQ8SU/sw5cvqedjmMJ. There probably were some changes but overall architecture is the same. As I said I don't think integration is to blame in that case though, we don't touch any rewriting or filters. |
|
This reproduces on master: |
|
So what's weird here is that normally you shouldn't get an OutputResource
in re-validate array, since it would have a 1 year TTL, so won't need
revalidation.
(Though I guess technically that could happen if a year passed..)
Hmm. You could probably trigger something like this via cache flushes,
though, looking at the code.
Probably OutputResource should use an ALWAYS_VALID InputInfo.Type or
something.
…On Mon, Jun 5, 2017 at 8:48 AM, cellscape ***@***.***> wrote:
CacheHtmlFlow was just the first place where I observed the crash in tests
with modification above. I just modified it to be able to reproduce
quickly. It's in no way specific and I believe the issue is general one.
Basically this test does rewrite (with image compression to cache extender
filter chain), expires stored metadata for InputInfo and then aborts when
trying to revalidate output of image compression. I might be wrong though,
don't fully understand rewriting process.
As for integration, I already described it some time ago on mailing list:
https://groups.google.com/forum/#!msg/mod-pagespeed-discuss/_LJJ0UmQ8SU/
sw5cvqedjmMJ. There probably were some changes but overall architecture
is the same. As I said I don't think integration is to blame in that case
though, we don't touch any rewriting or filters.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#1553 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADl1RAt4vCDVcRLNkSoJbZVBIyuXckRaks5sA_kEgaJpZM4NvmuS>
.
|
|
Oh, you are advancing time by 2 years.
On Mon, Jun 5, 2017 at 9:07 AM, Maksim Orlovich <morlovich@google.com>
wrote:
… So what's weird here is that normally you shouldn't get an OutputResource
in re-validate array, since it would have a 1 year TTL, so won't need
revalidation.
(Though I guess technically that could happen if a year passed..)
Hmm. You could probably trigger something like this via cache flushes,
though, looking at the code.
Probably OutputResource should use an ALWAYS_VALID InputInfo.Type or
something.
On Mon, Jun 5, 2017 at 8:48 AM, cellscape ***@***.***>
wrote:
> CacheHtmlFlow was just the first place where I observed the crash in
> tests with modification above. I just modified it to be able to reproduce
> quickly. It's in no way specific and I believe the issue is general one.
>
> Basically this test does rewrite (with image compression to cache
> extender filter chain), expires stored metadata for InputInfo and then
> aborts when trying to revalidate output of image compression. I might be
> wrong though, don't fully understand rewriting process.
>
> As for integration, I already described it some time ago on mailing list:
> https://groups.google.com/forum/#!msg/mod-pagespeed-discuss/
> _LJJ0UmQ8SU/sw5cvqedjmMJ. There probably were some changes but overall
> architecture is the same. As I said I don't think integration is to blame
> in that case though, we don't touch any rewriting or filters.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <#1553 (comment)>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ADl1RAt4vCDVcRLNkSoJbZVBIyuXckRaks5sA_kEgaJpZM4NvmuS>
> .
>
|
|
I want to note that in release mode there is no abort, and with the simple test that excites this the results seem to look good |
|
It would probably do a useless rewrite but otherwise behave, yeah.
…On Mon, Jun 5, 2017 at 9:14 AM, Otto van der Schaaf < ***@***.***> wrote:
I want to note that in release mode there is no abort, and with the simple
test that excites this the results seem to look good
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1553 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADl1RLWw8uBvp1g1LHikqM1OGEbrFhAiks5sA_85gaJpZM4NvmuS>
.
|
|
Looks like it's matter of having NDEBUG defined? This explains why nobody triggered that with apache or nginx then. |
|
We do run all of our system-tests with debug compiles so we get these extra
checks. But evidently didn't hit that case until your new unit tests.
…On Mon, Jun 5, 2017 at 9:25 AM, cellscape ***@***.***> wrote:
Looks like it's matter of having NDEBUG defined? This explains why nobody
triggered that with apache or nginx then.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1553 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AB2kPU-bydW1qY_LhS69lXCKPqQBnh95ks5sBAHOgaJpZM4NvmuS>
.
|
|
It sounds like in the short term what we could do is to remove the VLOG(DFATAL) with a warning for those running debug builds, and add a TODO for modifying @cellscape Would you like to create a PR for this? :) |
|
Is CLA still required? If yes then I pass otherwise will prepare one tomorrow. |
|
@cellscape a CLA is required.. so I'll create the PR then |
As a workaround, remove the LOG(DFATAL) and replace it with a TODO. #1553
As a workaround, remove the LOG(DFATAL) and replace it with a TODO. #1553
|
Closing, as the fix for this was merged |
I am using custom integration of PSOL and recently all instances started to crash with the same message:
Backtrace:
I've found metadata key it was using:
I've decoded InputInfo from that cache key and it was recently expired after being year in cache.
I'm using latest stable version of psol (1.11.33.4) but believe that last dev version still has the issue. It's also custom integration but I believe it doesn't matter. Well perhaps only except of long-living cache which helped to trigger the issue.
The text was updated successfully, but these errors were encountered: