New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CHECK failure on DNS timeout #625

Closed
melong007 opened this Issue Feb 24, 2014 · 11 comments

Comments

Projects
None yet
3 participants
@melong007
Copy link

melong007 commented Feb 24, 2014

the bt info are as follow:
#0 base::debug::BreakDebugger () at third_party/chromium/src/base/debug/debugger_posix.cc:243
#1 0x00000000004b6370 in (anonymous namespace)::LogMessageHandler (severity=4,

file=0xa422b8 "/data/cache9/pagespeed/ngx_pagespeed-master/src/ngx_fetch.cc", line=208, message_start=38, 
str=...) at /data/cache9/pagespeed/ngx_pagespeed-master/src/log_message_handler.cc:87

#2 0x00000000004d83d5 in logging::LogMessage::~LogMessage (this=0x7ffff2f41a50, __in_chrg=)

at third_party/chromium/src/base/logging.cc:574

#3 0x00000000004b7b8f in net_instaweb::NgxFetch::CallbackDone (this=0x2b1590005f00, success=false)

at /data/cache9/pagespeed/ngx_pagespeed-master/src/ngx_fetch.cc:208

#4 0x00000000004b7fc7 in net_instaweb::NgxFetch::NgxFetchResolveDone (resolver_ctx=0x2b159006b0f0)

at /data/cache9/pagespeed/ngx_pagespeed-master/src/ngx_fetch.cc:289

#5 0x000000000042e182 in ngx_resolver_timeout_handler (ev=0x2b159004ef70) at src/core/ngx_resolver.c:2043
#6 0x0000000000434007 in ngx_event_expire_timers () at src/event/ngx_event_timer.c:149
#7 0x0000000000431df3 in ngx_process_events_and_timers (cycle=0xd1dc9b0) at src/event/ngx_event.c:265
#8 0x000000000043f0a1 in ngx_worker_process_cycle (cycle=0xd1dc9b0, data=0x6)

at src/os/unix/ngx_process_cycle.c:807

#9 0x000000000043ba13 in ngx_spawn_process (cycle=0xd1dc9b0, proc=0x43eef1 <ngx_worker_process_cycle>, data=0x6,

name=0xa354d3 "worker process", respawn=6) at src/os/unix/ngx_process.c:198

#10 0x000000000043eafb in ngx_reap_children (cycle=0xd1dc9b0) at src/os/unix/ngx_process_cycle.c:619
#11 0x000000000043d7af in ngx_master_process_cycle (cycle=0xd1dc9b0) at src/os/unix/ngx_process_cycle.c:180
#12 0x000000000040d916 in main (argc=1, argv=0x7ffff2f42248) at src/core/nginx.c:412

@melong007

This comment has been minimized.

Copy link

melong007 commented Feb 24, 2014

207 if (async_fetch_ == NULL) {
208 LOG(FATAL)
209 << "BUG: NgxFetch callback called more than once on same fetch"
210 << str_url_.c_str() << "(" << this << ").Please report this at"
211 << "https://groups.google.com/forum/#!forum/ngx-pagespeed-discuss";
212 return;

@melong007

This comment has been minimized.

Copy link

melong007 commented Feb 24, 2014

oh, the bug was generated by the command "UseNativeFetcher on";

@oschaaf

This comment has been minimized.

Copy link
Member

oschaaf commented Feb 24, 2014

@melong007
Can you post or email the contents of the lines around this crash in error.log?
Starting about 2 minutes before the crasher would be ideal.
It would be especially interesting to know it this message was emitted before crashing:
NgxFetch: ngx_resolve_name failed, as that would confirm a suspicion about where this
originates from.

@jeffkaufman

This comment has been minimized.

Copy link
Contributor

jeffkaufman commented Feb 24, 2014

@melong007

This comment has been minimized.

Copy link

melong007 commented Feb 25, 2014

@oschaaf , I have send the nginx error.log file to your "we-amp.com email address";

@oschaaf

This comment has been minimized.

Copy link
Member

oschaaf commented Feb 25, 2014

@melong007 I received it - thanks!
@dinic I will forward the log so you can have a look as well

@oschaaf oschaaf self-assigned this Mar 26, 2014

@oschaaf

This comment has been minimized.

Copy link
Member

oschaaf commented Mar 26, 2014

It seems to me that two timers get armed in NgxFetch::Init():
NgxFetch::timeout_event_ and resolver_ctx_->timeout
Next to NgxFetch::timeout_event_, shouldn't we also clear resolver_ctx_->timeout if it is still armed in NgxFetch::CallbackDone()?

@dinic Any thoughts?

@oschaaf

This comment has been minimized.

Copy link
Member

oschaaf commented Apr 4, 2014

I'm able to reproduce this when I add this to nginx.conf:

    pagespeed UseNativeFetcher on;
    pagespeed MapProxyDomain mpd.example.com/gstatic_images http://www.gstatic.com/;   
    # point the resolver to google.com:81
    # the firewall will drop tcp SYN packets, which  is useful 
    # here to simulate a timeout for the dns lookup
    resolver 173.194.65.106:81;
    resolver_timeout 5s; # make it happen a little sooner

Subsequently, requesting
/mod_pagespeed_example/proxy_external_resource.html?PageSpeed=on&PageSpeedFilters=rewrite_images
will trigger the fetch, the dns timeout, and the check failure for the second call to NgxFetch::CallbackDone()

oschaaf added a commit that referenced this issue Apr 6, 2014

native-fetcher: fix CHECK failure on DNS timeout
Make sure we clear the timeout event associated with resolving
a hostname in `CallbackDone()`, which would otherwise end up
calling `CallbackDone()` a second time upon firing.

Fixes #625
@jeffkaufman

This comment has been minimized.

Copy link
Contributor

jeffkaufman commented Apr 9, 2014

@yaoweibin would you or @dinic have some time to look at how the native fetcher handles dns timeouts? @oschaaf has figured out how to reproduce this consistently with #625 (comment)

@jeffkaufman

This comment has been minimized.

Copy link
Contributor

jeffkaufman commented Apr 14, 2014

This is fixed in #655 right?

@oschaaf

This comment has been minimized.

Copy link
Member

oschaaf commented Apr 14, 2014

@jeffkaufman: "This is fixed in #655 right?"
Yes

@jmaessen jmaessen changed the title ngx_pagespeed coredump while ngx_resolver_timeout_handler CHECK failure on DNS timeout Apr 23, 2014

@pono pono unassigned oschaaf Jan 8, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment