Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

profiled process can get stuck in a loop calling clone() #20

Open
tmm1 opened this Issue Jan 29, 2011 · 3 comments

Comments

Projects
None yet
2 participants
Owner

tmm1 commented Jan 29, 2011

20:35:22.718345 --- SIGPROF (Profiling timer expired) @ 0 (0) ---
20:35:22.718515 rt_sigaction(SIGSEGV, {0x7ffe94cf1150, [SEGV], SA_RESTORER|SA_RESTART, 0x7ffe9770daf0}, {0x477e80, [SEGV], SA_RESTORER|SA_RESTART, 0x7ffe9770daf0}, 8) = 0 <0.000022>
20:35:22.718672 rt_sigaction(SIGSEGV, {0x477e80, [SEGV], SA_RESTORER|SA_RESTART, 0x7ffe9770daf0}, {0x7ffe94cf1150, [SEGV], SA_RESTORER|SA_RESTART, 0x7ffe9770daf0}, 8) = 0 <0.000021>
20:35:22.718817 rt_sigreturn(0x34d4f7c) = 56 <0.000023>
20:35:22.718897 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7ffe9877f9f0) = ? ERESTARTNOINTR (To be restarted) <0.050648>
20:35:22.769627 --- SIGPROF (Profiling timer expired) @ 0 (0) ---
20:35:22.769797 rt_sigaction(SIGSEGV, {0x7ffe94cf1150, [SEGV], SA_RESTORER|SA_RESTART, 0x7ffe9770daf0}, {0x477e80, [SEGV], SA_RESTORER|SA_RESTART, 0x7ffe9770daf0}, 8) = 0 <0.000021>
20:35:22.769952 rt_sigaction(SIGSEGV, {0x477e80, [SEGV], SA_RESTORER|SA_RESTART, 0x7ffe9770daf0}, {0x7ffe94cf1150, [SEGV], SA_RESTORER|SA_RESTART, 0x7ffe9770daf0}, 8) = 0 <0.000021>
20:35:22.770097 rt_sigreturn(0x34d4f7c) = 56 <0.000022>
20:35:22.770176 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7ffe9877f9f0) = ? ERESTARTNOINTR (To be restarted) <0.054384>
20:35:22.824653 --- SIGPROF (Profiling timer expired) @ 0 (0) ---
20:35:22.824841 rt_sigaction(SIGSEGV, {0x7ffe94cf1150, [SEGV], SA_RESTORER|SA_RESTART, 0x7ffe9770daf0}, {0x477e80, [SEGV], SA_RESTORER|SA_RESTART, 0x7ffe9770daf0}, 8) = 0 <0.000021>
20:35:22.824998 rt_sigaction(SIGSEGV, {0x477e80, [SEGV], SA_RESTORER|SA_RESTART, 0x7ffe9770daf0}, {0x7ffe94cf1150, [SEGV], SA_RESTORER|SA_RESTART, 0x7ffe9770daf0}, 8) = 0 <0.000021>
20:35:22.825143 rt_sigreturn(0x34d4f7c) = 56 <0.000023>
20:35:22.825224 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7ffe9877f9f0) = ? ERESTARTNOINTR (To be restarted) <0.052579>

See also http://code.google.com/p/google-perftools/issues/detail?id=279

I'm seeing this behavior as well. I don't think it's the same as the behavior in the "see also", since there don't seem to be any segfaults there, and I'm seeing SIGSEGV all over (just like you are)

I'm on Centos 5.5

Any ideas about how to mitigate this?

Owner

tmm1 commented Mar 16, 2011

There're no segfaults- that's just an internal signal handler for segfaults that is setup during profile captures.

The solution is to decrease the profiling frequency with CPUPROFILE_FREQUENCY. Alternatively, you can switch any calls you're making to fork/system/backtick/popen to use https://github.com/rtomayko/posix-spawn, which will speed up the calls to clone and stop the profiler from interrupting them.

@tmm1 tmm1 referenced this issue Apr 9, 2013

Merged

Fix for Ruby 2.0.0 #56

Owner

tmm1 commented Apr 30, 2013

Simple work around for the fork+exec case. This won't cover other clone() wrappers like Kernel.system, `Kernel.`` etc

  class << Kernel
    def fork_with_pause(*args, &block)
      PerfTools::CpuProfiler.pause
      fork_without_pause(*args, &block)
    ensure
      PerfTools::CpuProfiler.resume
    end
    alias :fork_without_pause :fork
    alias :fork :fork_with_pause
  end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment