[PROF-4756] Fix profiler not restarting on Process.daemon
#2150
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I vaguely remember this being the case, but still got slightly confused by seeing the log around encoding, so hopefully this clarifies that the log does not mean that a profile would be reported.
When the Ruby VM `fork`s, only the thread that called `fork` survives in the child process. Thus, we have the `Forking` monkey patch which allows us to run callbacks in the child process, including restarting the profiler. But there's a less known API in Ruby that also makes use of forking -- `Process.daemon`. One example of using this API is when calling `rails server -d`; that `-d` makes Rails call this API. Because we were missing monkey patching `Process.daemon`, the profiler would not restart on `Process.daemon`. A customer using exactly `rails server -d` ran into this issue. Easy one-liner to validate that the profiler restarts after becoming a daemon: ``` $ DD_TRACE_DEBUG=true DD_PROFILING_ENABLED=true bundle exec ddtracerb exec ruby -e 'Process.daemon(nil, true); exit!' D, [#38994] (dd-trace-rb/lib/datadog/core/configuration/components.rb:371:in `startup!') Profiling started D, [#38994] (dd-trace-rb/lib/datadog/core/workers/async.rb:133:in `start_worker') Starting thread for: #<Datadog::Profiling::Collectors::OldStack:0x00007fb9c828b3e0> D, [#38994] (dd-trace-rb/lib/datadog/core/workers/async.rb:133:in `start_worker') Starting thread for: #<Datadog::Profiling::Scheduler:0x00007fb9c8289ab8> # <-- notice the PID change here D, [#39013] (dd-trace-rb/lib/datadog/core/workers/async.rb:133:in `start_worker') Starting thread for: #<Datadog::Profiling::Collectors::OldStack:0x00007fb9c828b3e0> D, [#39013] (dd-trace-rb/lib/datadog/core/workers/async.rb:133:in `start_worker') Starting thread for: #<Datadog::Profiling::Scheduler:0x00007fb9c8289ab8> ```
Without this, the test fails if the `Forking` patch has been applied by a different spec (due to RSpec not being able to properly setup mocks).
Base automatically changed from
ivoanjo/prof-4756-simplify-forking-monkey-patch
to
master
July 13, 2022 12:43
marcotc
approved these changes
Jul 15, 2022
|
Thanks Marco! I actually raised some questions upstream about making this easier to hook/find: https://bugs.ruby-lang.org/issues/18911 . |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?:
Extend the
Forkingmonkey patch to also cover theProcess.daemonAPI (that is another way of forking a Ruby process).Motivation:
When the Ruby VM
forks, only the thread that calledforksurvives in the child process.Thus, we have the
Forkingmonkey patch which allows us to run callbacks in the child process, including restarting the profiler.But there's a less known API in Ruby that also makes use of forking --
Process.daemon. One example of using this API is when callingrails server -d; that-dmakes Rails call this API.Because we were missing monkey patching
Process.daemon, the profiler would not restart onProcess.daemon. A customer using exactlyrails server -dran into this issue.Additional Notes:
This PR is on top of #2149 just to avoid any conflicts because of close by lines being removed, but both PRs are entirely independent.
How to test the change?
Easy one-liner to validate that the profiler restarts after becoming a daemon:
The daemon mode seems... broken(?) in Rails 7. With puma, no webserver would be started, and with webrick, a process would stay behind but not reply to any HTTP requests.
Nevertheless, I believe this fix should be enough (the customer reported this a long ago, this was a low priority fix).