Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eliminate shutdown lag from the signal monitor #2766

Closed

Conversation

tobim
Copy link
Member

@tobim tobim commented Dec 6, 2022

The simple sleep_for used to introduce a shutdown delay of up to 750ms to every client command invocation. This replaces the naive sleep with wakeup on event logic, which is reused to signal the monitoring thread to stop.

Instructions for the reviewer:

  • Run count/export/etc. on an empty database and see for yourself that the command returns in less than 750ms.
  • Verify that regular interrupts still work with commands that run a little longer.

@tobim tobim added the performance Improvements or regressions of performance label Dec 6, 2022
@tobim tobim requested a review from lava December 6, 2022 22:33
The simple `sleep_for` used to introduce a shutdown delay of up to
750ms to every client command invocation. This replaces the naive
sleep with wakeup on event logic, which is reused to signal the
monitoring thread to stop.
@tobim tobim force-pushed the story/sc-39891/eliminate-signal-monitor-shutdown-lag branch from c52f6b9 to 85d32d4 Compare December 6, 2022 22:38
@@ -41,24 +41,30 @@ extern "C" void signal_monitor_handler(int sig) {
}
signals[0] = true;
signals[sig] = true;
cvp->notify_one();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The standard says:

The behavior of any function other than a [plain old function] used as a signal handler in a C++ program is implementation-defined.

The function notify_one is not from a "common subset", so the code is not well defined.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL! I dismissed my approval.

I found this great blog post on an approach that is not UB from what I can tell. This works by blocking signals in all threads except for the signal monitor one, and then having the signal monitor thread use sigwait rather than a condition variable.

https://thomastrapp.com/posts/signal-handlers-for-multithreaded-c++/#no-need-to-call-me-ill-sigwait-for-you

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I half expected that there was a catch. Well at least @dominiklohmann already found a blueprint for a more sound approach.

dominiklohmann
dominiklohmann previously approved these changes Dec 7, 2022
Copy link
Member

@dominiklohmann dominiklohmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the integration tests run around 90 seconds faster on my machine in a release build. Wow. Crazy to think we had not noticed this since Feb 2019 where this was introduced in 92fe590.

@@ -30,6 +29,7 @@ namespace {
// Keeps track of all signals by their value from 1 to 31. The flag at index 0
// is used to tell whether a signal has been raised or not.
std::atomic<bool> signals[32];
std::condition_variable* cvp;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand the comment above correctly then this means that signals[0] is no longer required, because this condition variable serves the same purpose.

Comment on lines +29 to +30
static std::condition_variable cv;
static std::mutex m;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can tell these two only need to be in the header because run_guarded is defined inline, which it doesn't have to be.

@@ -30,6 +29,7 @@ namespace {
// Keeps track of all signals by their value from 1 to 31. The flag at index 0
// is used to tell whether a signal has been raised or not.
std::atomic<bool> signals[32];
std::condition_variable* cvp;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need the cvp in addition to cv? Can't we always access cv directly?

Comment on lines 51 to +53
std::atomic<bool> signal_monitor::stop;
std::condition_variable signal_monitor::cv;
std::mutex signal_monitor::m;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can get rid of these three lines if you mark the variables as inline static in the header.

@dominiklohmann dominiklohmann dismissed their stale review December 7, 2022 08:09

Dismissing based on @mavam's review–did not know this was UB.

@tobim
Copy link
Member Author

tobim commented Dec 7, 2022

As explained in #2766 (comment) this is technically not allowed.

@tobim tobim closed this Dec 7, 2022
@tobim tobim deleted the story/sc-39891/eliminate-signal-monitor-shutdown-lag branch December 13, 2022 07:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Improvements or regressions of performance
Projects
None yet
3 participants