DisableManualCompaction may fail to cancel an unscheduled task #9659

jay-zhuang · 2022-03-04T20:38:52Z

#9625 didn't change the unschedule condition which was waiting for the background thread to clean-up the compaction.
make sure we only unschedule the task when it's scheduled.

facebook-github-bot · 2022-03-04T20:39:08Z

@jay-zhuang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-03-04T21:51:20Z

@jay-zhuang has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-03-04T21:51:37Z

@jay-zhuang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-03-04T22:31:25Z

@jay-zhuang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

ajkr · 2022-03-08T00:31:15Z

db/db_impl/db_impl_compaction_flush.cc

        manual.done = true;
        manual.status =
            Status::Incomplete(Status::SubCode::kManualCompactionPaused);


If a compaction has been scheduled and is currently executing, should we still wait for !manual.in_progress?

edit: For example if RunManualCompaction() ends while BG compaction is still running, BG compaction may attempt to modify an already-deleted ManualCompactionState.

What if we have this just below the bg_cv_.Wait() ?

if (manual.in_progress) { continue; }

Thanks for pointing out. after looking into it, just checking manual.in_progress might not be enough, because the BG thread might already picked the compaction task but not yet update the manual.in_progress (as the thread-pool queue is not protected by db.mutex, I just added an unittest for such case).
How about make manual a shared_ptr?

(other options I considered but has their problems:

env_->UnSchedule() returns the number of tasks it cancelled, we can check that to make sure a scheduled compaction is cancelled, otherwise wait. The problem is it cannot work for multiple non-exclusive manual compactions as one manual compaction will unschedule() 2 jobs, another won't unschedule() any.

add manual job id in the Tag (hacky and may have problem if there're too many manual compactions)

coordinate between all FG manual compaction threads to make sure all tasks are unscheduled, if not, wait. (I think it's possible, but seems more complicated than just make manual a shared_ptr.)

facebook-github-bot · 2022-03-09T20:12:13Z

@jay-zhuang has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-03-09T20:32:47Z

@jay-zhuang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-03-09T21:47:16Z

@jay-zhuang has updated the pull request. You must reimport the pull request before landing.

ajkr · 2022-03-11T22:23:25Z

db/db_compaction_test.cc

+  db_->DisableManualCompaction();
+
+  compact_thread.join();
+}


These tests work for me before the fix. Are they expected to fail?

DisableJustStartedManualCompaction should fail. I consistently get segfault on macos:

For DisableMultiManualCompaction, yes, it success without the fix. It is used to test one foreground manual compaction unschedule 2 tasks from the queue and the other one can still return with InComplete status.

Yeah, it's fine on linux, even passed ASAN and TSAN...

Thanks for the info - tests LGTM now. Some notes below -

The first test could produce a hang without the fix if I deleted the early-return in RunManualCompaction() based on manual compaction being paused. That early return seems to prevent us from actually scheduling background compactions to the low pri pool.

Regarding the second test, there might be something platform dependent about accessing a variable from an already-returned stack frame. I can't find an error even though the ManualCompactionState clearly went out of scope and should've been destroyed.

Yeah, it's fine on linux, even passed ASAN and TSAN...

I have some task about enabling ASAN_OPTIONS=detect_stack_use_after_return=1 (T94055374). Can't remember what motivated that but it'd be interesting to see if it'd help here.

yes, it helps with ASAN_OPTIONS="detect_stack_use_after_return=1":

(another hacky way to force it fail is calling some other functions after compactRange() to mess up the stack, but I think we should use detect_stack_use_after_return).

ajkr

LGTM!

facebook-github-bot · 2022-03-13T02:21:40Z

@jay-zhuang has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-03-13T02:22:15Z

@jay-zhuang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Summary: #9625 didn't change the unschedule condition which was waiting for the background thread to clean-up the compaction. make sure we only unschedule the task when it's scheduled. Pull Request resolved: #9659 Reviewed By: ajkr Differential Revision: D34651820 Pulled By: jay-zhuang fbshipit-source-id: 23f42081b15ec8886cd81cbf131b116e0c74dc2f

In facebook#9659, when `DisableManualCompaction()` is issued, the foreground manual compaction thread does not have to wait background compaction thread to finish. Which could be a problem that the user re-enable manual compaction with `EnableManualCompaction()`, it may re-enable the BG compaction which supposed be cancelled. This patch makes the FG compaction wait on `manual_compaction_state.done`, which either be set by BG compaction or Unschedule callback. Then when FG manual compaction thread returns, it should not have BG compaction running. So shared_ptr is no longer needed for `manual_compaction_state`. Test Plan: a StressTest and unittest

Summary: In #9659, when `DisableManualCompaction()` is issued, the foreground manual compaction thread does not have to wait background compaction thread to finish. Which could be a problem that the user re-enable manual compaction with `EnableManualCompaction()`, it may re-enable the BG compaction which supposed be cancelled. This patch makes the FG compaction wait on `manual_compaction_state.done`, which either be set by BG compaction or Unschedule callback. Then when FG manual compaction thread returns, it should not have BG compaction running. So shared_ptr is no longer needed for `manual_compaction_state`. Pull Request resolved: #9694 Test Plan: a StressTest and unittest Reviewed By: ajkr Differential Revision: D34885472 Pulled By: jay-zhuang fbshipit-source-id: e6476175b43e8c59cd49f5c09241036a0716c274

Summary: #9625 didn't change the unschedule condition which was waiting for the background thread to clean-up the compaction. make sure we only unschedule the task when it's scheduled. Pull Request resolved: #9659 Reviewed By: ajkr Differential Revision: D34651820 Pulled By: jay-zhuang fbshipit-source-id: 23f42081b15ec8886cd81cbf131b116e0c74dc2f

Summary: In #9659, when `DisableManualCompaction()` is issued, the foreground manual compaction thread does not have to wait background compaction thread to finish. Which could be a problem that the user re-enable manual compaction with `EnableManualCompaction()`, it may re-enable the BG compaction which supposed be cancelled. This patch makes the FG compaction wait on `manual_compaction_state.done`, which either be set by BG compaction or Unschedule callback. Then when FG manual compaction thread returns, it should not have BG compaction running. So shared_ptr is no longer needed for `manual_compaction_state`. Pull Request resolved: #9694 Test Plan: a StressTest and unittest Reviewed By: ajkr Differential Revision: D34885472 Pulled By: jay-zhuang fbshipit-source-id: e6476175b43e8c59cd49f5c09241036a0716c274

facebook-github-bot added the CLA Signed label Mar 4, 2022

jay-zhuang changed the title ~~Avoid DisableManualCompaction to unschedule an unscheduled task~~ Avoid DisableManualCompaction to cancel an unscheduled task Mar 4, 2022

jay-zhuang changed the title ~~Avoid DisableManualCompaction to cancel an unscheduled task~~ DisableManualCompaction may unable to cancel an unscheduled task Mar 4, 2022

jay-zhuang changed the title ~~DisableManualCompaction may unable to cancel an unscheduled task~~ DisableManualCompaction may fail to cancel an unscheduled task Mar 4, 2022

ajkr reviewed Mar 8, 2022

View reviewed changes

jay-zhuang force-pushed the unschedule_issue branch from 2c42088 to 627358f Compare March 9, 2022 20:12

jay-zhuang marked this pull request as ready for review March 9, 2022 20:32

ajkr reviewed Mar 11, 2022

View reviewed changes

ajkr approved these changes Mar 12, 2022

View reviewed changes

jay-zhuang added 7 commits March 12, 2022 18:11

Avoid DisableManualCompaction to unschedule an unscheduled task

259d3c2

Update HISTORY

658cc06

use shared manual compaction state

505e704

Update

f87184c

Add test

1c4778e

Fix test

9cd8fa7

Remove debug from facebook#9656

ce243df

jay-zhuang force-pushed the unschedule_issue branch from 302f192 to ce243df Compare March 13, 2022 02:21

facebook-github-bot closed this in 4dff279 Mar 13, 2022

jay-zhuang mentioned this pull request Mar 15, 2022

Fix a race condition when disable and enable manual compaction #9694

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DisableManualCompaction may fail to cancel an unscheduled task #9659

DisableManualCompaction may fail to cancel an unscheduled task #9659

jay-zhuang commented Mar 4, 2022 •

edited

Loading

facebook-github-bot commented Mar 4, 2022

facebook-github-bot commented Mar 4, 2022

facebook-github-bot commented Mar 4, 2022

facebook-github-bot commented Mar 4, 2022

ajkr Mar 8, 2022 •

edited

Loading

jay-zhuang Mar 9, 2022 •

edited

Loading

facebook-github-bot commented Mar 9, 2022

facebook-github-bot commented Mar 9, 2022

facebook-github-bot commented Mar 9, 2022

ajkr Mar 11, 2022

jay-zhuang Mar 12, 2022 •

edited

Loading

jay-zhuang Mar 12, 2022

ajkr Mar 12, 2022

ajkr Mar 12, 2022

jay-zhuang Mar 12, 2022

ajkr left a comment

facebook-github-bot commented Mar 13, 2022

facebook-github-bot commented Mar 13, 2022

DisableManualCompaction may fail to cancel an unscheduled task #9659

DisableManualCompaction may fail to cancel an unscheduled task #9659

Conversation

jay-zhuang commented Mar 4, 2022 • edited Loading

facebook-github-bot commented Mar 4, 2022

facebook-github-bot commented Mar 4, 2022

facebook-github-bot commented Mar 4, 2022

facebook-github-bot commented Mar 4, 2022

ajkr Mar 8, 2022 • edited Loading

Choose a reason for hiding this comment

jay-zhuang Mar 9, 2022 • edited Loading

Choose a reason for hiding this comment

facebook-github-bot commented Mar 9, 2022

facebook-github-bot commented Mar 9, 2022

facebook-github-bot commented Mar 9, 2022

ajkr Mar 11, 2022

Choose a reason for hiding this comment

jay-zhuang Mar 12, 2022 • edited Loading

Choose a reason for hiding this comment

jay-zhuang Mar 12, 2022

Choose a reason for hiding this comment

ajkr Mar 12, 2022

Choose a reason for hiding this comment

ajkr Mar 12, 2022

Choose a reason for hiding this comment

jay-zhuang Mar 12, 2022

Choose a reason for hiding this comment

ajkr left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Mar 13, 2022

facebook-github-bot commented Mar 13, 2022

jay-zhuang commented Mar 4, 2022 •

edited

Loading

ajkr Mar 8, 2022 •

edited

Loading

jay-zhuang Mar 9, 2022 •

edited

Loading

jay-zhuang Mar 12, 2022 •

edited

Loading