options: change the default value for compaction threads to 8 (#169) #194
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
isaac-io
force-pushed
the
169-change-default-compaction-threads-from-1-to-8
branch
2 times, most recently
from
09f9928
to
49ce560
Compare
Yuval-Ariel
reviewed
Oct 20, 2022
Yuval-Ariel
reviewed
Oct 20, 2022
Yuval-Ariel
reviewed
Oct 20, 2022
Yuval-Ariel
approved these changes
Oct 20, 2022
This is done through the deprecated `max_background_compactions` option, which overrides `max_background_jobs`, so in order to avoid having an adverse effect on users who set `max_background_jobs` to a high value we try to detect that case and shift back to the user's choice in case there are enough background jobs to be enough for both flushes and compactions.
Yuval-Ariel
force-pushed
the
169-change-default-compaction-threads-from-1-to-8
branch
from
49ce560
to
0010a87
Compare
isaac-io
deleted the
169-change-default-compaction-threads-from-1-to-8
branch
isaac-io
added a commit
that referenced
this pull request
Oct 23, 2022
In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail, but they were fixed. Alas, the fixes seems to not be complete, as under stress some tests still fail. Make it so the tests are truly fixed now. This can be checked by running them in a loop with the machine overloaded: ``` $ while ./db_compaction_test --gtest_filter=DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1; do sleep 1; done ``` And ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCopyOptions; do sleep 1; done ```
isaac-io
added a commit
that referenced
this pull request
Oct 23, 2022
In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail, but they were fixed. Alas, the fixes seems to not be complete, as under stress some tests still fail. Make it so the tests are truly fixed now. This can be checked by running them in a loop with the machine overloaded: ``` $ while ./db_compaction_test --gtest_filter=DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1; do sleep 1; done ``` And ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCopyOptions; do sleep 1; done ```
isaac-io
added a commit
that referenced
this pull request
Oct 23, 2022
In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail, but they were fixed. Alas, the fixes seems to not be complete, as under stress some tests still fail. Make it so the tests are truly fixed now. This can be checked by running them in a loop with the machine overloaded: ``` $ while ./db_compaction_test --gtest_filter=DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1; do sleep 1; done ``` And ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCopyOptions; do sleep 1; done ```
isaac-io
added a commit
that referenced
this pull request
Oct 24, 2022
In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail, but they were fixed. Alas, the fixes seems to not be complete, as under stress some tests still fail. Make it so the tests are truly fixed now. This can be checked by running them in a loop with the machine overloaded: ``` $ while ./db_compaction_test --gtest_filter=DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1; do sleep 1; done ``` And ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCopyOptions; do sleep 1; done ```
isaac-io
added a commit
that referenced
this pull request
Oct 24, 2022
In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail, but they were fixed. Alas, the fixes seems to not be complete, as under stress some tests still fail. Make it so the tests are truly fixed now. This can be checked by running them in a loop with the machine overloaded: ``` $ while ./db_compaction_test --gtest_filter=DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1; do sleep 1; done ``` And ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCopyOptions; do sleep 1; done ```
isaac-io
added a commit
that referenced
this pull request
Oct 24, 2022
In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail, but they were fixed. Alas, the fixes seems to not be complete, as under stress some tests still fail. Make it so the tests are truly fixed now. This can be checked by running them in a loop with the machine overloaded: ``` $ while ./db_compaction_test --gtest_filter=DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1; do sleep 1; done ``` And ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCopyOptions; do sleep 1; done ```
isaac-io
added a commit
that referenced
this pull request
Nov 7, 2022
…#217) In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail and fixes were implemented as part of #197. Alas, it seems that the fixes weren't complete, as under stress a test still fail. Make it so the test is truly fixed now. This can be verified by running them in a loop with the machine overloaded: ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCFDropTest; do sleep 1; done ```
isaac-io
added a commit
that referenced
this pull request
Nov 8, 2022
…#217) In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail and fixes were implemented as part of #197. Alas, it seems that the fixes weren't complete, as under stress a test still fail. Make it so the test is truly fixed now. This can be verified by running them in a loop with the machine overloaded: ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCFDropTest; do sleep 1; done ```
Yuval-Ariel
pushed a commit
that referenced
this pull request
Nov 15, 2022
…#217) In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail and fixes were implemented as part of #197. Alas, it seems that the fixes weren't complete, as under stress a test still fail. Make it so the test is truly fixed now. This can be verified by running them in a loop with the machine overloaded: ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCFDropTest; do sleep 1; done ```
Yuval-Ariel
pushed a commit
that referenced
this pull request
Nov 24, 2022
In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail, but they were fixed. Alas, the fixes seems to not be complete, as under stress some tests still fail. Make it so the tests are truly fixed now. This can be checked by running them in a loop with the machine overloaded: ``` $ while ./db_compaction_test --gtest_filter=DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1; do sleep 1; done ``` And ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCopyOptions; do sleep 1; done ```
Yuval-Ariel
pushed a commit
that referenced
this pull request
Nov 25, 2022
In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail, but they were fixed. Alas, the fixes seems to not be complete, as under stress some tests still fail. Make it so the tests are truly fixed now. This can be checked by running them in a loop with the machine overloaded: ``` $ while ./db_compaction_test --gtest_filter=DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1; do sleep 1; done ``` And ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCopyOptions; do sleep 1; done ```
Yuval-Ariel
pushed a commit
that referenced
this pull request
Nov 25, 2022
…#217) In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail and fixes were implemented as part of #197. Alas, it seems that the fixes weren't complete, as under stress a test still fail. Make it so the test is truly fixed now. This can be verified by running them in a loop with the machine overloaded: ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCFDropTest; do sleep 1; done ```
Yuval-Ariel
pushed a commit
that referenced
this pull request
Nov 25, 2022
…#217) In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail and fixes were implemented as part of #197. Alas, it seems that the fixes weren't complete, as under stress a test still fail. Make it so the test is truly fixed now. This can be verified by running them in a loop with the machine overloaded: ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCFDropTest; do sleep 1; done ```
Yuval-Ariel
pushed a commit
that referenced
this pull request
Apr 30, 2023
In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail, but they were fixed. Alas, the fixes seems to not be complete, as under stress some tests still fail. Make it so the tests are truly fixed now. This can be checked by running them in a loop with the machine overloaded: ``` $ while ./db_compaction_test --gtest_filter=DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1; do sleep 1; done ``` And ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCopyOptions; do sleep 1; done ```
Yuval-Ariel
pushed a commit
that referenced
this pull request
Apr 30, 2023
…#217) In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail and fixes were implemented as part of #197. Alas, it seems that the fixes weren't complete, as under stress a test still fail. Make it so the test is truly fixed now. This can be verified by running them in a loop with the machine overloaded: ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCFDropTest; do sleep 1; done ```
Yuval-Ariel
pushed a commit
that referenced
this pull request
May 4, 2023
In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail, but they were fixed. Alas, the fixes seems to not be complete, as under stress some tests still fail. Make it so the tests are truly fixed now. This can be checked by running them in a loop with the machine overloaded: ``` $ while ./db_compaction_test --gtest_filter=DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1; do sleep 1; done ``` And ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCopyOptions; do sleep 1; done ```
Yuval-Ariel
pushed a commit
that referenced
this pull request
May 4, 2023
…#217) In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail and fixes were implemented as part of #197. Alas, it seems that the fixes weren't complete, as under stress a test still fail. Make it so the test is truly fixed now. This can be verified by running them in a loop with the machine overloaded: ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCFDropTest; do sleep 1; done ```
Yuval-Ariel
added a commit
that referenced
this pull request
Jul 24, 2023
The test sometimes failed in CI since the test code assumed there was only 1 bg compaction thread which needed to be blocked but in #194, the default was changed to 8 so the fix is simply to block all the bg threads.
Yuval-Ariel
added a commit
that referenced
this pull request
Aug 2, 2023
) The test sometimes failed in CI since the test code assumed there was only 1 bg compaction thread which needed to be blocked but in #194, the default was changed to 8 so the fix is simply to block all the bg threads.
udi-speedb
pushed a commit
that referenced
this pull request
Nov 13, 2023
In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail, but they were fixed. Alas, the fixes seems to not be complete, as under stress some tests still fail. Make it so the tests are truly fixed now. This can be checked by running them in a loop with the machine overloaded: ``` $ while ./db_compaction_test --gtest_filter=DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1; do sleep 1; done ``` And ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCopyOptions; do sleep 1; done ```
udi-speedb
pushed a commit
that referenced
this pull request
Nov 13, 2023
…#217) In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail and fixes were implemented as part of #197. Alas, it seems that the fixes weren't complete, as under stress a test still fail. Make it so the test is truly fixed now. This can be verified by running them in a loop with the machine overloaded: ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCFDropTest; do sleep 1; done ```
udi-speedb
pushed a commit
that referenced
this pull request
Nov 15, 2023
…#217) In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail and fixes were implemented as part of #197. Alas, it seems that the fixes weren't complete, as under stress a test still fail. Make it so the test is truly fixed now. This can be verified by running them in a loop with the machine overloaded: ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCFDropTest; do sleep 1; done ```
udi-speedb
pushed a commit
that referenced
this pull request
Nov 22, 2023
) The test sometimes failed in CI since the test code assumed there was only 1 bg compaction thread which needed to be blocked but in #194, the default was changed to 8 so the fix is simply to block all the bg threads.
udi-speedb
pushed a commit
that referenced
this pull request
Dec 3, 2023
In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail, but they were fixed. Alas, the fixes seems to not be complete, as under stress some tests still fail. Make it so the tests are truly fixed now. This can be checked by running them in a loop with the machine overloaded: ``` $ while ./db_compaction_test --gtest_filter=DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1; do sleep 1; done ``` And ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCopyOptions; do sleep 1; done ```
udi-speedb
pushed a commit
that referenced
this pull request
Dec 3, 2023
…#217) In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail and fixes were implemented as part of #197. Alas, it seems that the fixes weren't complete, as under stress a test still fail. Make it so the test is truly fixed now. This can be verified by running them in a loop with the machine overloaded: ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCFDropTest; do sleep 1; done ```
udi-speedb
pushed a commit
that referenced
this pull request
Dec 5, 2023
) The test sometimes failed in CI since the test code assumed there was only 1 bg compaction thread which needed to be blocked but in #194, the default was changed to 8 so the fix is simply to block all the bg threads.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is done through the deprecated
max_background_compactionsoption, which overridesmax_background_jobs, so in order to avoid having an adverse effect on users who setmax_background_jobsto a high value we try to detect that case and shift back to the user's choice in case there are enough background jobs to be enough for both flushes and compactions.Test plan: run
make checkwith the fixed unit tests.