ccl/kvccl/kvtenantccl: TestTenantUpgradeFailure failed #97445
Assignees
Labels
branch-master
Failures on the master branch.
C-test-failure
Broken test (automatically or manually discovered).
O-robot
Originated from a bot.
Milestone
Comments
cockroach-teamcity
added
branch-master
|
Caused by #97424. I'll look into this. |
kvoli
added a commit
to kvoli/cockroach
that referenced
this issue
Feb 23, 2023
Previously, changing the rebalance objective could lead to inconsistent locking order between the load based splitter and rebalance objective. When the objective was updated, the previous method also blocked batch requests from completing until every replica lb splitter was reset. This commit moves the split objective to be a variable owned by the decider, rather than inferred on each decider operation. The split objective is updated on a rebalance objective change atomically over each replica but not atomically over a store. This removes the need for blocking batch requests until every replica is updated. Resolves: cockroachdb#97000 Resolves: cockroachdb#97445 Resolves: cockroachdb#97450 Resolves: cockroachdb#97452 Resolves: cockroachdb#97457 Release note: None
|
ccl/kvccl/kvtenantccl.TestTenantUpgradeFailure failed with artifacts on master @ e028ce5b14505dfd17ef8b13001c0ab8ac811e3c: Parameters: |
kvoli
added a commit
to kvoli/cockroach
that referenced
this issue
Feb 23, 2023
Previously, changing the rebalance objective could lead to inconsistent locking order between the load based splitter and rebalance objective. When the objective was updated, the previous method also blocked batch requests from completing until every replica lb splitter was reset. This commit moves the split objective to be a variable owned by the decider, rather than inferred on each decider operation. The split objective is updated on a rebalance objective change atomically over each replica but not atomically over a store. This removes the need for blocking batch requests until every replica is updated. Resolves: cockroachdb#97000 Resolves: cockroachdb#97445 Resolves: cockroachdb#97450 Resolves: cockroachdb#97452 Resolves: cockroachdb#97457 Release note: None
craig bot
pushed a commit
that referenced
this issue
Feb 24, 2023
97148: changefeedccl: Expire protected timestamps r=miretskiy a=miretskiy Changefeeds utilize protected timestamp system (PTS) to ensure that the data targeted by changefeed is not garbage collected prematurely. PTS record is managed by running changefeed by periodically updating PTS record timestamp, so that the data older than the that timestamp may be GCed. However, if the changefeed stops running when it is paused (either due to operator action, or due to `on_error=pause` option, the PTS record remains so that the changefeed can be resumed at a later time. However, it is also possible that operator may not notice that the job is paused for too long, thus causing buildup of garbage data. Excessive buildup of GC work is not great since it impacts overall cluster performance, and, once GC can resume, its cost is proportional to how much GC work needs to be done. This PR introduces a new changefeed option `gc_protect_expires_after` to automatically expire PTS records that are too old. This automatic expiration is a safety mechanism in case changefeed job gets paused by an operator or due to an error, while holding onto PTS record due to `protect_gc_on_pause` option. The operator is still expected to monitor changefeed jobs, and to restart paused changefeeds expediently. If the changefeed job remains paused, and the underlying PTS records expires, then the changefeed job will be canceled to prevent build up of GC data. Epic: [CRDB-21953](https://cockroachlabs.atlassian.net/browse/CRDB-21953) Informs #84598 Release note (enterprise change): Changefeed will automatically expire PTS records for paused jobs if changefeed is configured with `gc_protect_expires_after` option. 97539: kvserver: fix deadlock on rebalance obj change r=kvoli a=kvoli Previously, changing the rebalance objective could lead to inconsistent locking order between the load based splitter and rebalance objective. When the objective was updated, the previous method also blocked batch requests from completing until every replica lb splitter was reset. This commit moves the split objective to be a variable owned by the decider, rather than inferred on each decider operation. The split objective is updated on a rebalance objective change atomically over each replica but not atomically over a store. This removes the need for blocking batch requests until every replica is updated. Resolves: #97000 Resolves: #97445 Resolves: #97450 Resolves: #97452 Resolves: #97457 Release note: None Co-authored-by: Yevgeniy Miretskiy <yevgeniy@cockroachlabs.com> Co-authored-by: Austen McClernon <austen@cockroachlabs.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
ccl/kvccl/kvtenantccl.TestTenantUpgradeFailure failed with artifacts on master @ 286b3e235171a39b8f9910555affcc7ce310741a:
Parameters:
TAGS=bazel,gss,deadlockHelp
See also: How To Investigate a Go Test Failure (internal)
This test on roachdash | Improve this report!
Jira issue: CRDB-24699
The text was updated successfully, but these errors were encountered: