Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/backupccl: TestRestorePauseOnError failed #121342

Open
cockroach-teamcity opened this issue Mar 29, 2024 · 4 comments
Open

ccl/backupccl: TestRestorePauseOnError failed #121342

cockroach-teamcity opened this issue Mar 29, 2024 · 4 comments
Labels
branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-disaster-recovery
Projects
Disaster Recovery Backlog
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 29, 2024 •
edited by cockroach-jira-scripts

ccl/backupccl.TestRestorePauseOnError failed on master @ a3ac7ebf958f25201c2696a17f996c0b9f86830f:

=== RUN   TestRestorePauseOnError
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestRestorePauseOnError98142672
    backup_test.go:9235: expected jobID 955604274711265281 to have status paused got succeeded
    testutils.go:289: no Invalid Descriptors
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestRestorePauseOnError98142672
--- FAIL: TestRestorePauseOnError (46.33s)

Parameters:

  • attempt=1
  • run=1
  • shard=12
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-37200
@cockroach-teamcity cockroach-teamcity added branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery labels Mar 29, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Mar 29, 2024
@lyang24 lyang24 mentioned this issue Mar 29, 2024
Merged
@rafiss rafiss mentioned this issue Mar 29, 2024
Merged
@cockroach-teamcity
Copy link
Member Author

ccl/backupccl.TestRestorePauseOnError failed on master @ 2a5e231716c436781f12452d800651f51c6383b7:

=== RUN   TestRestorePauseOnError
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestRestorePauseOnError2732496999
    test_server_shim.go:157: automatically injected a shared process virtual cluster under test; see comment at top of test_server_shim.go for details.
    backup_test.go:9235: expected jobID 955872203116740609 to have status paused got succeeded
    testutils.go:289: no Invalid Descriptors
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestRestorePauseOnError2732496999
--- FAIL: TestRestorePauseOnError (47.10s)

Parameters:

  • attempt=1
  • run=1
  • shard=12
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 30, 2024
Open
@cockroach-teamcity
Copy link
Member Author

ccl/backupccl.TestRestorePauseOnError failed on master @ 7fc4c7bcbbf0c75a62d056da0bf79a5a32714650:

=== RUN   TestRestorePauseOnError
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestRestorePauseOnError3675220197
    backup_test.go:9235: expected jobID 956168747870191617 to have status paused got succeeded
    testutils.go:289: no Invalid Descriptors
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestRestorePauseOnError3675220197
--- FAIL: TestRestorePauseOnError (46.48s)

Parameters:

  • attempt=1
  • run=1
  • shard=12
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/backupccl.TestRestorePauseOnError failed on master @ 7fc4c7bcbbf0c75a62d056da0bf79a5a32714650:

=== RUN   TestRestorePauseOnError
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestRestorePauseOnError2125256115
    test_server_shim.go:157: automatically injected an external process virtual cluster under test; see comment at top of test_server_shim.go for details.
    backup_test.go:9235: expected jobID 956422934812819457 to have status paused got succeeded
    testutils.go:289: no Invalid Descriptors
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestRestorePauseOnError2125256115
--- FAIL: TestRestorePauseOnError (47.92s)

Parameters:

  • attempt=1
  • run=3
  • shard=12
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@rytaft
Copy link
Collaborator

rytaft commented Apr 1, 2024

Hit this here: https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/14643589?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandBuildProblemsSection=true&expandBuildTestsSection=true

rickystewart added a commit to rickystewart/cockroach that referenced this issue Apr 1, 2024
@rickystewart
backupccl: skip TestRestorePauseOnError
cfd7267 
Flaky. See cockroachdb#121342

Epic: None
Release note: None
@rickystewart rickystewart mentioned this issue Apr 1, 2024
Merged
craig bot pushed a commit that referenced this issue Apr 1, 2024
@dt @jbowens
@mgartner @yuzefovich @rickystewart @itsbilal
Merge #121207 #121257 #121367 #121369 #121390 #121452 #121482 #121484 #…
c43f54c 
…121487

121207: rangefeed: add batch version of onValues r=dt a=dt

Release note: none.
Epic: none.

121257: kv/kvclient/kvtenant: allow downloading ones own spans r=stevendanna a=dt

Release note: none.
Epic: none.

121367: roachtest: expand WAL failover test assertions r=sumeerbhola a=jbowens

Update the WAL failover disk stall roachtest to assert that the stalled store does failover to the secondary and that SQL tail latencies remain bounded.

Epic: none
Release note: none

121369: ui: add goroutine scheduling latency graph to runtime dashboard r=mgartner a=mgartner

#### ui: add goroutine scheduling latency graph to runtime dashboard

Epic: None

Release note (ui change): The "Goroutine Scheduling Latency" graph has
been added to the Runtime page in Metrics.


121390: util/json,builtins: miscellaneous improvements around json objects r=yuzefovich a=yuzefovich

Fixes: #121326.

Release note: None

121452: dbconsole: use second granularity for jobs times r=dt a=dt

These milliseconds are not helpful to humans and just add noise that makes the actual times harder to spot.

Release note: none.
Epic: none.

121482: stress: reduce default `--runs_per_test` to 25 r=rail a=rickystewart

The nightlies are taking longer to run since the branch cut, so this should speed things up slightly.

Epic: CRDB-8308
Release note: None

121484: backupccl: skip `TestRestorePauseOnError` r=rail a=rickystewart

Flaky. See #121342

Epic: None
Release note: None

121487: go.mod: bump Pebble to eae0efc2a391 r=aadityasondhi a=itsbilal

Changes:

 * [`eae0efc2`](cockroachdb/pebble@eae0efc2) db: trigger flushable ingest in TestDeterminism
 * [`cc5b658d`](cockroachdb/pebble@cc5b658d) db: minor jobID cleanup
 * [`295c0099`](cockroachdb/pebble@295c0099) db: cancel compactions that overlap with flushable ingest+excise
 * [`a306d7e0`](cockroachdb/pebble@a306d7e0) db: add TestDeterminism
 * [`d3f89bff`](cockroachdb/pebble@d3f89bff) manifest: cleaner implementation of assertNotL0Cmp
 * [`1c7bcd1c`](cockroachdb/pebble@1c7bcd1c) manifest: add a helper for iterating through all levels
 * [`a9087af9`](cockroachdb/pebble@a9087af9) manifest: assert that we don't use Seek on L0 LevelMetadata
 * [`4f4be429`](cockroachdb/pebble@4f4be429) metamorphic: disable download op
 * [`e9624315`](cockroachdb/pebble@e9624315) base: add AssertionFailedf wrapper
 * [`016ef889`](cockroachdb/pebble@016ef889) metamorphic: run the level checker less frequently
 * [`1dc54dfd`](cockroachdb/pebble@1dc54dfd) metamorphic: add timestamps to logs
 * [`34502438`](cockroachdb/pebble@34502438) db: label download compactions logs

Fixes #121263.

Release note: none.
Epic: none.

Co-authored-by: David Taylor <tinystatemachine@gmail.com>
Co-authored-by: Jackson Owens <jackson@cockroachlabs.com>
Co-authored-by: Marcus Gartner <marcus@cockroachlabs.com>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
Co-authored-by: Ricky Stewart <ricky@cockroachlabs.com>
Co-authored-by: Bilal Akhtar <bilal@cockroachlabs.com>
@msbutler msbutler removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Apr 2, 2024
@msbutler msbutler added the P-2 Issues/test failures with a fix SLA of 3 months label Apr 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-disaster-recovery
Projects
Disaster Recovery Backlog
Backlog
Milestone
24.1
Development

No branches or pull requests
3 participants
@rytaft @cockroach-teamcity @msbutler