Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: failover/chaos/read-write failed #124030

Closed
cockroach-teamcity opened this issue May 13, 2024 · 1 comment · Fixed by #124035
Closed

roachtest: failover/chaos/read-write failed #124030

cockroach-teamcity opened this issue May 13, 2024 · 1 comment · Fixed by #124035
Labels
A-testing Testing tools and infrastructure branch-release-23.2.5-rc C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-kv KV Team
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 13, 2024 •
edited by cockroach-jira-scripts
Loading

roachtest.failover/chaos/read-write failed with artifacts on release-23.2.5-rc @ 57ab10786d6ed277959c8990e2823772893ac873:

(test_runner.go:1134).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/failover/chaos/read-write/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-38670
@cockroach-teamcity cockroach-teamcity added branch-release-23.2.5-rc C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels May 13, 2024
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone May 13, 2024
@nvanbenschoten
Copy link
Member

It looks like this test just isn't being given enough time. It runs 20 iterations of failures, each of which include 3 different sleeps. It makes it to iteration 20 just as the test timeout hits. I'll bump the timeout and add more logging.

@nvanbenschoten nvanbenschoten added A-testing Testing tools and infrastructure C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels May 13, 2024
@nvanbenschoten nvanbenschoten mentioned this issue May 13, 2024
Merged
@cockroach-teamcity cockroach-teamcity mentioned this issue May 13, 2024
Closed
craig bot pushed a commit that referenced this issue May 15, 2024
@nvanbenschoten
Merge #124035
28bce06 
124035: roachtest: bump timeout of failover/chaos tests r=nvanbenschoten a=nvanbenschoten

Fixes #124030.

The test runs 20 iterations of failures, each of which include 3 sleeps, two of which take 1 minute. This is not quite enough time so the test occasionally times out when it is otherwise making progress.

Release note: None

Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
@craig craig bot closed this as completed in 3e5a211 May 15, 2024
blathers-crl bot pushed a commit that referenced this issue May 15, 2024
@nvanbenschoten
roachtest: bump timeout of failover/chaos tests
8ff8779 
Fixes #124030.

The test runs 20 iterations of failures, each of which include 3 sleeps,
two of which take 1 minute. This is not quite enough time so the test
occasionally times out when it is otherwise making progress.

Release note: None
@blathers-crl blathers-crl bot mentioned this issue May 15, 2024
Merged
blathers-crl bot pushed a commit that referenced this issue May 15, 2024
@nvanbenschoten
roachtest: bump timeout of failover/chaos tests
855567e 
Fixes #124030.

The test runs 20 iterations of failures, each of which include 3 sleeps,
two of which take 1 minute. This is not quite enough time so the test
occasionally times out when it is otherwise making progress.

Release note: None
@blathers-crl blathers-crl bot mentioned this issue May 15, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-testing Testing tools and infrastructure branch-release-23.2.5-rc C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-kv KV Team
Projects
No open projects
KV
Status: roachtest/unit test backlog
Milestone
23.2
Development

Successfully merging a pull request may close this issue.

roachtest: bump timeout of failover/chaos tests nvanbenschoten/cockroach
2 participants
@nvanbenschoten @cockroach-teamcity