Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: tpcc-nowait/isolation-level=mixed/nodes=3/w=1 failed #122678

Closed
cockroach-teamcity opened this issue Apr 19, 2024 · 7 comments
Closed

roachtest: tpcc-nowait/isolation-level=mixed/nodes=3/w=1 failed #122678

cockroach-teamcity opened this issue Apr 19, 2024 · 7 comments
Assignees
@nvanbenschoten
Labels
branch-release-24.1 Used to mark GA and release blockers and technical advisories for 24.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team
Projects
KV
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Apr 19, 2024 •
edited by cockroach-jira-scripts

roachtest.tpcc-nowait/isolation-level=mixed/nodes=3/w=1 failed with artifacts on release-24.1 @ 7185e6c311160061e0753e9ef1d66e8bb828fc98:

(monitor.go:154).Wait: monitor failure: full command output in run_121325.978388524_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/tpcc-nowait/isolation-level=mixed/nodes=3/w=1/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-38023
@cockroach-teamcity cockroach-teamcity added branch-release-24.1 Used to mark GA and release blockers and technical advisories for 24.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-testeng TestEng Team labels Apr 19, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Apr 19, 2024
@renatolabs
Copy link
Collaborator

Workload error:

Error: error in delivery: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): "sql txn" meta={id=c5128ed6 key=/Table/111/1/4/1 iso=ReadCommitted pri=0.01966623 epo=0 ts=1713529421.960800113,0 min=1713529421.882891658,0 seq=17} lock=true stat=ABORTED rts=1713529421.960800113,0 wto=false gul=1713529422.382891658,0 (SQLSTATE 40001)

Reassigning to KV.

@renatolabs renatolabs added T-kv KV Team and removed T-testeng TestEng Team labels Apr 19, 2024
@blathers-crl blathers-crl bot added this to Incoming in KV Apr 19, 2024
@renatolabs renatolabs removed this from Triage in Test Engineering Apr 19, 2024
@miraradeva
Copy link
Contributor

Looks like a duplicate of #119511, which should be fixed after #122293 is merged in.

@miraradeva miraradeva added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. GA-blocker P-3 Issues/test failures with no fix SLA and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Apr 19, 2024
@arulajmani arulajmani mentioned this issue Apr 22, 2024
Closed
@arulajmani
Copy link
Collaborator

@nvanbenschoten just a nudge to press merge on #122293.

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 23, 2024
Closed
@nvanbenschoten
Copy link
Member

#122293 isn't quite what we want, because I don't want to have to backport #122207. I'll put together a more targeted fix.

@nvanbenschoten nvanbenschoten mentioned this issue Apr 23, 2024
Merged
@nvanbenschoten
Copy link
Member

#122904 fixes this test on the release-24.1 branch without needing to backport #122207.

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 24, 2024
Closed
@nvanbenschoten
Copy link
Member

Removing the GA-blocker label, because #122904 is close to merging and this is a testing issue.

nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Apr 25, 2024
@nvanbenschoten
release-24.1: roachtest: use durable locking in tpcc-nowait/isolation…
d65b3d8 
…-level=mixed

Fixes cockroachdb#122678.

This commit updates the `tpcc-nowait/isolation-level=mixed/nodes=3/w=1`
roachtest to use durable locking for serializable transactions. This
ensures that we do not run into issues with best-effort locks acquired
by SELECT FOR UPDATE being lost and creating lock order inversions which
lead to transaction deadlocks.

Release note: None
@nvanbenschoten
Copy link
Member

Fixed by #122904.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nvanbenschoten nvanbenschoten

Labels
branch-release-24.1 Used to mark GA and release blockers and technical advisories for 24.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team
Projects
KV
Incoming
Milestone
24.1
Development

No branches or pull requests
5 participants
@nvanbenschoten @arulajmani @cockroach-teamcity @renatolabs @miraradeva