Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: vm_preemption failed #123543

Open
cockroach-teamcity opened this issue May 3, 2024 · 48 comments
Open

roachtest: vm_preemption failed #123543

cockroach-teamcity opened this issue May 3, 2024 · 48 comments
Labels
branch-release-24.1.0-rc Used to mark GA and release blockers and technical advisories for 24.1.0-rc O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Projects
Test Engineering
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 3, 2024 •
edited by cockroach-jira-scripts

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 010b7b3b3f4ad23b2f0483d8d2b450961a51470c:

test cdc/tpcc-1000/sink=kafka failed: VMs preempted during the test run: teamcity-15094313-1714715342-33-n4cpu16-0001 (us-east1-c), teamcity-15094313-1714715342-33-n4cpu16-0003 (us-east1-c), teamcity-15094313-1714715342-33-n4cpu16-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_081708.834753091_n4_workload-run-tpcc-wa.log: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=kafka/run_1/ssh/ssh_081708.871427406_n4_workload-run-tpcc-wa.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:385).Close: error shutting down prometheus/grafana: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=kafka/run_1/ssh/ssh_083303.925682215_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=kafka/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-38407
@cockroach-teamcity cockroach-teamcity added branch-release-24.1.0-rc Used to mark GA and release blockers and technical advisories for 24.1.0-rc O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue labels May 3, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone May 3, 2024
@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 010b7b3b3f4ad23b2f0483d8d2b450961a51470c:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control failed: VMs preempted during the test run: teamcity-15094313-1714715342-35-n7cpu16-0002 (us-east1-c), teamcity-15094313-1714715342-35-n7cpu16-0005 (us-east1-c), teamcity-15094313-1714715342-35-n7cpu16-0003 (us-east1-c), teamcity-15094313-1714715342-35-n7cpu16-0007 (us-east1-c)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_082627.636236894_n7_cockroach-workload-i.log: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control/run_1/ssh/ssh_082627.657166653_n7_cockroach-workload-i.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 010b7b3b3f4ad23b2f0483d8d2b450961a51470c:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15094313-1714715342-36-n7cpu16-0002 (us-east1-c), teamcity-15094313-1714715342-36-n7cpu16-0005 (us-east1-c), teamcity-15094313-1714715342-36-n7cpu16-0003 (us-east1-c), teamcity-15094313-1714715342-36-n7cpu16-0004 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue May 3, 2024
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 010b7b3b3f4ad23b2f0483d8d2b450961a51470c:

test failover/partial/lease-liveness/lease=expiration failed: VMs preempted during the test run: teamcity-15094313-1714715342-61-n8cpu2-0007 (us-east1-d), teamcity-15094313-1714715342-61-n8cpu2-0006 (us-east1-d), teamcity-15094313-1714715342-61-n8cpu2-0008 (us-east1-d), teamcity-15094313-1714715342-61-n8cpu2-0002 (us-east1-d), teamcity-15094313-1714715342-61-n8cpu2-0001 (us-east1-d)

**Other Failures:**
(failover.go:1766).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2349).Run: context canceled
test artifacts and logs in: /artifacts/failover/partial/lease-liveness/lease=expiration/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 010b7b3b3f4ad23b2f0483d8d2b450961a51470c:

test ycsb/F/nodes=3 failed: VMs preempted during the test run: teamcity-15094313-1714715342-109-n4cpu8-0002 (us-east1-c), teamcity-15094313-1714715342-109-n4cpu8-0004 (us-east1-c), teamcity-15094313-1714715342-109-n4cpu8-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_121247.416874005_n4_workload-run-ycsb-in.log: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/F/nodes=3/run_1/ssh/ssh_121247.449931983_n4_workload-run-ycsb-in.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/F/nodes=3/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 010b7b3b3f4ad23b2f0483d8d2b450961a51470c:

test ycsb/C/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-15094313-1714715342-113-n4cpu32-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_124502.086957261_n4_workload-run-ycsb-in.log: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/C/nodes=3/cpu=32/run_1/ssh/ssh_124502.110364266_n4_workload-run-ycsb-in.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/C/nodes=3/cpu=32/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 34bbed697e77466f52012a80f0f6ae496c0d2889:

test cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15141345-1715060908-28-n6cpu16-0002 (us-east1-c), teamcity-15141345-1715060908-28-n6cpu16-0006 (us-east1-c)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_081612.379001022_n6_cockroach-workload-i.log: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1/ssh/ssh_081612.453956784_n6_cockroach-workload-i.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 34bbed697e77466f52012a80f0f6ae496c0d2889:

test failover/non-system/blackhole-recv/lease=expiration failed: VMs preempted during the test run: teamcity-15141345-1715060908-53-n7cpu2-0004 (us-east1-c), teamcity-15141345-1715060908-53-n7cpu2-0007 (us-east1-c)

**Other Failures:**
(failover.go:1766).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: full command output in run_093111.892895533_n7_cockroach-workload-r.log: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/failover/non-system/blackhole-recv/lease=expiration/run_1/ssh/ssh_093111.945250442_n7_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cluster.go:2349).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole-recv/lease=expiration/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 6205244e922606f85761dad2137b842f43a53716:

test tpccbench/nodes=3/cpu=16/lease=expiration failed: VMs preempted during the test run: teamcity-15157964-1715147575-97-n4cpu16-0003 (us-east1-c), teamcity-15157964-1715147575-97-n4cpu16-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2235).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/lease=expiration/run_1/ssh/ssh_142620.352566071_n3_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/lease=expiration/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 6205244e922606f85761dad2137b842f43a53716:

test tpccbench/nodes=9/cpu=4/multi-region failed: VMs preempted during the test run: teamcity-15157964-1715147575-95-n12cpu4hm-geo-0001 (us-east1-b), teamcity-15157964-1715147575-95-n12cpu4hm-geo-0004 (us-east1-b), teamcity-15157964-1715147575-95-n12cpu4hm-geo-0002 (us-east1-b), teamcity-15157964-1715147575-95-n12cpu4hm-geo-0003 (us-east1-b)

**Other Failures:**
(cluster.go:2286).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1/ssh/ssh_142736.859263082_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue May 9, 2024
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 6205244e922606f85761dad2137b842f43a53716:

test kv95/enc=false/nodes=4/ssds=8/raid0 failed: VMs preempted during the test run: teamcity-15175684-1715233756-77-n5cpu8-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2349).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=4/ssds=8/raid0/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=8
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 6205244e922606f85761dad2137b842f43a53716:

test decommissionBench/nodes=8/warehouses=3000 failed: VMs preempted during the test run: teamcity-15194589-1715320189-54-n9cpu16-0004 (us-east1-c), teamcity-15194589-1715320189-54-n9cpu16-0009 (us-east1-c), teamcity-15194589-1715320189-54-n9cpu16-0008 (us-east1-c)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_101510.948978808_n1_cockroach-workload-f.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/decommissionBench/nodes=8/warehouses=3000/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 6205244e922606f85761dad2137b842f43a53716:

test failover/chaos/read-write failed: VMs preempted during the test run: teamcity-15194589-1715320189-53-n10cpu2-0008 (us-east1-c), teamcity-15194589-1715320189-53-n10cpu2-0004 (us-east1-c), teamcity-15194589-1715320189-53-n10cpu2-0007 (us-east1-c), teamcity-15194589-1715320189-53-n10cpu2-0005 (us-east1-c), teamcity-15194589-1715320189-53-n10cpu2-0001 (us-east1-c), teamcity-15194589-1715320189-53-n10cpu2-0003 (us-east1-c)

**Other Failures:**
(failover.go:1766).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2349).Run: context canceled
(cluster.go:2349).Run: context canceled
(cluster.go:2349).Run: context canceled
(cluster.go:2349).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 6205244e922606f85761dad2137b842f43a53716:

test cdc/tpcc-1000/sink=kafka failed: VMs preempted during the test run: teamcity-15235383-1715579572-27-n4cpu16-0002 (us-east1-d), teamcity-15235383-1715579572-27-n4cpu16-0001 (us-east1-d), teamcity-15235383-1715579572-27-n4cpu16-0003 (us-east1-d), teamcity-15235383-1715579572-27-n4cpu16-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_074739.113614819_n4_workload-run-tpcc-wa.log: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=kafka/run_1/ssh/ssh_074739.171864295_n4_workload-run-tpcc-wa.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:385).Close: error shutting down prometheus/grafana: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=kafka/run_1/ssh/ssh_075752.866564132_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=kafka/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 1f64dc2c8b6735c692c212b8e21ed11a6ec3fefc:

test failover/non-system/crash/lease=expiration failed: VMs preempted during the test run: teamcity-15254676-1715665784-46-n7cpu2-0005 (us-east1-d), teamcity-15254676-1715665784-46-n7cpu2-0004 (us-east1-d), teamcity-15254676-1715665784-46-n7cpu2-0003 (us-east1-d), teamcity-15254676-1715665784-46-n7cpu2-0001 (us-east1-d), teamcity-15254676-1715665784-46-n7cpu2-0006 (us-east1-d), teamcity-15254676-1715665784-46-n7cpu2-0002 (us-east1-d), teamcity-15254676-1715665784-46-n7cpu2-0007 (us-east1-d)

**Other Failures:**
(failover.go:1766).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/failover/non-system/crash/lease=expiration/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 9d77dad6162bc528e725d0c57ec01c6a07d2aea2:

test allocbench/nodes=7/cpu=8/kv/r=50/ops=skew failed: VMs preempted during the test run: teamcity-15273625-1715752224-10-n8cpu8-0003 (us-east1-d), teamcity-15273625-1715752224-10-n8cpu8-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2341).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_073301.678204966_n4_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 9d77dad6162bc528e725d0c57ec01c6a07d2aea2:

test failover/chaos/read-write failed: VMs preempted during the test run: teamcity-15273625-1715752224-51-n10cpu2-0003 (us-east1-b), teamcity-15273625-1715752224-51-n10cpu2-0007 (us-east1-b), teamcity-15273625-1715752224-51-n10cpu2-0008 (us-east1-b), teamcity-15273625-1715752224-51-n10cpu2-0005 (us-east1-b), teamcity-15273625-1715752224-51-n10cpu2-0004 (us-east1-b)

**Other Failures:**
(failover.go:1766).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2349).Run: context canceled
(cluster.go:2349).Run: context canceled
(cluster.go:2349).Run: context canceled
(cluster.go:2349).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test allocbench/nodes=7/cpu=8/kv/r=95/ops=skew failed: VMs preempted during the test run: teamcity-15301994-1715925032-14-n8cpu8-0002 (us-east1-d), teamcity-15301994-1715925032-14-n8cpu8-0006 (us-east1-d)

**Other Failures:**
(cluster.go:2341).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1/ssh/ssh_062700.036079171_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test allocbench/nodes=7/cpu=8/kv/r=95/ops=skew failed: VMs preempted during the test run: teamcity-15314268-1716011412-07-n8cpu8-0007 (us-east1-d)

**Other Failures:**
(assertions.go:333).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:396
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
	            				main/pkg/cmd/roachtest/test_runner.go:1192
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	monitor failure: EOF [owner=test-eng]
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:142
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:395
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:1192
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) monitor failure
	            	Wraps: (3) secondary error attachment
	            	  | context canceled
	            	  | (1) context canceled
	            	  | Error types: (1) *errors.errorString
	            	Wraps: (4) EOF [owner=test-eng]
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) registry.ErrorWithOwnership
	Test:       	allocbench/nodes=7/cpu=8/kv/r=95/ops=skew
(require.go:1360).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test kv95/enc=false/nodes=3/cpu=96 failed: VMs preempted during the test run: teamcity-15314268-1716011412-107-n4cpu96-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2349).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=96/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=96
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test allocbench/nodes=7/cpu=8/kv/r=95/ops=skew failed: VMs preempted during the test run: teamcity-15321017-1716097919-10-n8cpu8-0006 (us-east1-c)

**Other Failures:**
(cluster.go:2341).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1/ssh/ssh_083233.469139098_n6_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15321017-1716097919-30-n6cpu16-0001 (us-east1-c), teamcity-15321017-1716097919-30-n6cpu16-0004 (us-east1-c), teamcity-15321017-1716097919-30-n6cpu16-0005 (us-east1-c), teamcity-15321017-1716097919-30-n6cpu16-0002 (us-east1-c), teamcity-15321017-1716097919-30-n6cpu16-0006 (us-east1-c)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_082341.757360761_n6_cockroach-workload-i.log: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1/ssh/ssh_082341.773275210_n6_cockroach-workload-i.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15321017-1716097919-29-n6cpu16-0003 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:142
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:395
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:1192
	            	Wraps: (2) monitor failure
	            	Wraps: (3) attached stack trace
	            	  -- stack trace:
	            	  | main.(*clusterImpl).RunE
	            	  | 	main/pkg/cmd/roachtest/cluster.go:2386
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.kvAllocBenchEventRunner.run
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:176
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchEvent
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:328
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:392
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	main/pkg/cmd/roachtest/monitor.go:120
	            	  | [...repeated from below...]
	            	Wraps: (4) full command output in run_055718.106601987_n8_workload-run-kv-dbsm.log
	            	Wraps: (5) Node 8. Command with error:
	            	  | ``````
	            	  | ./workload run kv --db=smallreads1 --read-percent=95 --min-block-bytes=1 --max-block-bytes=1 --max-rate=16500 --zipfian --tolerate-errors --concurrency=512 --duration=30m0s {pgurl:1-7}
	            	  | ``````
	            	  | <truncated> ...  3984 workload/pgx_helpers.go:240  [-] 4215  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
	            	  | W240521 05:57:22.439231 3969 workload/pgx_helpers.go:240  [-] 4216  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
	            	  | W240521 05:57:22.439284 3730 workload/pgx_helpers.go:240  [-] 4217  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
	            	  | W240521 05:57:22.439490 3844 workload/pgx_helpers.go:240  [-] 4218  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
	            	  | Connection to 34.139.148.87 closed by remote host.
	            	Wraps: (6) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
	            	  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
	            	  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (7) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1/ssh/ssh_055718.120456134_n8_workload-run-kv-dbsm.log)
	            	Wraps: (8) TRANSIENT_ERROR(ssh_problem): exit status 255
	            	  | (1) TRANSIENT_ERROR(ssh_problem): exit status 255
	            	  | Error types: (1) errors.TransientError
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *hintdetail.withDetail (6) *withstack.withStack (7) *errutil.withPrefix (8) errors.TransientError
	Test:       	allocbench/nodes=7/cpu=8/kv/r=95/access=skew
(require.go:1360).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15340294-1716270550-29-n6cpu16-0002 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: failed to list cockroach processes: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1/ssh/ssh_083211.756395763_n2_list-processes.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test tpccbench/nodes=3/cpu=16/mt-shared-process failed: VMs preempted during the test run: teamcity-15340294-1716270550-92-n4cpu16-0001 (us-east1-d), teamcity-15340294-1716270550-92-n4cpu16-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2235).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/mt-shared-process/run_1/ssh/ssh_120252.342301050_n2_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/mt-shared-process/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test cdc/tpcc-1000/sink=kafka failed: VMs preempted during the test run: teamcity-15356690-1716357341-28-n4cpu16-0002 (us-east1-c), teamcity-15356690-1716357341-28-n4cpu16-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_073637.466249922_n4_workload-run-tpcc-wa.log: COMMAND_PROBLEM: exit status 1
(cdc.go:385).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=kafka/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15356690-1716357341-26-n6cpu16-0006 (us-east1-c), teamcity-15356690-1716357341-26-n6cpu16-0003 (us-east1-c), teamcity-15356690-1716357341-26-n6cpu16-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_074142.151249797_n6_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test import/tpch/nodes=8 failed: VMs preempted during the test run: teamcity-15356690-1716357341-59-n8cpu4-0007 (us-east1-d), teamcity-15356690-1716357341-59-n8cpu4-0005 (us-east1-d)

**Other Failures:**
(test_runner.go:1216).runTest: test timed out (10h0m0s)
test artifacts and logs in: /artifacts/import/tpch/nodes=8/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test allocbench/nodes=7/cpu=8/kv/r=95/ops=skew failed: VMs preempted during the test run: teamcity-15371638-1716443308-14-n8cpu8-0002 (us-east1-b), teamcity-15371638-1716443308-14-n8cpu8-0005 (us-east1-b)

**Other Failures:**
(cluster.go:2341).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1/ssh/ssh_072941.473846640_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test c2c/tpcc/warehouses=1000/duration=60/cutover=30 failed: VMs preempted during the test run: teamcity-15371638-1716443308-25-n9cpu8-0009 (us-east1-c), teamcity-15371638-1716443308-25-n9cpu8-0006 (us-east1-c), teamcity-15371638-1716443308-25-n9cpu8-0005 (us-east1-c), teamcity-15371638-1716443308-25-n9cpu8-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_080043.678231439_n9_workload-init-tpcc-d.log: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/c2c/tpcc/warehouses=1000/duration=60/cutover=30/run_1/ssh/ssh_080043.698679304_n9_workload-init-tpcc-d.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/tpcc/warehouses=1000/duration=60/cutover=30/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15371638-1716443308-30-n6cpu16-0001 (us-east1-c), teamcity-15371638-1716443308-30-n6cpu16-0005 (us-east1-c), teamcity-15371638-1716443308-30-n6cpu16-0002 (us-east1-c), teamcity-15371638-1716443308-30-n6cpu16-0006 (us-east1-c)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_082131.514146926_n6_cockroach-workload-i.log: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1/ssh/ssh_082131.537475847_n6_cockroach-workload-i.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test allocbench/nodes=7/cpu=8/kv/r=50/ops=skew failed: VMs preempted during the test run: teamcity-15386564-1716529796-16-n8cpu8-0007 (us-east1-b), teamcity-15386564-1716529796-16-n8cpu8-0002 (us-east1-b)

**Other Failures:**
(assertions.go:333).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:396
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
	            				main/pkg/cmd/roachtest/test_runner.go:1192
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	monitor failure: EOF [owner=test-eng]
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:142
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:395
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:1192
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) monitor failure
	            	Wraps: (3) secondary error attachment
	            	  | context canceled
	            	  | (1) context canceled
	            	  | Error types: (1) *errors.errorString
	            	Wraps: (4) EOF [owner=test-eng]
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) registry.ErrorWithOwnership
	Test:       	allocbench/nodes=7/cpu=8/kv/r=50/ops=skew
(require.go:1360).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15386564-1716529796-39-n7cpu16-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_081125.578549853_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test failover/non-system/blackhole failed: VMs preempted during the test run: teamcity-15386564-1716529796-52-n7cpu2-0005 (us-east1-c), teamcity-15386564-1716529796-52-n7cpu2-0001 (us-east1-c), teamcity-15386564-1716529796-52-n7cpu2-0006 (us-east1-c), teamcity-15386564-1716529796-52-n7cpu2-0002 (us-east1-c), teamcity-15386564-1716529796-52-n7cpu2-0003 (us-east1-c), teamcity-15386564-1716529796-52-n7cpu2-0007 (us-east1-c)

**Other Failures:**
(failover.go:1766).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2349).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test failover/partial/lease-gateway failed: VMs preempted during the test run: teamcity-15386564-1716529796-58-n8cpu2-0008 (us-east1-d), teamcity-15386564-1716529796-58-n8cpu2-0005 (us-east1-d)

**Other Failures:**
(failover.go:1766).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: full command output in run_092259.413203853_n8_cockroach-workload-r.log: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/failover/partial/lease-gateway/run_1/ssh/ssh_092259.438270217_n8_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cluster.go:2349).Run: context canceled
test artifacts and logs in: /artifacts/failover/partial/lease-gateway/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test failover/chaos/read-write/lease=expiration failed: VMs preempted during the test run: teamcity-15386564-1716529796-51-n10cpu2-0004 (us-east1-d)

**Other Failures:**
(failover.go:1766).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2349).Run: context canceled
(cluster.go:2349).Run: context canceled
(cluster.go:2349).Run: context canceled
(cluster.go:2349).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/lease=expiration/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test kv95limited-spans/enc=false/nodes=1/cpu=32/seq/no-load-splitting failed: VMs preempted during the test run: teamcity-15399764-1716616465-84-n2cpu32-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2349).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95limited-spans/enc=false/nodes=1/cpu=32/seq/no-load-splitting/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test kv95/enc=false/nodes=3/size=64kb failed: VMs preempted during the test run: teamcity-15399764-1716616465-77-n4cpu8-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_110131.613201087_n4_workload-run-kv-tole.log: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=3/size=64kb/run_1/ssh/ssh_110131.652774480_n4_workload-run-kv-tole.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/size=64kb/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test c2c/tpcc/warehouses=1000/duration=60/cutover=30 failed: VMs preempted during the test run: teamcity-15409166-1716702582-26-n9cpu8-0003 (us-east1-b), teamcity-15409166-1716702582-26-n9cpu8-0002 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/tpcc/warehouses=1000/duration=60/cutover=30/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test cdc/tpcc-1000/sink=kafka failed: VMs preempted during the test run: teamcity-15417877-1716788934-26-n4cpu16-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_074145.760163416_n4_workload-run-tpcc-wa.log: COMMAND_PROBLEM: exit status 1
(cdc.go:385).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=kafka/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test kv95/enc=false/nodes=1/cpu=32 failed: VMs preempted during the test run: teamcity-15427633-1716875386-68-n2cpu32-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2349).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=1/cpu=32/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test kv95limited-spans/enc=false/nodes=1/cpu=32/seq/no-load-splitting failed: VMs preempted during the test run: teamcity-15427633-1716875386-70-n2cpu32-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_110202.579849134_n2_workload-run-kv-tole.log: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95limited-spans/enc=false/nodes=1/cpu=32/seq/no-load-splitting/run_1/ssh/ssh_110202.601319868_n2_workload-run-kv-tole.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95limited-spans/enc=false/nodes=1/cpu=32/seq/no-load-splitting/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control failed: VMs preempted during the test run: teamcity-15444778-1716961762-26-n7cpu16-0007 (us-east1-c), teamcity-15444778-1716961762-26-n7cpu16-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_075707.908189183_n7_cockroach-workload-i.log: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control/run_1/ssh/ssh_075707.936585545_n7_cockroach-workload-i.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test cdc/initial-scan-only/parquet failed: VMs preempted during the test run: teamcity-15454998-1717048104-28-n4cpu16-0001 (us-east1-d), teamcity-15454998-1717048104-28-n4cpu16-0004 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: pq: server is not accepting clients, try another node
test artifacts and logs in: /artifacts/cdc/initial-scan-only/parquet/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test allocbench/nodes=7/cpu=8/kv/r=50/ops=skew failed: VMs preempted during the test run: teamcity-15454998-1717048104-10-n8cpu8-0001 (us-east1-d), teamcity-15454998-1717048104-10-n8cpu8-0004 (us-east1-d), teamcity-15454998-1717048104-10-n8cpu8-0002 (us-east1-d), teamcity-15454998-1717048104-10-n8cpu8-0005 (us-east1-d), teamcity-15454998-1717048104-10-n8cpu8-0006 (us-east1-d)

**Other Failures:**
(cluster.go:2341).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_080030.750245993_n6_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue May 31, 2024
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test decommissionBench/nodes=4/warehouses=1000 failed: VMs preempted during the test run: teamcity-15465849-1717134699-40-n5cpu16-0004 (us-east1-b)

**Other Failures:**
(decommissionbench.go:721).runDecommissionBench: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/decommissionBench/nodes=4/warehouses=1000/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test tpccbench/nodes=9/cpu=4/multi-region failed: VMs preempted during the test run: teamcity-15465849-1717134699-101-n12cpu4hm-geo-0007 (us-west1-b), teamcity-15465849-1717134699-101-n12cpu4hm-geo-0005 (us-west1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

test import/tpch/nodes=8 failed: VMs preempted during the test run: teamcity-15465849-1717134699-64-n8cpu4-0002 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/import/tpch/nodes=8/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=true
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-24.1.0-rc Used to mark GA and release blockers and technical advisories for 24.1.0-rc O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Projects
Test Engineering
Triage
Milestone
24.1
Development

No branches or pull requests
1 participant
@cockroach-teamcity