Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: vm_preemption failed #121348

Open
cockroach-teamcity opened this issue Mar 29, 2024 · 274 comments
Open

roachtest: vm_preemption failed #121348

cockroach-teamcity opened this issue Mar 29, 2024 · 274 comments
Labels
branch-release-24.1 Used to mark GA and release blockers and technical advisories for 24.1 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Projects
Test Engineering
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 29, 2024 •
edited by cockroach-jira-scripts

roachtest.vm_preemption failed with artifacts on release-24.1 @ 557f4e6fe89d683d49082d88ba5adf6660e8f16f:

test allocbench/nodes=7/cpu=8/kv/r=95/ops=skew failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14610723-1711690376-23-n8cpu8-0006

**Other Failure**
(cluster.go:2336).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-37206
@cockroach-teamcity cockroach-teamcity added branch-release-24.1 Used to mark GA and release blockers and technical advisories for 24.1 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue labels Mar 29, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Mar 29, 2024
@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 557f4e6fe89d683d49082d88ba5adf6660e8f16f:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=processor/protocol=mux/format=json/sink=null failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14610723-1711690376-75-n7cpu16-0005

**Other Failure**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=processor/protocol=mux/format=json/sink=null/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 5d952f80b3e1efe2e9aaed73f1fd68433880fcb7:

test cdc/tpcc-1000/sink=kafka failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-14626096-1711776586-53-n4cpu16-0001

**Other Failure**
(cluster.go:2344).Run: full command output in run_110007.845201502_n4_workload-run-tpcc-wa.log: COMMAND_PROBLEM: exit status 1
(cdc.go:385).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=kafka/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 5d952f80b3e1efe2e9aaed73f1fd68433880fcb7:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-14626096-1711776586-62-n7cpu16-0002, projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-14626096-1711776586-62-n7cpu16-0006

**Other Failure**
(cluster.go:2230).Start: failed to find services to register: failed to find 1 open ports: expected 1 ports, got 0
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 1, 2024
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 5d952f80b3e1efe2e9aaed73f1fd68433880fcb7:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/control failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-62-n7cpu16-0006, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-62-n7cpu16-0002, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-62-n7cpu16-0003, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-62-n7cpu16-0001, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-62-n7cpu16-0004

**Other Failure**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100/control/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 5d952f80b3e1efe2e9aaed73f1fd68433880fcb7:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-61-n7cpu16-0002, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-61-n7cpu16-0001, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-61-n7cpu16-0005, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-61-n7cpu16-0006, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-61-n7cpu16-0003, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-61-n7cpu16-0007

**Other Failure**
(assertions.go:333).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_bench.go:506
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_bench.go:147
	            				main/pkg/cmd/roachtest/test_runner.go:1161
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	failed 5 attempts to get changefeed info: dial tcp 34.148.92.85:29000: connect: connection timed out
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForChangefeed
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_bench.go:592
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runCDCBenchWorkload
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_bench.go:498
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDCBench.func3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_bench.go:147
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:1161
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) failed 5 attempts to get changefeed info
	            	Wraps: (3) dial tcp 34.148.92.85:29000
	            	Wraps: (4) connect
	            	Wraps: (5) connection timed out
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *net.OpError (4) *os.SyscallError (5) syscall.Errno
	Test:       	cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null
(require.go:1360).NoError: FailNow called
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 5d952f80b3e1efe2e9aaed73f1fd68433880fcb7:

test failover/non-system/deadlock failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-94-n7cpu2-0004, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-94-n7cpu2-0001, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-94-n7cpu2-0002

**Other Failure**
(failover.go:1774).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/failover/non-system/deadlock/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 5d952f80b3e1efe2e9aaed73f1fd68433880fcb7:

test kv95/enc=false/nodes=3/size=4kb failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-14644007-1711949435-140-n4cpu8-0004

**Other Failure**
(cluster.go:2344).Run: full command output in run_173755.033960686_n4_workload-run-kv-tole.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/size=4kb/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 5d952f80b3e1efe2e9aaed73f1fd68433880fcb7:

test kv95/enc=false/nodes=4 failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-151-n5cpu8-0001

**Other Failure**
(cluster.go:2344).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=4/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 5d952f80b3e1efe2e9aaed73f1fd68433880fcb7:

test restore/tpce/400GB/gce/nodes=4/cpus=8/lowmem failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14644007-1711949435-158-n4cpu8lm-0002

**Other Failure**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/restore/tpce/400GB/gce/nodes=4/cpus=8/lowmem/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 5d952f80b3e1efe2e9aaed73f1fd68433880fcb7:

test ycsb/B/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-173-n4cpu32-0002

**Other Failure**
(cluster.go:2344).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/B/nodes=3/cpu=32/mvcc-range-keys=global/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 5d952f80b3e1efe2e9aaed73f1fd68433880fcb7:

test kv50/enc=false/nodes=4/cpu=96/batch=64 failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-181-n5cpu96-0003, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14644007-1711949435-181-n5cpu96-0004

**Other Failure**
(cluster.go:2344).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv50/enc=false/nodes=4/cpu=96/batch=64/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=96
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ db4521b322585f7b70b4a79b62a0623c5c1aee7b:

test allocbench/nodes=7/cpu=8/kv/r=50/ops=skew failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-14660571-1712035805-22-n8cpu8-0006, projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-14660571-1712035805-22-n8cpu8-0004

**Other Failure**
(cluster.go:2336).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ db4521b322585f7b70b4a79b62a0623c5c1aee7b:

test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14660571-1712035805-53-n6cpu16-0006

**Other Failure**
(cluster.go:2344).Run: full command output in run_102407.050226468_n6_cockroach-workload-i.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ db4521b322585f7b70b4a79b62a0623c5c1aee7b:

test cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14660571-1712035805-50-n6cpu16-0006

**Other Failure**
(cluster.go:2344).Run: full command output in run_102932.937902761_n6_cockroach-workload-i.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ db4521b322585f7b70b4a79b62a0623c5c1aee7b:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/control failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14660571-1712035805-68-n7cpu16-0001, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14660571-1712035805-68-n7cpu16-0003, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14660571-1712035805-68-n7cpu16-0007

**Other Failure**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100/control/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ db4521b322585f7b70b4a79b62a0623c5c1aee7b:

test failover/chaos/read-only failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-79-n10cpu2-0003

**Other Failure**
(failover.go:1774).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2344).Run: context canceled
(cluster.go:2344).Run: context canceled
(cluster.go:2344).Run: context canceled
(cluster.go:2344).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-only/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ db4521b322585f7b70b4a79b62a0623c5c1aee7b:

test decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=3/multi-region failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14660571-1712035805-107-n8cpu16-geo-0001

**Other Failure**
(cluster.go:2230).Start: failed to find services to register: failed to find 1 open ports: expected 1 ports, got 0
test artifacts and logs in: /artifacts/decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=3/multi-region/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ db4521b322585f7b70b4a79b62a0623c5c1aee7b:

test import/tpcc/warehouses=1000/nodes=32 failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0015, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0008, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0026, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0023, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0005, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0028, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0011, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0020, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0027, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0001, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0019, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0003, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0006, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0010, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0017, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0012, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0018, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0031, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0030, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0013, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0032, projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-14660571-1712035805-124-n32cpu4-0009

**Other Failure**
(cluster.go:2344).Run: context canceled
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/import/tpcc/warehouses=1000/nodes=32/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=true
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ db4521b322585f7b70b4a79b62a0623c5c1aee7b:

test indexes/2/nodes=6/multi-region failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14660571-1712035805-127-n7cpu16-geo-0002

**Other Failure**
(cluster.go:2344).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/indexes/2/nodes=6/multi-region/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ db4521b322585f7b70b4a79b62a0623c5c1aee7b:

test kv95/enc=false/nodes=1/cpu=32/mt-shared-process failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14660571-1712035805-144-n2cpu32-0002

**Other Failure**
(cluster.go:2344).Run: full command output in run_171125.838970330_n2_workload-run-kv-tole.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=1/cpu=32/mt-shared-process/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ db4521b322585f7b70b4a79b62a0623c5c1aee7b:

test kv95/enc=false/nodes=3/size=64kb failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-14660571-1712035805-148-n4cpu8-0004

**Other Failure**
(cluster.go:2344).Run: full command output in run_172959.241923035_n4_workload-run-kv-tole.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/size=64kb/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ db4521b322585f7b70b4a79b62a0623c5c1aee7b:

test scrub/all-checks/tpcc/w=100 failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14660571-1712035805-170-n5cpu4-0004

**Other Failure**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/scrub/all-checks/tpcc/w=100/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 7c259a1b369b9de45995d76609e41028e5ee0ab4:

test kv95/enc=false/nodes=3/size=64kb failed: VMs preempted during the test run: teamcity-15157949-1715147360-149-n4cpu8-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2349).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/size=64kb/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 7c259a1b369b9de45995d76609e41028e5ee0ab4:

test kv95/enc=true/nodes=3 failed: VMs preempted during the test run: teamcity-15157949-1715147360-152-n4cpu8-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2349).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=true/nodes=3/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=true
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 7c259a1b369b9de45995d76609e41028e5ee0ab4:

test tpcc/headroom/isolation-level=read-committed/n4cpu16 failed: VMs preempted during the test run: teamcity-15157949-1715147360-162-n4cpu16-0003 (us-east1-c), teamcity-15157949-1715147360-162-n4cpu16-0002 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=true
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 4665969fb6f4a4c1b58f10d9f99799f1939a0d62:

test cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15175669-1715233897-50-n6cpu16-0006 (us-east1-d)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_105639.733197071_n6_cockroach-workload-i.log: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null/run_1/ssh/ssh_105639.756548223_n6_cockroach-workload-i.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 4665969fb6f4a4c1b58f10d9f99799f1939a0d62:

test c2c/tpcc/warehouses=1000/duration=60/cutover=30 failed: VMs preempted during the test run: teamcity-15175669-1715233897-39-n9cpu8-0009 (us-east1-d), teamcity-15175669-1715233897-39-n9cpu8-0002 (us-east1-d), teamcity-15175669-1715233897-39-n9cpu8-0008 (us-east1-d), teamcity-15175669-1715233897-39-n9cpu8-0001 (us-east1-d), teamcity-15175669-1715233897-39-n9cpu8-0004 (us-east1-d), teamcity-15175669-1715233897-39-n9cpu8-0007 (us-east1-d), teamcity-15175669-1715233897-39-n9cpu8-0006 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/tpcc/warehouses=1000/duration=60/cutover=30/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 4665969fb6f4a4c1b58f10d9f99799f1939a0d62:

test connection_latency/nodes=9/multiregion/password failed: VMs preempted during the test run: teamcity-15175669-1715233897-61-n12cpu4-geo-0012 (europe-west2-b)

**Other Failures:**
(assertions.go:333).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/connection_latency.go:38
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/connection_latency.go:144
	            				main/pkg/cmd/roachtest/test_runner.go:1205
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/workload.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/workload.linux-amd64 ubuntu@34.89.59.223:./workload
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*clusterImpl).PutE
	            	  | 	main/pkg/cmd/roachtest/cluster.go:1836
	            	  | [...repeated from below...]
	            	Wraps: (2) cluster.PutE
	            	Wraps: (3) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Put
	            	  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2230
	            	  | github.com/cockroachdb/cockroach/pkg/roachprod.Put
	            	  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:892
	            	  | main.(*clusterImpl).PutE
	            	  | 	main/pkg/cmd/roachtest/cluster.go:1836
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runConnectionLatencyTest
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/connection_latency.go:37
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerConnectionLatencyTest.func3
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/connection_latency.go:144
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:1205
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (4) put "/go/src/github.com/cockroachdb/cockroach/bin/workload.linux-amd64" failed
	            	Wraps: (5) TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/workload.linux-amd64 ubuntu@34.89.59.223:./workload
	            	  | (1) TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/workload.linux-amd64 ubuntu@34.89.59.223:./workload
	            	  |   | ssh: connect to host 34.89.59.223 port 22: Connection timed out
	            	  |   | lost connection: exit status 1
	            	  | Error types: (1) errors.TransientError
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) errors.TransientError
	Test:       	connection_latency/nodes=9/multiregion/password
(require.go:1360).NoError: FailNow called
test artifacts and logs in: /artifacts/connection_latency/nodes=9/multiregion/password/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 4665969fb6f4a4c1b58f10d9f99799f1939a0d62:

test connection_latency/nodes=9/multiregion/certs failed: VMs preempted during the test run: teamcity-15175669-1715233897-60-n12cpu4-geo-0009 (europe-west2-b)

**Other Failures:**
(assertions.go:333).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/connection_latency.go:83
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/connection_latency.go:95
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/connection_latency.go:132
	            				main/pkg/cmd/roachtest/test_runner.go:1205
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	full command output in run_115107.671577696_n12_workload-run-connect.log: COMMAND_PROBLEM: exit status 1
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*clusterImpl).RunE
	            	  | 	main/pkg/cmd/roachtest/cluster.go:2386
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runConnectionLatencyTest.func2
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/connection_latency.go:82
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runConnectionLatencyTest
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/connection_latency.go:95
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerConnectionLatencyTest.func2
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/connection_latency.go:132
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:1205
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) full command output in run_115107.671577696_n12_workload-run-connect.log
	            	Wraps: (3) Node 12. Command with error:
	            	  | ``````
	            	  | ./workload run connectionlatency 'postgres://roachprod@34.39.8.174:{pgport:1}?sslcert=certs/client.roachprod.crt&sslkey=certs/client.roachprod.key&sslrootcert=certs/ca.crt&sslmode=require' 'postgres://roachprod@34.142.84.111:{pgport:1}?sslcert=certs/client.roachprod.crt&sslkey=certs/client.roachprod.key&sslrootcert=certs/ca.crt&sslmode=require' 'postgres://roachprod@34.105.148.10:{pgport:1}?sslcert=certs/client.roachprod.crt&sslkey=certs/client.roachprod.key&sslrootcert=certs/ca.crt&sslmode=require' --secure --duration 30s --histograms=perf/stats.json --locality europe-west2-b
	            	  | ``````
	            	  | <truncated> ...       416.9          410.5      6.6      8.4      9.4     10.0 connect-from-europe-west2-b-to-europe-west2-b
	            	  |    21.0s        0          415.9          410.4      7.1      9.4     10.5     11.0 select
	            	  |    22.0s        0          414.4          410.6      6.6      8.1      9.4     10.5 connect-from-europe-west2-b-to-europe-west2-b
	            	  |    22.0s        0          415.4          410.6      7.3      8.9     10.5     11.0 select
	            	  |    23.0s        0          412.1          410.7      6.8      8.1      8.9      9.4 connect-from-europe-west2-b-to-europe-west2-b
	            	  |    23.0s        0          411.1          410.7      7.3      8.9     10.0     10.5 select
	            	  |
	            	  | stderr:I240509 11:51:09.155427 1 workload/cli/run.go:431  [-] 1  creating load generator...
	            	  | I240509 11:51:14.803233 1 workload/cli/run.go:470  [-] 2  creating load generator... done (took 5.647804626s)
	            	  | Error: failed to connect to ``host=34.39.8.174 user=roachprod database=connectionlatency``: server error (ERROR: server is not accepting clients, try another node (SQLSTATE 57P01))
	            	Wraps: (4) COMMAND_PROBLEM
	            	Wraps: (5) exit status 1
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *hintdetail.withDetail (4) errors.Cmd (5) *exec.ExitError
	Test:       	connection_latency/nodes=9/multiregion/certs
(require.go:1360).NoError: FailNow called
test artifacts and logs in: /artifacts/connection_latency/nodes=9/multiregion/certs/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 4665969fb6f4a4c1b58f10d9f99799f1939a0d62:

test failover/chaos/read-write/lease=expiration failed: VMs preempted during the test run: teamcity-15175669-1715233897-87-n10cpu2-0008 (us-east1-c), teamcity-15175669-1715233897-87-n10cpu2-0001 (us-east1-c), teamcity-15175669-1715233897-87-n10cpu2-0007 (us-east1-c)

**Other Failures:**
(failover.go:1766).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2349).Run: context canceled
(cluster.go:2349).Run: context canceled
(cluster.go:2349).Run: context canceled
(cluster.go:2349).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/lease=expiration/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 4665969fb6f4a4c1b58f10d9f99799f1939a0d62:

test kv0/enc=false/nodes=1/cpu=32 failed: VMs preempted during the test run: teamcity-15175669-1715233897-132-n2cpu32-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2349).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/cpu=32/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 4665969fb6f4a4c1b58f10d9f99799f1939a0d62:

test kv0/enc=false/nodes=1/cpu=32/mt-shared-process failed: VMs preempted during the test run: teamcity-15175669-1715233897-134-n2cpu32-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2349).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/cpu=32/mt-shared-process/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 4665969fb6f4a4c1b58f10d9f99799f1939a0d62:

test tpcc/headroom/isolation-level=read-committed/n4cpu16 failed: VMs preempted during the test run: teamcity-15175669-1715233897-163-n4cpu16-0002 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: failed to list cockroach processes: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/run_1/ssh/ssh_191921.969819602_n2_list-processes.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]
test artifacts and logs in: /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 4665969fb6f4a4c1b58f10d9f99799f1939a0d62:

test restore/tpce/400GB/gce/nodes=4/cpus=8 failed: VMs preempted during the test run: teamcity-15175669-1715233897-158-n4cpu8-0003 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/restore/tpce/400GB/gce/nodes=4/cpus=8/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 4665969fb6f4a4c1b58f10d9f99799f1939a0d62:

test ycsb/C/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-15175669-1715233897-172-n4cpu32-0004 (us-east1-b), teamcity-15175669-1715233897-172-n4cpu32-0002 (us-east1-b)

**Other Failures:**
(cluster.go:2349).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/C/nodes=3/cpu=32/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 4665969fb6f4a4c1b58f10d9f99799f1939a0d62:

test tpccbench/nodes=9/cpu=4/multi-region failed: VMs preempted during the test run: teamcity-15175669-1715233897-169-n12cpu4hm-geo-0003 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 7c483c276934758d4bf736b40594d4541377d526:

test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15194573-1715320161-56-n6cpu16-0005 (us-east1-c), teamcity-15194573-1715320161-56-n6cpu16-0003 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 7c483c276934758d4bf736b40594d4541377d526:

test decommissionBench/nodes=4/warehouses=1000/duration=1h0m0s failed: VMs preempted during the test run: teamcity-15194573-1715320161-76-n5cpu16-0001 (us-east1-c)

**Other Failures:**
(decommissionbench.go:847).runDecommissionBenchLong: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/decommissionBench/nodes=4/warehouses=1000/duration=1h0m0s/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 7c483c276934758d4bf736b40594d4541377d526:

test decommissionBench/nodes=4/warehouses=1000/while-down failed: VMs preempted during the test run: teamcity-15194573-1715320161-77-n5cpu16-0002 (us-east1-d), teamcity-15194573-1715320161-77-n5cpu16-0003 (us-east1-d), teamcity-15194573-1715320161-77-n5cpu16-0004 (us-east1-d)

**Other Failures:**
(test_runner.go:1229).runTest: test timed out (2h0m0s)
test artifacts and logs in: /artifacts/decommissionBench/nodes=4/warehouses=1000/while-down/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 7c483c276934758d4bf736b40594d4541377d526:

test decommissionBench/nodes=8/warehouses=3000 failed: VMs preempted during the test run: teamcity-15194573-1715320161-96-n9cpu16-0005 (us-east1-b), teamcity-15194573-1715320161-96-n9cpu16-0002 (us-east1-b), teamcity-15194573-1715320161-96-n9cpu16-0007 (us-east1-b)

**Other Failures:**
(decommissionbench.go:721).runDecommissionBench: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/decommissionBench/nodes=8/warehouses=3000/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 7c483c276934758d4bf736b40594d4541377d526:

test kv95/enc=false/nodes=3 failed: VMs preempted during the test run: teamcity-15194573-1715320161-145-n4cpu8-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2349).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 26d7bea76ca80f98b8aa20ac25266256735e366f:

test tpccbench/nodes=9/cpu=4/multi-region failed: VMs preempted during the test run: teamcity-15213536-1715406601-164-n12cpu4hm-geo-0011 (europe-west2-b), teamcity-15213536-1715406601-164-n12cpu4hm-geo-0006 (us-west1-b), teamcity-15213536-1715406601-164-n12cpu4hm-geo-0005 (us-west1-b), teamcity-15213536-1715406601-164-n12cpu4hm-geo-0007 (us-west1-b), teamcity-15213536-1715406601-164-n12cpu4hm-geo-0008 (us-west1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 26d7bea76ca80f98b8aa20ac25266256735e366f:

test decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region failed: VMs preempted during the test run: teamcity-15235367-1715579338-78-n8cpu16-geo-0003 (us-west1-b)

**Other Failures:**
(decommissionbench.go:721).runDecommissionBench: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 26d7bea76ca80f98b8aa20ac25266256735e366f:

test restore/tpce/400GB/gce/nodes=4/cpus=8 failed: VMs preempted during the test run: teamcity-15235367-1715579338-147-n4cpu8-0004 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/restore/tpce/400GB/gce/nodes=4/cpus=8/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 26d7bea76ca80f98b8aa20ac25266256735e366f:

test ycsb/C/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-15235367-1715579338-166-n4cpu32-0002 (us-east1-b), teamcity-15235367-1715579338-166-n4cpu32-0001 (us-east1-b), teamcity-15235367-1715579338-166-n4cpu32-0003 (us-east1-b), teamcity-15235367-1715579338-166-n4cpu32-0004 (us-east1-b)

**Other Failures:**
(cluster.go:2349).Run: full command output in run_194304.712664015_n4_workload-run-ycsb-in.log: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/C/nodes=3/cpu=32/run_1/ssh/ssh_194304.726739550_n4_workload-run-ycsb-in.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/C/nodes=3/cpu=32/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 26d7bea76ca80f98b8aa20ac25266256735e366f:

test tpccbench/nodes=6/cpu=16/multi-az failed: VMs preempted during the test run: teamcity-15235367-1715579338-164-n7cpu16-geo-0005 (us-central1-c)

**Other Failures:**
(cluster.go:2286).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1/ssh/ssh_210604.144736447_n5_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 9575819131ada5aa4e23870d854beef5caaed8b0:

test allocbench/nodes=7/cpu=8/kv/r=0/ops=skew failed: VMs preempted during the test run: teamcity-15254661-1715665774-21-n8cpu8-0006 (us-east1-d), teamcity-15254661-1715665774-21-n8cpu8-0007 (us-east1-d), teamcity-15254661-1715665774-21-n8cpu8-0003 (us-east1-d), teamcity-15254661-1715665774-21-n8cpu8-0005 (us-east1-d), teamcity-15254661-1715665774-21-n8cpu8-0002 (us-east1-d), teamcity-15254661-1715665774-21-n8cpu8-0008 (us-east1-d), teamcity-15254661-1715665774-21-n8cpu8-0001 (us-east1-d), teamcity-15254661-1715665774-21-n8cpu8-0004 (us-east1-d)

**Other Failures:**
(assertions.go:333).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:396
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
	            				main/pkg/cmd/roachtest/test_runner.go:1205
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	monitor failure: EOF [owner=test-eng]
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:142
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:395
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:1205
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) monitor failure
	            	Wraps: (3) secondary error attachment
	            	  | context canceled
	            	  | (1) context canceled
	            	  | Error types: (1) *errors.errorString
	            	Wraps: (4) EOF [owner=test-eng]
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) registry.ErrorWithOwnership
	Test:       	allocbench/nodes=7/cpu=8/kv/r=0/ops=skew
(require.go:1360).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/ops=skew/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 9575819131ada5aa4e23870d854beef5caaed8b0:

test allocbench/nodes=7/cpu=8/kv/r=50/ops=skew failed: VMs preempted during the test run: teamcity-15254661-1715665774-22-n8cpu8-0002 (us-east1-b), teamcity-15254661-1715665774-22-n8cpu8-0001 (us-east1-b), teamcity-15254661-1715665774-22-n8cpu8-0008 (us-east1-b)

**Other Failures:**
(assertions.go:333).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:396
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
	            				main/pkg/cmd/roachtest/test_runner.go:1205
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	monitor failure: EOF [owner=test-eng]
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:142
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:395
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:1205
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) monitor failure
	            	Wraps: (3) secondary error attachment
	            	  | context canceled
	            	  | (1) context canceled
	            	  | Error types: (1) *errors.errorString
	            	Wraps: (4) EOF [owner=test-eng]
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) registry.ErrorWithOwnership
	Test:       	allocbench/nodes=7/cpu=8/kv/r=50/ops=skew
(require.go:1360).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 9575819131ada5aa4e23870d854beef5caaed8b0:

test failover/chaos/read-write/lease=expiration failed: VMs preempted during the test run: teamcity-15254661-1715665774-81-n10cpu2-0009 (us-east1-c), teamcity-15254661-1715665774-81-n10cpu2-0008 (us-east1-c), teamcity-15254661-1715665774-81-n10cpu2-0006 (us-east1-c), teamcity-15254661-1715665774-81-n10cpu2-0007 (us-east1-c)

**Other Failures:**
(failover.go:1769).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2349).Run: context canceled
(cluster.go:2349).Run: context canceled
(cluster.go:2349).Run: context canceled
(cluster.go:2349).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/lease=expiration/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 9575819131ada5aa4e23870d854beef5caaed8b0:

test failover/partial/lease-gateway/lease=expiration failed: VMs preempted during the test run: teamcity-15254661-1715665774-97-n8cpu2-0005 (us-east1-d), teamcity-15254661-1715665774-97-n8cpu2-0006 (us-east1-d)

**Other Failures:**
(cluster.go:2235).Start: failed to find services to register: failed to find 1 open ports: TRANSIENT_ERROR(open_ports): expected 1 ports, got 0
(cluster.go:2349).Run: context canceled
test artifacts and logs in: /artifacts/failover/partial/lease-gateway/lease=expiration/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 9575819131ada5aa4e23870d854beef5caaed8b0:

test indexes/2/nodes=6/multi-region failed: VMs preempted during the test run: teamcity-15254661-1715665774-122-n7cpu16-geo-0003 (us-west1-b), teamcity-15254661-1715665774-122-n7cpu16-geo-0004 (us-west1-b)

**Other Failures:**
(cluster.go:2349).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/indexes/2/nodes=6/multi-region/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.vm_preemption failed with artifacts on release-24.1 @ 9575819131ada5aa4e23870d854beef5caaed8b0:

test kv95/enc=false/nodes=3/size=64kb failed: VMs preempted during the test run: teamcity-15254661-1715665774-134-n4cpu8-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2235).Start: failed to find services to register: failed to find 1 open ports: TRANSIENT_ERROR(open_ports): expected 1 ports, got 0
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/size=64kb/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-24.1 Used to mark GA and release blockers and technical advisories for 24.1 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Projects
Test Engineering
Triage
Milestone
24.1
Development

No branches or pull requests
1 participant
@cockroach-teamcity