roachtest: cluster_creation failed

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-25.2.1-rc @ 592da43f5cc80f32a8277aa90e09c7e3362ff53c:

test ycsb/E/nodes=3 failed: (test_runner.go:864).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-9Q-fXq1rViXE9fHkJpmkq_nMrKY= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-19780279-1748237951-89-n4cpu8},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19780279-1748237951-89-n4cpu8},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-05-26T07:06:22Z},{Key=Name,Value=teamcity-19780279-1748237951-89-n4cpu8-0003}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-19780279-1748237951-89-n4cpu8},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19780279-1748237951-89-n4cpu8},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-05-26T07:06:22Z},{Key=Name,Value=teamcity-19780279-1748237951-89-n4cpu8-0003}] --user-data file:///tmp/aws-startup-script3446377817 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3656360229 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

• cloud=aws
• coverageBuild=false
• cpu=8
• runtimeAssertionsBuild=false
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

• roachtest: cluster_creation failed #147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
• roachtest: cluster_creation failed #147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
• roachtest: cluster_creation failed #147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
• roachtest: cluster_creation failed #147205 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
• roachtest: cluster_creation failed #147204 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
• roachtest: cluster_creation failed #147203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
• roachtest: cluster_creation failed #147202 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
• roachtest: cluster_creation failed #147201 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
• roachtest: cluster_creation failed #147200 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
• roachtest: cluster_creation failed #147194 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-25.2.1-rc @ 592da43f5cc80f32a8277aa90e09c7e3362ff53c:

test ycsb/F/nodes=3 failed: (test_runner.go:864).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-9Q-fXq1rViXE9fHkJpmkq_nMrKY= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-05-26T07:08:21Z},{Key=Name,Value=teamcity-19780279-1748237951-92-n4cpu8-0001},{Key=Cluster,Value=teamcity-19780279-1748237951-92-n4cpu8},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19780279-1748237951-92-n4cpu8}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-05-26T07:08:21Z},{Key=Name,Value=teamcity-19780279-1748237951-92-n4cpu8-0001},{Key=Cluster,Value=teamcity-19780279-1748237951-92-n4cpu8},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19780279-1748237951-92-n4cpu8}] --user-data file:///tmp/aws-startup-script1398159708 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2119166429 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

• cloud=aws
• coverageBuild=false
• cpu=8
• runtimeAssertionsBuild=false
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-25.2.1-rc @ 592da43f5cc80f32a8277aa90e09c7e3362ff53c:

test ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:864).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-9Q-fXq1rViXE9fHkJpmkq_nMrKY= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-19780279-1748237951-98-n4cpu32-0002},{Key=Cluster,Value=teamcity-19780279-1748237951-98-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19780279-1748237951-98-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-05-26T07:13:19Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-19780279-1748237951-98-n4cpu32-0002},{Key=Cluster,Value=teamcity-19780279-1748237951-98-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19780279-1748237951-98-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-05-26T07:13:19Z}] --user-data file:///tmp/aws-startup-script1798987732 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3033552005 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c7gd.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c7gd.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

• cloud=aws
• coverageBuild=false
• cpu=32
• runtimeAssertionsBuild=false
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

• roachtest: cluster_creation failed #147279 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-master]
• roachtest: cluster_creation failed #147278 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-master]
• roachtest: cluster_creation failed #147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
• roachtest: cluster_creation failed #147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
• roachtest: cluster_creation failed #147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
• roachtest: cluster_creation failed #147205 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
• roachtest: cluster_creation failed #147204 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
• roachtest: cluster_creation failed #147203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
• roachtest: cluster_creation failed #147202 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
• roachtest: cluster_creation failed #147201 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!