Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: cluster_creation failed #108629

Closed
cockroach-teamcity opened this issue Aug 11, 2023 · 376 comments
Closed

roachtest: cluster_creation failed #108629

cockroach-teamcity opened this issue Aug 11, 2023 · 376 comments
Labels
branch-master Failures on the master branch. O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Projects
Test Engineering
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 11, 2023 •
edited by cockroach-jira-scripts
Loading

roachtest.cluster_creation failed with artifacts on master @ 6fa90e9e1003b0e0355c1ff42eb34b1b64ab36f2:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,cluster=teamcity-11277539-1691732917-150-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-08-11t19_36_24z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1905306309 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-11277539-1691732917-150-n9cpu8-geo-0007 teamcity-11277539-1691732917-150-n9cpu8-geo-0008 teamcity-11277539-1691732917-150-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/dev-inf

This test on roachdash | Improve this report!

Jira issue: CRDB-30556
@cockroach-teamcity cockroach-teamcity added branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-dev-inf labels Aug 11, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Aug 11, 2023
@srosenberg srosenberg mentioned this issue Aug 12, 2023
Closed
@srosenberg srosenberg added the T-testeng TestEng Team label Aug 12, 2023
@blathers-crl
Copy link

blathers-crl bot commented Aug 12, 2023

cc @cockroachdb/test-eng

@blathers-crl blathers-crl bot added this to Triage in Test Engineering Aug 12, 2023
@srosenberg srosenberg mentioned this issue Aug 12, 2023
Closed
This was referenced Aug 12, 2023
Open
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 7da4ef5af83c38680c536e7c18a9db61274b9d25:

test restore/tpce/8TB/aws/nodes=10/cpus=8 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6i.2xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-EjL4_iQ8G6Syt-D9lzdsnURCehA= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-004c6ad7121a8d5a7 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11290567-1691819425-31-n10cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-12T08:06:01Z},{Key=Name,Value=teamcity-11290567-1691819425-31-n10cpu8-0002}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11290567-1691819425-31-n10cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-12T08:06:01Z},{Key=Name,Value=teamcity-11290567-1691819425-31-n10cpu8-0002}] --user-data file:///tmp/aws-startup-script3645116444 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping283082095 --output json: stderr: 
Parameter validation failed:
Unknown parameter in BlockDeviceMappings[0].Ebs: "Throughput", must be one of: DeleteOnTermination, Iops, SnapshotId, VolumeSize, VolumeType, KmsKeyId, Encrypted: exit status 252

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@pav-kv
Copy link
Collaborator

pav-kv commented Aug 12, 2023 •
edited
Loading

The latest one seems to be related to #108427, which started populating the "Throughput" field. The field existed before the PR though, it was just not being populated.

While working on the PR, I ran this roachtest manually a few times, and it worked. I also ran this just now on master, and it works. So this failure perhaps has to do with other code that's not directly related to running the test.

The BlockDeviceMappings type listed in the error is here:

cockroach/pkg/roachprod/vm/aws/aws.go

Lines 845 to 850 in a2ba442

Disk struct {
AttachTime time.Time `json:"AttachTime"`
DeleteOnTermination bool `json:"DeleteOnTermination"`
Status string `json:"Status"`
VolumeID string `json:"VolumeId"`
} `json:"Ebs"`

This func looks like a listing function that is not used when running the test directly. And indeed it does not have the "Throughput" field, so I think we should add it.

@pav-kv
Copy link
Collaborator

pav-kv commented Aug 12, 2023 •
edited
Loading

I am puzzled by the list of field names in "must be one of":

Unknown parameter in BlockDeviceMappings[0].Ebs: "Throughput", must be one of: DeleteOnTermination, Iops, SnapshotId, VolumeSize, VolumeType, KmsKeyId, Encrypted

This does not match the BlockDeviceMappings struct:

cockroach/pkg/roachprod/vm/aws/aws.go

Lines 845 to 850 in a2ba442

Disk struct {
AttachTime time.Time `json:"AttachTime"`
DeleteOnTermination bool `json:"DeleteOnTermination"`
Status string `json:"Status"`
VolumeID string `json:"VolumeId"`
} `json:"Ebs"`

@cockroachdb/test-eng Are you familiar with this code to locate the struct that misses this field declaration?

@pav-kv
Copy link
Collaborator

pav-kv commented Aug 12, 2023 •
edited
Loading

Ok, the required JSON structure seems to be this one, which is defined by AWS APIs. So likely what happens is that the Throughput field slips through into one of these JSONs.

Is this a bug or a feature though that this JSON spec misses the Throughput field? In contrast, it does have the Iops field which plays a similar role to Throughput and I would expect them to come in pair.

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 7da4ef5af83c38680c536e7c18a9db61274b9d25:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-11290566-1691819282-146-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-08-12t19_28_04z --metadata-from-file startup-script=/tmp/gce-startup-script1311714136 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-11290566-1691819282-146-n9cpu8-geo-0004 teamcity-11290566-1691819282-146-n9cpu8-geo-0005 teamcity-11290566-1691819282-146-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 6b0f43a13c4d2b36b4da5961e523161dabe212c0:

test restore/tpce/8TB/aws/nodes=10/cpus=8 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6i.2xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-E81Qt6_eRxYFOHJBdg5I8pCRUmA= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-004c6ad7121a8d5a7 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11300089-1691905649-31-n10cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-13T08:02:20Z},{Key=Name,Value=teamcity-11300089-1691905649-31-n10cpu8-0002}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11300089-1691905649-31-n10cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-13T08:02:20Z},{Key=Name,Value=teamcity-11300089-1691905649-31-n10cpu8-0002}] --user-data file:///tmp/aws-startup-script214626434 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping633368368 --output json: stderr: 
Parameter validation failed:
Unknown parameter in BlockDeviceMappings[0].Ebs: "Throughput", must be one of: DeleteOnTermination, Iops, SnapshotId, VolumeSize, VolumeType, KmsKeyId, Encrypted: exit status 252

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 6b0f43a13c4d2b36b4da5961e523161dabe212c0:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2023-08-13t19_41_09z,roachprod=true,cluster=teamcity-11300088-1691905772-154-n9cpu8-geo --metadata-from-file startup-script=/tmp/gce-startup-script4104685822 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-11300088-1691905772-154-n9cpu8-geo-0007 teamcity-11300088-1691905772-154-n9cpu8-geo-0008 teamcity-11300088-1691905772-154-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ c13bf7633cbb416d9e43f8c57b1e309fab1110ce:

test restore/tpce/8TB/aws/nodes=10/cpus=8 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6i.2xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-YtjS7-RDYbZ_QIQ3Y_h3O3Su_a0= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-004c6ad7121a8d5a7 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11308686-1691992207-31-n10cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-14T08:05:37Z},{Key=Name,Value=teamcity-11308686-1691992207-31-n10cpu8-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11308686-1691992207-31-n10cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-14T08:05:37Z},{Key=Name,Value=teamcity-11308686-1691992207-31-n10cpu8-0001}] --user-data file:///tmp/aws-startup-script1303095529 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1868857832 --output json: stderr: 
Parameter validation failed:
Unknown parameter in BlockDeviceMappings[0].Ebs: "Throughput", must be one of: DeleteOnTermination, Iops, SnapshotId, VolumeSize, VolumeType, KmsKeyId, Encrypted: exit status 252

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ c13bf7633cbb416d9e43f8c57b1e309fab1110ce:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-11308685-1691992291-163-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-08-14t19_41_25z --metadata-from-file startup-script=/tmp/gce-startup-script231679367 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-11308685-1691992291-163-n9cpu8-geo-0007 teamcity-11308685-1691992291-163-n9cpu8-geo-0008 teamcity-11308685-1691992291-163-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@renatolabs renatolabs added C-test-failure Broken test (automatically or manually discovered). and removed C-test-failure Broken test (automatically or manually discovered). labels Aug 14, 2023
renatolabs added a commit to renatolabs/cockroach that referenced this issue Aug 14, 2023
@renatolabs
build: update aws-cli
306d57c 
The version being previously used was quite old, and didn't support
setting the `Throughput` parameter on `gp3` volumes.

See: cockroachdb#108629 (comment).

Epic: none

Release note: None
@renatolabs renatolabs mentioned this issue Aug 14, 2023
Merged
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ abf61bbf38a990f938b56a80284a834e2d087ea1:

test restore/tpce/8TB/aws/nodes=10/cpus=8 was skipped due to (test_runner.go:745).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6i.2xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-IOq8IGjQFtpybUCGzrebmMmvQeo= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-004c6ad7121a8d5a7 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-15T08:02:34Z},{Key=Name,Value=teamcity-11324205-1692078582-31-n10cpu8-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11324205-1692078582-31-n10cpu8},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-15T08:02:34Z},{Key=Name,Value=teamcity-11324205-1692078582-31-n10cpu8-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11324205-1692078582-31-n10cpu8},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script943596904 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping822709940 --output json: stderr: 
Parameter validation failed:
Unknown parameter in BlockDeviceMappings[0].Ebs: "Throughput", must be one of: DeleteOnTermination, Iops, SnapshotId, VolumeSize, VolumeType, KmsKeyId, Encrypted: exit status 252

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

renatolabs added a commit to renatolabs/cockroach that referenced this issue Aug 15, 2023
@renatolabs
build: update aws-cli
e2065a2 
The version being previously used was quite old, and didn't support
setting the `Throughput` parameter on `gp3` volumes.

See: cockroachdb#108629 (comment).

Epic: none

Release note: None
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 14733120c08ba1a0a050bb5442c3311090206566:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-d, us-east1-c
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-d,us-east1-c
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-d, us-east1-c
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-d,us-east1-c
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 14733120c08ba1a0a050bb5442c3311090206566:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-d, us-east1-c
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-d,us-east1-c
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-d, us-east1-c
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-d,us-east1-c
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 14733120c08ba1a0a050bb5442c3311090206566:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-d, us-east1-c
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-d,us-east1-c
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-d, us-east1-c
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-d,us-east1-c
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 64dacab2808c5420e3fd99d2562e8ecb97b7742b:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:729).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12772114-1700548298-152-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-11-21t20_44_38z --metadata-from-file startup-script=/tmp/gce-startup-script4051103077 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12772114-1700548298-152-n9cpu8-geo-0001 teamcity-12772114-1700548298-152-n9cpu8-geo-0002 teamcity-12772114-1700548298-152-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 23a7242ece434c4dcec48953010621fa6bdb7824:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:729).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12807010-1700721009-157-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-11-23t20_49_56z --metadata-from-file startup-script=/tmp/gce-startup-script3386292036 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-12807010-1700721009-157-n9cpu8-geo-0007 teamcity-12807010-1700721009-157-n9cpu8-geo-0008 teamcity-12807010-1700721009-157-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 5774ee0f6bf004d3fd22a08dcb1803a3026205c0:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:729).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,cluster=teamcity-12821147-1700807431-156-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-11-24t20_55_08z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script3517603997 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-12821147-1700807431-156-n9cpu8-geo-0004 teamcity-12821147-1700807431-156-n9cpu8-geo-0005 teamcity-12821147-1700807431-156-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ caff15394fcbe37208b46b2973714c27cc3a1417:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:729).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12832189-1700893839-143-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-11-25t20_53_00z --metadata-from-file startup-script=/tmp/gce-startup-script3651125896 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12832189-1700893839-143-n9cpu8-geo-0001 teamcity-12832189-1700893839-143-n9cpu8-geo-0002 teamcity-12832189-1700893839-143-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 01e65172dcb17384db33e8229d16461f6f99c01d:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:729).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12842327-1700980219-145-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-11-26t20_32_33z --metadata-from-file startup-script=/tmp/gce-startup-script2888167296 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-12842327-1700980219-145-n9cpu8-geo-0004 teamcity-12842327-1700980219-145-n9cpu8-geo-0005 teamcity-12842327-1700980219-145-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 0b82e156811000e4a0f8147a1e6325352417285a:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:729).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2023-11-27t20_21_20z,roachprod=true,cluster=teamcity-12852207-1701066653-146-n9cpu8-geo --metadata-from-file startup-script=/tmp/gce-startup-script3232180800 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12852207-1701066653-146-n9cpu8-geo-0001 teamcity-12852207-1701066653-146-n9cpu8-geo-0002 teamcity-12852207-1701066653-146-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ ba6423f916dc4006bb590c786cf53ad2b0213417:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:729).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12867503-1701153001-153-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-11-28t21_09_09z --metadata-from-file startup-script=/tmp/gce-startup-script1533638492 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-12867503-1701153001-153-n9cpu8-geo-0007 teamcity-12867503-1701153001-153-n9cpu8-geo-0008 teamcity-12867503-1701153001-153-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ d5765263b1d2d65475bdcdb661d595197ef5faef:

test kv0/enc=false/nodes=3/cpu=32/mt-shared-process was skipped due to (test_runner.go:729).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-SKb_yAEvwlTpW2PnOGqP-Ieu81c= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e71591c1fe06645e --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-11-29T06:43:08Z},{Key=Name,Value=teamcity-12889591-1701239458-13-n4cpu32-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12889591-1701239458-13-n4cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-11-29T06:43:08Z},{Key=Name,Value=teamcity-12889591-1701239458-13-n4cpu32-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12889591-1701239458-13-n4cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script2896133199 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping704179138 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ d5765263b1d2d65475bdcdb661d595197ef5faef:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:729).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-SKb_yAEvwlTpW2PnOGqP-Ieu81c= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12889591-1701239458-39-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-11-29T10:05:39Z},{Key=Name,Value=teamcity-12889591-1701239458-39-n4cpu96-0002}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12889591-1701239458-39-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-11-29T10:05:39Z},{Key=Name,Value=teamcity-12889591-1701239458-39-n4cpu96-0002}] --user-data file:///tmp/aws-startup-script3536752237 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1530407609 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ d5765263b1d2d65475bdcdb661d595197ef5faef:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:729).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-SKb_yAEvwlTpW2PnOGqP-Ieu81c= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-11-29T10:07:22Z},{Key=Name,Value=teamcity-12889591-1701239458-42-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12889591-1701239458-42-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-11-29T10:07:22Z},{Key=Name,Value=teamcity-12889591-1701239458-42-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12889591-1701239458-42-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] --user-data file:///tmp/aws-startup-script1421770365 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping662053296 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ d5765263b1d2d65475bdcdb661d595197ef5faef:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:729).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,arch=amd64,created=2023-11-29t20_36_27z,roachprod=true,cluster=teamcity-12889590-1701239410-147-n9cpu8-geo,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script2501109481 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12889590-1701239410-147-n9cpu8-geo-0001 teamcity-12889590-1701239410-147-n9cpu8-geo-0002 teamcity-12889590-1701239410-147-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ ccc3d549aa7eacb6eff1ef8137a82a69acab291c:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:729).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2023-11-30t20_37_37z,roachprod=true,cluster=teamcity-12908540-1701325816-147-n9cpu8-geo --metadata-from-file startup-script=/tmp/gce-startup-script2720750825 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-12908540-1701325816-147-n9cpu8-geo-0007 teamcity-12908540-1701325816-147-n9cpu8-geo-0008 teamcity-12908540-1701325816-147-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 02e46d54d0b0bf63f43592709d551534edb54be6:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:729).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-EfwAdQl8LFZ6S38PwxKyptFK1UM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-12925675-1701412217-36-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12925675-1701412217-36-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-12-01T10:27:27Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-12925675-1701412217-36-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12925675-1701412217-36-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-12-01T10:27:27Z}] --user-data file:///tmp/aws-startup-script1757404170 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1337540745 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 02e46d54d0b0bf63f43592709d551534edb54be6:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:729).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-EfwAdQl8LFZ6S38PwxKyptFK1UM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-12-01T10:29:09Z},{Key=Name,Value=teamcity-12925675-1701412217-39-n4cpu96-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12925675-1701412217-39-n4cpu96},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-12-01T10:29:09Z},{Key=Name,Value=teamcity-12925675-1701412217-39-n4cpu96-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12925675-1701412217-39-n4cpu96},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script2778563605 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1248080854 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 02e46d54d0b0bf63f43592709d551534edb54be6:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:729).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,created=2023-12-01t20_38_10z,roachprod=true,cluster=teamcity-12925674-1701412164-147-n9cpu8-geo,lifetime=12h0m0s,arch=amd64 --metadata-from-file startup-script=/tmp/gce-startup-script4135969571 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12925674-1701412164-147-n9cpu8-geo-0001 teamcity-12925674-1701412164-147-n9cpu8-geo-0002 teamcity-12925674-1701412164-147-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 74ae9a18d82fdc80e5e9b71b3f06b258ae5cb91b:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:729).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12942600-1701498584-145-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-12-02t20_40_30z --metadata-from-file startup-script=/tmp/gce-startup-script3223784832 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-12942600-1701498584-145-n9cpu8-geo-0004 teamcity-12942600-1701498584-145-n9cpu8-geo-0005 teamcity-12942600-1701498584-145-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 74ae9a18d82fdc80e5e9b71b3f06b258ae5cb91b:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:729).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,created=2023-12-03t20_43_31z,roachprod=true,cluster=teamcity-12952568-1701585009-155-n9cpu8-geo,lifetime=12h0m0s,arch=amd64 --metadata-from-file startup-script=/tmp/gce-startup-script3646528212 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-12952568-1701585009-155-n9cpu8-geo-0004 teamcity-12952568-1701585009-155-n9cpu8-geo-0005 teamcity-12952568-1701585009-155-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 8b246d0009d40d7ecad14df14bc2fb61b31f4be5:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:729).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,cluster=teamcity-12963570-1701671368-147-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-12-04t21_06_48z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script2982714235 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-12963570-1701671368-147-n9cpu8-geo-0004 teamcity-12963570-1701671368-147-n9cpu8-geo-0005 teamcity-12963570-1701671368-147-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 26c81115359b96b85319b0b545ed01467e71d493:

test failover/non-system/disk-stall/lease=expiration was skipped due to (test_runner.go:729).func4: cluster.PutE: put /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 failed: ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@34.74.163.165:./cockroach
ssh: connect to host 34.74.163.165 port 22: Connection timed out
lost connection: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=2 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 26c81115359b96b85319b0b545ed01467e71d493:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:729).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2023-12-05t20_54_27z,roachprod=true,cluster=teamcity-12979495-1701757919-154-n9cpu8-geo --metadata-from-file startup-script=/tmp/gce-startup-script1356248502 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-12979495-1701757919-154-n9cpu8-geo-0004 teamcity-12979495-1701757919-154-n9cpu8-geo-0005 teamcity-12979495-1701757919-154-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 7f38906fd05002cecd9188bbeb1e9da4cb4811f4:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:733).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12999656-1701844199-153-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-12-06t20_56_27z --metadata-from-file startup-script=/tmp/gce-startup-script2992448165 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-12999656-1701844199-153-n9cpu8-geo-0007 teamcity-12999656-1701844199-153-n9cpu8-geo-0008 teamcity-12999656-1701844199-153-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Dec 7, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ da982d571a4822164beeaa3c33e61f677c80a9fa:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:733).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-13016918-1701930631-162-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-12-07t21_28_04z --metadata-from-file startup-script=/tmp/gce-startup-script3016126254 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-13016918-1701930631-162-n9cpu8-geo-0007 teamcity-13016918-1701930631-162-n9cpu8-geo-0008 teamcity-13016918-1701930631-162-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 37ad01a3972cb4d34bfc6dfb4b9cfcac360b15dd:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:733).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,created=2023-12-08t21_27_01z,roachprod=true,cluster=teamcity-13038112-1702017106-150-n9cpu8-geo,lifetime=12h0m0s,arch=amd64 --metadata-from-file startup-script=/tmp/gce-startup-script3162474518 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-13038112-1702017106-150-n9cpu8-geo-0004 teamcity-13038112-1702017106-150-n9cpu8-geo-0005 teamcity-13038112-1702017106-150-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@renatolabs
Copy link
Collaborator

Closing as no new failures will be posted here (master is now part of the 24.1 milestone).

Test Engineering automation moved this from Triage to Done Dec 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures on the master branch. O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Projects
Test Engineering
Done
Milestone
23.2
Development

No branches or pull requests
4 participants
@srosenberg @pav-kv @cockroach-teamcity @renatolabs