Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SPLAT-657: ipi/aws-local-zones: increase zone coverage in opted-out zones #40606

Conversation

mtulio
Copy link
Contributor

@mtulio mtulio commented Jun 23, 2023

AWS is constantly adding new locations in Local Zones, we need to test it to prevent unwanted scenarios in features we are declaring supported. It will also not increase the cost of the test as it is selecting randomly one single zone.

openshift/enhancements#1232

Increasing the zone coverage by setting the flag --all-availability-zones to show zones which are not yet opted-in - the installer must opt-in when creating the cluster and that zone name is set in install-config.yaml:

  • us-east-1 in CI account without the flag --all-availability-zones
$ AWS_PROFILE=aws-local-zones aws ec2 describe-availability-zones  --region us-east-1  --filters Name=state,Values=available Name=zone-type,Values=local-zone --query 'AvailabilityZones[].ZoneName' | jq -r .[]
us-east-1-lim-1a
us-east-1-msp-1a
us-east-1-qro-1a
  • us-east-1 in CI account with the flag --all-availability-zones
$ AWS_PROFILE=aws-local-zones aws --region us-east-1 ec2 describe-availability-zones --all-availability-zones --filter Name=state,Values=available Name=zone-type,Values=local-zone | jq -r '.AvailabilityZones[].ZoneName' 
us-east-1-atl-1a
us-east-1-bos-1a
us-east-1-bue-1a
us-east-1-chi-1a
us-east-1-dfw-1a
us-east-1-iah-1a
us-east-1-lim-1a
us-east-1-mci-1a
us-east-1-mia-1a
us-east-1-msp-1a
us-east-1-nyc-1a
us-east-1-phl-1a
us-east-1-qro-1a
us-east-1-scl-1a
  • us-west-2 in CI account without the flag --all-availability-zones
$ AWS_PROFILE=aws-local-zones aws --region us-west-2 ec2 describe-availability-zones  --filter Name=state,Values=available Name=zone-type,Values=local-zone | jq -r '.AvailabilityZones[].ZoneName' 
us-west-2-las-1a
us-west-2-sea-1a
  • us-west-2 in CI account with the flag --all-availability-zones
$ AWS_PROFILE=aws-local-zones aws --region us-west-2 ec2 describe-availability-zones --all-availability-zones --filter Name=state,Values=available Name=zone-type,Values=local-zone | jq -r '.AvailabilityZones[].ZoneName' 
us-west-2-den-1a
us-west-2-las-1a
us-west-2-lax-1a
us-west-2-lax-1b
us-west-2-pdx-1a
us-west-2-phx-1a
us-west-2-sea-1a

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 23, 2023
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 23, 2023
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jun 23, 2023

@mtulio: This pull request references SPLAT-657 which is a valid jira issue.

In response to this:

Increasing the zone coverage by setting the flag --all-availability-zones to show zones which are not yet opted-in - the installer must opt-in when creating the cluster and that zone name is set in install-config.yaml:

  • us-east-1 in CI account without the flag --all-availability-zones
$ AWS_PROFILE=aws-local-zones aws ec2 describe-availability-zones  --region us-east-1  --filters Name=state,Values=available Name=zone-type,Values=local-zone --query 'AvailabilityZones[].ZoneName' | jq -r .[]
us-east-1-lim-1a
us-east-1-msp-1a
us-east-1-qro-1a
  • us-east-1 in CI account with the flag --all-availability-zones
$ AWS_PROFILE=aws-local-zones aws --region us-east-1 ec2 describe-availability-zones --all-availability-zones --filter Name=state,Values=available Name=zone-type,Values=local-zone | jq -r '.AvailabilityZones[].ZoneName' 
us-east-1-atl-1a
us-east-1-bos-1a
us-east-1-bue-1a
us-east-1-chi-1a
us-east-1-dfw-1a
us-east-1-iah-1a
us-east-1-lim-1a
us-east-1-mci-1a
us-east-1-mia-1a
us-east-1-msp-1a
us-east-1-nyc-1a
us-east-1-phl-1a
us-east-1-qro-1a
us-east-1-scl-1a
  • us-west-2 in CI account without the flag --all-availability-zones
$ AWS_PROFILE=aws-local-zones aws --region us-west-2 ec2 describe-availability-zones  --filter Name=state,Values=available Name=zone-type,Values=local-zone | jq -r '.AvailabilityZones[].ZoneName' 
us-west-2-las-1a
us-west-2-sea-1a
  • us-west-2 in CI account with the flag --all-availability-zones
$ AWS_PROFILE=aws-local-zones aws --region us-west-2 ec2 describe-availability-zones --all-availability-zones --filter Name=state,Values=available Name=zone-type,Values=local-zone | jq -r '.AvailabilityZones[].ZoneName' 
us-west-2-den-1a
us-west-2-las-1a
us-west-2-lax-1a
us-west-2-lax-1b
us-west-2-pdx-1a
us-west-2-phx-1a
us-west-2-sea-1a

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 23, 2023

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@mtulio mtulio force-pushed the splat-657-presubmits-local-zones-coverage branch from f46c5b8 to c5a9a76 Compare June 23, 2023 18:52
@mtulio mtulio marked this pull request as ready for review June 23, 2023 18:53
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 23, 2023
@openshift-ci-robot
Copy link
Contributor

[REHEARSALNOTIFIER]
@mtulio: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-build-test-images-master-e2e-aws openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-build-test-images-release-4.15-e2e-aws openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-build-test-images-release-4.14-e2e-aws openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-build-test-images-release-4.13-e2e-aws openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-build-test-images-release-4.12-e2e-aws openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-build-test-images-release-4.11-e2e-aws openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-build-test-images-release-4.10-e2e-aws openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-build-test-images-release-4.9-e2e-aws openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-build-test-images-release-4.8-e2e-aws openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-build-test-images-master-e2e-aws-builds openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-build-test-images-release-4.15-e2e-aws-builds openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-build-test-images-release-4.14-e2e-aws-builds openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-build-test-images-release-4.13-e2e-aws-builds openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-build-test-images-release-4.12-e2e-aws-builds openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-build-test-images-release-4.11-e2e-aws-builds openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-build-test-images-release-4.10-e2e-aws-builds openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-build-test-images-release-4.9-e2e-aws-builds openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-build-test-images-release-4.8-e2e-aws-builds openshift/build-test-images presubmit Registry content changed
pull-ci-openshift-jenkins-master-e2e-aws-jenkins-sync-plugin openshift/jenkins presubmit Registry content changed
pull-ci-openshift-jenkins-master-e2e-aws-jenkins-client-plugin openshift/jenkins presubmit Registry content changed
pull-ci-openshift-jenkins-release-4.15-e2e-aws-jenkins-sync-plugin openshift/jenkins presubmit Registry content changed
pull-ci-openshift-jenkins-release-4.15-e2e-aws-jenkins-client-plugin openshift/jenkins presubmit Registry content changed
pull-ci-openshift-jenkins-release-4.14-e2e-aws-jenkins-sync-plugin openshift/jenkins presubmit Registry content changed
pull-ci-openshift-jenkins-release-4.14-e2e-aws-jenkins-client-plugin openshift/jenkins presubmit Registry content changed
pull-ci-openshift-jenkins-release-4.13-e2e-aws-jenkins-sync-plugin openshift/jenkins presubmit Registry content changed
pull-ci-openshift-jenkins-release-4.13-e2e-aws-jenkins-client-plugin openshift/jenkins presubmit Registry content changed
pull-ci-openshift-jenkins-release-4.12-e2e-aws-jenkins-sync-plugin openshift/jenkins presubmit Registry content changed
pull-ci-openshift-jenkins-release-4.12-e2e-aws-jenkins-client-plugin openshift/jenkins presubmit Registry content changed
pull-ci-openshift-jenkins-release-4.11-e2e-aws-jenkins-sync-plugin openshift/jenkins presubmit Registry content changed
pull-ci-openshift-jenkins-release-4.11-e2e-aws-jenkins-client-plugin openshift/jenkins presubmit Registry content changed
pull-ci-openshift-jenkins-release-4.10-e2e-aws-jenkins-sync-plugin openshift/jenkins presubmit Registry content changed
pull-ci-openshift-jenkins-release-4.10-e2e-aws-jenkins-client-plugin openshift/jenkins presubmit Registry content changed
pull-ci-openshift-jenkins-master-e2e-aws openshift/jenkins presubmit Registry content changed
pull-ci-openshift-jenkins-release-4.15-e2e-aws openshift/jenkins presubmit Registry content changed
pull-ci-openshift-jenkins-release-4.14-e2e-aws openshift/jenkins presubmit Registry content changed

A total of 7099 jobs have been affected by this change. The above listing is non-exhaustive and limited to 35 jobs.

A full list of affected jobs can be found here
Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 10 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 20 rehearsals
Comment: /pj-rehearse max to run up to 35 rehearsals
Comment: /pj-rehearse auto-ack to run up to 10 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse abort to abort all active rehearsals

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@openshift-ci-robot
Copy link
Contributor

[REHEARSALNOTIFIER]
@mtulio: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-3scale-3scale-operator-master-test-e2e 3scale/3scale-operator presubmit Registry content changed
pull-ci-3scale-3scale-operator-3scale-2.12-candidate-test-e2e 3scale/3scale-operator presubmit Registry content changed
pull-ci-3scale-3scale-operator-3scale-2.11-stable-test-e2e 3scale/3scale-operator presubmit Registry content changed
pull-ci-3scale-3scale-operator-3scale-2.11-candidate-test-e2e 3scale/3scale-operator presubmit Registry content changed
pull-ci-3scale-3scale-operator-3scale-2.12-stable-test-e2e 3scale/3scale-operator presubmit Registry content changed
pull-ci-redhat-openshift-ecosystem-okd-operators-main-4.8-deploy-operator-on-openshift redhat-openshift-ecosystem/okd-operators presubmit Registry content changed
pull-ci-redhat-openshift-ecosystem-okd-operators-main-4.11-deploy-operator-on-openshift redhat-openshift-ecosystem/okd-operators presubmit Registry content changed
pull-ci-redhat-openshift-ecosystem-okd-operators-main-4.10-deploy-operator-on-openshift redhat-openshift-ecosystem/okd-operators presubmit Registry content changed
pull-ci-redhat-openshift-ecosystem-okd-operators-main-4.9-deploy-operator-on-openshift redhat-openshift-ecosystem/okd-operators presubmit Registry content changed
pull-ci-redhat-openshift-ecosystem-okd-operators-main-4.7-deploy-operator-on-openshift redhat-openshift-ecosystem/okd-operators presubmit Registry content changed
pull-ci-redhat-openshift-ecosystem-okd-operators-main-4.6-deploy-operator-on-openshift redhat-openshift-ecosystem/okd-operators presubmit Registry content changed
pull-ci-quay-quay-bridge-operator-master-ocp-latest-e2e quay/quay-bridge-operator presubmit Registry content changed
pull-ci-quay-quay-bridge-operator-master-ocp-4.6-e2e quay/quay-bridge-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-master-e2e-aws-operator openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.15-e2e-aws-operator openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.14-e2e-aws-operator openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.13-e2e-aws-operator openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.12-e2e-aws-operator openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.11-e2e-aws-operator openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.10-e2e-aws-operator openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.9-e2e-aws-operator openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.8-e2e-aws-operator openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.7-e2e-aws-operator openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.6-e2e-aws-operator openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.5-e2e-aws-operator openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.4-e2e-aws-operator openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.3-e2e-aws-operator openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.2-e2e-aws-operator openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.1-e2e-aws-operator openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-master-e2e-aws openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.15-e2e-aws openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.14-e2e-aws openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.13-e2e-aws openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.12-e2e-aws openshift/cluster-autoscaler-operator presubmit Registry content changed
pull-ci-openshift-cluster-autoscaler-operator-release-4.11-e2e-aws openshift/cluster-autoscaler-operator presubmit Registry content changed

A total of 7099 jobs have been affected by this change. The above listing is non-exhaustive and limited to 35 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 10 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 20 rehearsals
Comment: /pj-rehearse max to run up to 35 rehearsals
Comment: /pj-rehearse auto-ack to run up to 10 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse abort to abort all active rehearsals

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@mtulio
Copy link
Contributor Author

mtulio commented Jun 23, 2023

More improvements to increase the coverage in installer picking random zones even if it's not yet opted-in. Ref PR openshift/installer#7137

For IPI owners, ptal?
@vrutkovs @patrickdillon @yunjiang29

@mtulio
Copy link
Contributor Author

mtulio commented Jun 23, 2023

/assign @dgoodwin

@mtulio
Copy link
Contributor Author

mtulio commented Jun 23, 2023

/pj-rehearse skip

@openshift-ci-robot openshift-ci-robot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Jun 23, 2023
@mtulio
Copy link
Contributor Author

mtulio commented Jun 26, 2023

@vrutkovs would you mind taking a look? it's an small improvement in the zone query

@vrutkovs
Copy link
Member

Why do we want to increase the coverage? This should be in commit message / PR description

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jun 27, 2023

@mtulio: This pull request references SPLAT-657 which is a valid jira issue.

In response to this:

AWS is constantly adding new locations in Local Zones, we need to test it to prevent unwanted scenarios in features we are declaring supported.

openshift/enhancements#1232

Increasing the zone coverage by setting the flag --all-availability-zones to show zones which are not yet opted-in - the installer must opt-in when creating the cluster and that zone name is set in install-config.yaml:

  • us-east-1 in CI account without the flag --all-availability-zones
$ AWS_PROFILE=aws-local-zones aws ec2 describe-availability-zones  --region us-east-1  --filters Name=state,Values=available Name=zone-type,Values=local-zone --query 'AvailabilityZones[].ZoneName' | jq -r .[]
us-east-1-lim-1a
us-east-1-msp-1a
us-east-1-qro-1a
  • us-east-1 in CI account with the flag --all-availability-zones
$ AWS_PROFILE=aws-local-zones aws --region us-east-1 ec2 describe-availability-zones --all-availability-zones --filter Name=state,Values=available Name=zone-type,Values=local-zone | jq -r '.AvailabilityZones[].ZoneName' 
us-east-1-atl-1a
us-east-1-bos-1a
us-east-1-bue-1a
us-east-1-chi-1a
us-east-1-dfw-1a
us-east-1-iah-1a
us-east-1-lim-1a
us-east-1-mci-1a
us-east-1-mia-1a
us-east-1-msp-1a
us-east-1-nyc-1a
us-east-1-phl-1a
us-east-1-qro-1a
us-east-1-scl-1a
  • us-west-2 in CI account without the flag --all-availability-zones
$ AWS_PROFILE=aws-local-zones aws --region us-west-2 ec2 describe-availability-zones  --filter Name=state,Values=available Name=zone-type,Values=local-zone | jq -r '.AvailabilityZones[].ZoneName' 
us-west-2-las-1a
us-west-2-sea-1a
  • us-west-2 in CI account with the flag --all-availability-zones
$ AWS_PROFILE=aws-local-zones aws --region us-west-2 ec2 describe-availability-zones --all-availability-zones --filter Name=state,Values=available Name=zone-type,Values=local-zone | jq -r '.AvailabilityZones[].ZoneName' 
us-west-2-den-1a
us-west-2-las-1a
us-west-2-lax-1a
us-west-2-lax-1b
us-west-2-pdx-1a
us-west-2-phx-1a
us-west-2-sea-1a

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jun 27, 2023

@mtulio: This pull request references SPLAT-657 which is a valid jira issue.

In response to this:

AWS is constantly adding new locations in Local Zones, we need to test it to prevent unwanted scenarios in features we are declaring supported. It will also not increase the cost of the test as it is selecting randomly one single zone.

openshift/enhancements#1232

Increasing the zone coverage by setting the flag --all-availability-zones to show zones which are not yet opted-in - the installer must opt-in when creating the cluster and that zone name is set in install-config.yaml:

  • us-east-1 in CI account without the flag --all-availability-zones
$ AWS_PROFILE=aws-local-zones aws ec2 describe-availability-zones  --region us-east-1  --filters Name=state,Values=available Name=zone-type,Values=local-zone --query 'AvailabilityZones[].ZoneName' | jq -r .[]
us-east-1-lim-1a
us-east-1-msp-1a
us-east-1-qro-1a
  • us-east-1 in CI account with the flag --all-availability-zones
$ AWS_PROFILE=aws-local-zones aws --region us-east-1 ec2 describe-availability-zones --all-availability-zones --filter Name=state,Values=available Name=zone-type,Values=local-zone | jq -r '.AvailabilityZones[].ZoneName' 
us-east-1-atl-1a
us-east-1-bos-1a
us-east-1-bue-1a
us-east-1-chi-1a
us-east-1-dfw-1a
us-east-1-iah-1a
us-east-1-lim-1a
us-east-1-mci-1a
us-east-1-mia-1a
us-east-1-msp-1a
us-east-1-nyc-1a
us-east-1-phl-1a
us-east-1-qro-1a
us-east-1-scl-1a
  • us-west-2 in CI account without the flag --all-availability-zones
$ AWS_PROFILE=aws-local-zones aws --region us-west-2 ec2 describe-availability-zones  --filter Name=state,Values=available Name=zone-type,Values=local-zone | jq -r '.AvailabilityZones[].ZoneName' 
us-west-2-las-1a
us-west-2-sea-1a
  • us-west-2 in CI account with the flag --all-availability-zones
$ AWS_PROFILE=aws-local-zones aws --region us-west-2 ec2 describe-availability-zones --all-availability-zones --filter Name=state,Values=available Name=zone-type,Values=local-zone | jq -r '.AvailabilityZones[].ZoneName' 
us-west-2-den-1a
us-west-2-las-1a
us-west-2-lax-1a
us-west-2-lax-1b
us-west-2-pdx-1a
us-west-2-phx-1a
us-west-2-sea-1a

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@mtulio
Copy link
Contributor Author

mtulio commented Jun 27, 2023

Why do we want to increase the coverage? This should be in commit message / PR description

Updated.

That update is also also covered in the RFE for Local Zones.

@vrutkovs
Copy link
Member

Oh, I see. I don't know the implications of using not-yet-supported zones, I'll leave it for Devan to approve. I don't think picking random zone is a good idea cost-wise tbh

@mtulio
Copy link
Contributor Author

mtulio commented Jun 27, 2023

@vrutkovs - Thanks for sharing your thoughts.

I don't know the implications of using not-yet-supported zones, I'll leave it for Devan to approve.

Which zone is not supported? We are declaring support for Local Zones in 4.12. If we'll not test in CI, we are not testing what we are declaring as supported to our customers.

I don't think picking random zone is a good idea cost-wise tbh

What is the cost increase of getting static zone and random zone?[1][2] This suggestion was to prevent high costs from getting all zones in all tests (alongside increasing the total execution time, and risk for failures). The cost will increase for compute almost 14x (in us-east-1, N*zones) for each execution if we'll not pick one zone for each execution. Another cons of getting one static zone is we'll also not cover the customer scenario.

[1] https://github.com/openshift/enhancements/blob/master/enhancements/installer/aws-custom-edge-machineset-local-zones.md#infrastructure-costs
[2] https://aws.amazon.com/about-aws/global-infrastructure/localzones/pricing/

@mtulio
Copy link
Contributor Author

mtulio commented Jun 27, 2023

Hi Yunfei, as you are familiar with Local Zones, could you please take a look too?
/assign @yunjiang29

@patrickdillon
Copy link
Contributor

This LGTM but why skip the rehearsals? I would feel more confident with rehearsals. I think marco is out for the moment, so I'm going to go ahead and run them

/pj-rehearse

@mtulio
Copy link
Contributor Author

mtulio commented Jun 28, 2023

This LGTM but why skip the rehearsals? I would feel more confident with rehearsals. I think marco is out for the moment, so I'm going to go ahead and run them

/pj-rehearse

Thanks. Reason for skip:
The piece of the code that was changed is reachable only by the job pull-ci-openshift-installer-master-e2e-aws-ovn-localzones (or 4.14), which I think is expected to fail as the PR (openshift/installer#7137) in the installer is still open. So I wouldn't allocate the infra cost for that small change.

rehearsals history exercised when introducing this job:

Here you can see the merged version of this change (job pull-ci-openshift-installer-master-e2e-aws-ovn-localzones) running in the installer's PR mentioned above:
https://prow.ci.openshift.org/pr-history/?org=openshift&repo=installer&pr=7137

@mtulio
Copy link
Contributor Author

mtulio commented Jun 28, 2023

/pj-rehearse pull-ci-openshift-installer-master-e2e-aws-ovn-localzones

@openshift-ci-robot
Copy link
Contributor

@patrickdillon, pj-rehearse: failed to create rehearsal jobs ERROR:

failed to ensure imagestreamtags in cluster build05: failed waiting for imagestreamtag openshift/knative-v0.18.0:knative-eventing-contrib-src to appear: timed out waiting for the condition

If the problem persists, please contact Test Platform.

@mtulio
Copy link
Contributor Author

mtulio commented Jun 29, 2023

/pj-rehearse pull-ci-openshift-installer-master-e2e-aws-ovn-localzones

failed as expected (due the open PR): https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_release/40606/rehearse-40606-pull-ci-openshift-installer-master-e2e-aws-ovn-localzones/1674173677668667392#1:build-log.txt%3A129

zone selected:

- architecture: amd64
  name: edge
  platform:
    aws:
      zones: [us-east-1-lim-1a]

Triggering other tests which are more related to this change:
/pj-rehearse pull-ci-openshift-installer-master-e2e-aws-ovn
/pj-rehearse pull-ci-openshift-installer-master-e2e-aws-ovn-imdsv2
/pj-rehearse pull-ci-openshift-installer-master-e2e-aws-ovn-fips

@mtulio
Copy link
Contributor Author

mtulio commented Jun 29, 2023

/pj-rehearse pull-ci-openshift-installer-master-e2e-aws-ovn-imdsv2

@mtulio
Copy link
Contributor Author

mtulio commented Jun 29, 2023

Unrelated failure in the test [sig-api-machinery][Feature:ClusterResourceQuota] Cluster resource quota should control resource limits across namespaces [apigroup:quota.openshift.io][apigroup:image.openshift.io][apigroup:monitoring.coreos.com][apigroup:template.openshift.io] [Suite:openshift/conformance/parallel]
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_release/40606/rehearse-40606-pull-ci-openshift-installer-master-e2e-aws-ovn/1674287909835902976

/pj-rehearse pull-ci-openshift-installer-master-e2e-aws-ovn

@mtulio
Copy link
Contributor Author

mtulio commented Jun 29, 2023

This LGTM but why skip the rehearsals? I would feel more confident with rehearsals. I think marco is out for the moment, so I'm going to go ahead and run them

/pj-rehearse

@patrickdillon what is the pass ratio of those jobs to get confidence in your PoV?

@patrickdillon
Copy link
Contributor

This LGTM but why skip the rehearsals? I would feel more confident with rehearsals. I think marco is out for the moment, so I'm going to go ahead and run them
/pj-rehearse

@patrickdillon what is the pass ratio of those jobs to get confidence in your PoV?

Just a sanity check to make sure nothing is obviously broken. Is this an expected failure https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_release/40606/rehearse-40606-pull-ci-openshift-installer-master-e2e-aws-ovn-localzones/1674173677668667392

@mtulio
Copy link
Contributor Author

mtulio commented Jun 30, 2023

This LGTM but why skip the rehearsals? I would feel more confident with rehearsals. I think marco is out for the moment, so I'm going to go ahead and run them
/pj-rehearse

@patrickdillon what is the pass ratio of those jobs to get confidence in your PoV?

Just a sanity check to make sure nothing is obviously broken. Is this an expected failure https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_release/40606/rehearse-40606-pull-ci-openshift-installer-master-e2e-aws-ovn-localzones/1674173677668667392

@patrickdillon - Yes, expected fail preventing to install in non-existing VPC: edge machine pool is valid when installing in existing VPC (installer PR Phase II isn't in master, master supports only Phase I). My comment has more details.

@rvanderp3
Copy link
Contributor

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 30, 2023
@mtulio
Copy link
Contributor Author

mtulio commented Jul 3, 2023

/pj-rehearse pull-ci-openshift-installer-master-e2e-aws-ovn-fips

@mtulio
Copy link
Contributor Author

mtulio commented Jul 3, 2023

/pj-rehearse pull-ci-openshift-installer-master-e2e-aws-ovn

@mtulio
Copy link
Contributor Author

mtulio commented Jul 3, 2023

  • e2e-aws-ovn seems to be very flaky recently.
  • e2e-aws-ovn-fips passing.

@patrickdillon can you see any blocker in this PR?

@patrickdillon
Copy link
Contributor

/approve

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 3, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mtulio, patrickdillon, rvanderp3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 3, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 3, 2023

@mtulio: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/openshift/installer/master/e2e-aws-ovn-localzones c5a9a76 link unknown /pj-rehearse pull-ci-openshift-installer-master-e2e-aws-ovn-localzones
ci/rehearse/red-hat-storage/ocs-operator/main/red-hat-storage-ocs-ci-e2e-aws c5a9a76 link unknown /pj-rehearse pull-ci-red-hat-storage-ocs-operator-main-red-hat-storage-ocs-ci-e2e-aws
ci/rehearse/red-hat-storage/ocs-operator/release-4.8/red-hat-storage-ocs-ci-e2e-aws c5a9a76 link unknown /pj-rehearse pull-ci-red-hat-storage-ocs-operator-release-4.8-red-hat-storage-ocs-ci-e2e-aws
ci/rehearse/red-hat-storage/ocs-operator/release-4.6/red-hat-storage-ocs-ci-e2e-aws c5a9a76 link unknown /pj-rehearse pull-ci-red-hat-storage-ocs-operator-release-4.6-red-hat-storage-ocs-ci-e2e-aws
ci/rehearse/red-hat-storage/ocs-operator/release-4.5/ocs-operator-e2e-aws c5a9a76 link unknown /pj-rehearse pull-ci-red-hat-storage-ocs-operator-release-4.5-ocs-operator-e2e-aws
ci/rehearse/red-hat-storage/ocs-operator/release-4.9/red-hat-storage-ocs-ci-e2e-aws c5a9a76 link unknown /pj-rehearse pull-ci-red-hat-storage-ocs-operator-release-4.9-red-hat-storage-ocs-ci-e2e-aws
ci/rehearse/red-hat-storage/ocs-operator/release-4.13/red-hat-storage-ocs-ci-e2e-aws c5a9a76 link unknown /pj-rehearse pull-ci-red-hat-storage-ocs-operator-release-4.13-red-hat-storage-ocs-ci-e2e-aws
ci/rehearse/red-hat-storage/ocs-operator/release-4.7/red-hat-storage-ocs-ci-e2e-aws c5a9a76 link unknown /pj-rehearse pull-ci-red-hat-storage-ocs-operator-release-4.7-red-hat-storage-ocs-ci-e2e-aws
ci/rehearse/red-hat-storage/ocs-operator/release-4.11/red-hat-storage-ocs-ci-e2e-aws c5a9a76 link unknown /pj-rehearse pull-ci-red-hat-storage-ocs-operator-release-4.11-red-hat-storage-ocs-ci-e2e-aws
ci/rehearse/red-hat-storage/ocs-operator/release-4.10/red-hat-storage-ocs-ci-e2e-aws c5a9a76 link unknown /pj-rehearse pull-ci-red-hat-storage-ocs-operator-release-4.10-red-hat-storage-ocs-ci-e2e-aws
ci/rehearse/red-hat-storage/ocs-operator/release-4.12/red-hat-storage-ocs-ci-e2e-aws c5a9a76 link unknown /pj-rehearse pull-ci-red-hat-storage-ocs-operator-release-4.12-red-hat-storage-ocs-ci-e2e-aws
ci/rehearse/openshift/installer/master/e2e-aws-ovn c5a9a76 link unknown /pj-rehearse pull-ci-openshift-installer-master-e2e-aws-ovn

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot openshift-merge-robot merged commit a23dafa into openshift:master Jul 3, 2023
15 of 27 checks passed
@mtulio mtulio deleted the splat-657-presubmits-local-zones-coverage branch July 3, 2023 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. rehearsals-ack Signifies that rehearsal jobs have been acknowledged
Projects
None yet
8 participants