Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PODAUTO-21: Add keda e2e tests #43390

Merged
merged 1 commit into from
Sep 28, 2023
Merged

PODAUTO-21: Add keda e2e tests #43390

merged 1 commit into from
Sep 28, 2023

Conversation

jkyros
Copy link
Contributor

@jkyros jkyros commented Sep 14, 2023 •
edited

Neither keda nor the custom-metrics-autoscaler-operator had e2e tests, so we need to add some, but we need to do keda first -- I can't rehearse them both together.

This:

  • adds image builds for the keda components (the controller, the metrics adapter, and the webhook) that we can use both in the keda and in the CMA tests

  • coupled with the hacks in PODAUTO-21: Adjust e2e tests so they can run in OpenShift / OpenShif tCI kedacore-keda#15, will let us run the keda e2e suite here in openshift CI.

  • currently uses the upstream tools container in a Dockerfile literal to run the tests, but we hope to either get rid of that or mirror it/build it in CI eventually

  • currently skips the scalers/cpu and internals/global_custom_ca_test because they do not work in OpenShift for some reason, but we will figure that out and re-add them

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 14, 2023
@jkyros
Copy link
Contributor Author

jkyros commented Sep 14, 2023

/pj-rehearse pull-ci-openshift-kedacore-keda-main-keda-e2e-aws-ovn

@jkyros jkyros changed the title [WIP] Add Pod Autoscaling e2e tests [WIP]PODAUTO-21: Add Pod Autoscaling e2e tests Sep 15, 2023
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Sep 15, 2023 •
edited by openshift-ci bot

@jkyros: This pull request references PODAUTO-21 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.15.0" version, but no target version was set.

In response to this:

Ignore this for now.

Neither keda nor the custom-metrics-autoscaler-operator had e2e tests, so we need to add some.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 15, 2023
@jkyros
Copy link
Contributor Author

jkyros commented Sep 15, 2023

/pj-rehearse pull-ci-openshift-kedacore-keda-main-keda-e2e-aws-ovn

@jkyros
Copy link
Contributor Author

jkyros commented Sep 15, 2023

/pj-rehearse pull-ci-openshift-kedacore-keda-main-keda-e2e-aws-ovn

1 similar comment
@jkyros
Copy link
Contributor Author

jkyros commented Sep 15, 2023

/pj-rehearse pull-ci-openshift-kedacore-keda-main-keda-e2e-aws-ovn

@jkyros
Copy link
Contributor Author

jkyros commented Sep 15, 2023

/pj-rehearse pull-ci-openshift-kedacore-keda-main-keda-e2e-aws-ovn

@jkyros
Copy link
Contributor Author

jkyros commented Sep 15, 2023

/pj-rehearse pull-ci-openshift-kedacore-keda-main-keda-e2e-aws-ovn

@jkyros
Copy link
Contributor Author

jkyros commented Sep 15, 2023

/pj-rehearse pull-ci-openshift-kedacore-keda-main-keda-e2e-aws-ovn

@jkyros
Copy link
Contributor Author

jkyros commented Sep 15, 2023

/pj-rehearse pull-ci-openshift-kedacore-keda-main-keda-e2e-aws-ovn

@jkyros
Copy link
Contributor Author

jkyros commented Sep 16, 2023

/pj-rehearse pull-ci-openshift-kedacore-keda-main-keda-e2e-aws-ovn

@jkyros
Copy link
Contributor Author

jkyros commented Sep 16, 2023

/pj-rehearse pull-ci-openshift-kedacore-keda-main-keda-e2e-aws-ovn

@jkyros
Copy link
Contributor Author

jkyros commented Sep 17, 2023

/pj-rehearse pull-ci-openshift-kedacore-keda-main-keda-e2e-aws-ovn

@jkyros jkyros force-pushed the podauto-21-e2e-tests branch 3 times, most recently from 3e77101 to 16a638b Compare September 23, 2023 03:02
@jkyros
Copy link
Contributor Author

jkyros commented Sep 23, 2023

/pj-rehearse pull-ci-openshift-custom-metrics-autoscaler-operator-main-cma-e2e-aws-ovn

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Sep 28, 2023
@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Sep 28, 2023
@openshift-ci-robot openshift-ci-robot removed the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Sep 28, 2023
@jkyros
Copy link
Contributor Author

jkyros commented Sep 28, 2023

custom-metrics-autoscaler-test-tools-src renamed to custom-metrics-autoscaler-tests per conversation with Ben Parees, since the tests are in there -- and in the event that someday it's not "source" anymore (say we compile the tests to a binary or something if we figure out how to do that and still run a "partial test suite" ).

@jkyros
Copy link
Contributor Author

jkyros commented Sep 28, 2023

/pj-rehearse pull-ci-openshift-kedacore-keda-main-keda-e2e-aws-ovn

bparees
bparees reviewed Sep 28, 2023
View reviewed changes
ci-operator/config/openshift/kedacore-keda/openshift-kedacore-keda-main.yaml
ENV USE_SUDO=false
ENV PATH=/src/helm:/src/:$PATH
ENV HELM_INSTALL_DIR=/src
RUN curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 && chmod 700 get_helm.sh && ./get_helm.sh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think it would be better if this dockerfile lived in the repo instead of inlined here. but you can fix that as a followup.

ci-operator/config/openshift/kedacore-keda/openshift-kedacore-keda-main.yaml Outdated
@@ -44,6 +70,58 @@ tests:
clientset-generate
container:
from: src
- always_run: false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why false? should this be paired with a run_if_changed spec?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh hey, good catch. That was me originally being paranoid because I wasn't sure how stable this was going to be, but this seems decently stable.

I've removed that and added a skip_if_only_changed to exclude some of our doc (it was easier to exclude than include since so little was excluded).

@bparees
Copy link
Contributor

bparees commented Sep 28, 2023

/approve

name is approved and my other comments can be addressed as followups if any action is needed.

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 28, 2023
@jkyros
add keda e2e tests and component image builds
9797d73 
This adds:
- image builds for the keda components (controller, metrics adapter,
  webhooks)
- an image build deriving from the upstream keda-tools image that the
test suite expects to run in
- a test that runs the keda e2e suite against an aws cluster

The keda test suite was not designed to run in OpenShift CI. It mostly
expects to be run on a "vanilla kube" cluster, so there are some
quirks, mostly around privilege/permissions, the helm charts that it
tries to install, and the myriad public images it tries to pull.

The goal here is just to get some e2e tests running so we have some
signal and then clean this up over time, potentially with some upstream
test refactoring.

The scalers/cpu and internal/nternals/global_custom_ca_test are
currently not working and have been intentionally omitted, but they will
be re-added later once we figure out why.
@openshift-ci-robot
Copy link
Contributor

[REHEARSALNOTIFIER]
@jkyros: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-kedacore-keda-main-keda-e2e-aws-ovn openshift/kedacore-keda presubmit Presubmit changed
pull-ci-openshift-kedacore-keda-main-clientset-verify openshift/kedacore-keda presubmit Ci-operator config changed
pull-ci-openshift-kedacore-keda-main-cma-verify-history openshift/kedacore-keda presubmit Ci-operator config changed
pull-ci-openshift-kedacore-keda-main-gofmt openshift/kedacore-keda presubmit Ci-operator config changed
pull-ci-openshift-kedacore-keda-main-govet openshift/kedacore-keda presubmit Ci-operator config changed
pull-ci-openshift-kedacore-keda-main-images openshift/kedacore-keda presubmit Ci-operator config changed
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 10 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 20 rehearsals
Comment: /pj-rehearse max to run up to 35 rehearsals
Comment: /pj-rehearse auto-ack to run up to 10 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse abort to abort all active rehearsals

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@jkyros
Copy link
Contributor Author

jkyros commented Sep 28, 2023

uh...so I guess build09 had an accident and barfed all over? Anyway, let's try this again:

/pj-rehearse pull-ci-openshift-kedacore-keda-main-keda-e2e-aws-ovn

@jkyros
Copy link
Contributor Author

jkyros commented Sep 28, 2023

hmm, no. Maybe:
/test build09-dry

@jkyros
Copy link
Contributor Author

jkyros commented Sep 28, 2023

I figured that would work (once build09 was done doing...whatever it was doing)
/pj-rehearse ack

@openshift-ci-robot openshift-ci-robot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Sep 28, 2023
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Sep 28, 2023 •
edited by openshift-ci bot

@jkyros: This pull request references PODAUTO-21 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.15.0" version, but no target version was set.

In response to this:

Neither keda nor the custom-metrics-autoscaler-operator had e2e tests, so we need to add some, but we need to do keda first -- I can't rehearse them both together.

This:

  • adds image builds for the keda components (the controller, the metrics adapter, and the webhook) that we can use both in the keda and in the CMA tests

  • coupled with the hacks in PODAUTO-21: Adjust e2e tests so they can run in OpenShift / OpenShif tCI kedacore-keda#15, will let us run the keda e2e suite here in openshift CI.

  • currently uses the upstream tools container in a Dockerfile literal to run the tests, but we hope to either get rid of that or mirror it/build it in CI eventually

  • currently skips the scalers/cpu and internals/global_custom_ca_test because they do not work in OpenShift for some reason, but we will figure that out and re-add them

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@joelsmith
Copy link
Contributor

/lgtm
Thanks!

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Sep 28, 2023
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Sep 28, 2023 •
edited by openshift-ci bot

@jkyros: This pull request references PODAUTO-21 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.15.0" version, but no target version was set.

In response to this:

Neither keda nor the custom-metrics-autoscaler-operator had e2e tests, so we need to add some, but we need to do keda first -- I can't rehearse them both together.

This:

  • adds image builds for the keda components (the controller, the metrics adapter, and the webhook) that we can use both in the keda and in the CMA tests

  • coupled with the hacks in PODAUTO-21: Adjust e2e tests so they can run in OpenShift / OpenShif tCI kedacore-keda#15, will let us run the keda e2e suite here in openshift CI.

  • currently uses the upstream tools container in a Dockerfile literal to run the tests, but we hope to either get rid of that or mirror it/build it in CI eventually

  • currently skips the scalers/cpu and internals/global_custom_ca_test because they do not work in OpenShift for some reason, but we will figure that out and re-add them

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 28, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bparees, jkyros, joelsmith

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@joelsmith
Copy link
Contributor

Will we need to override this last test?

/test build09-dry

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 28, 2023 •
edited

@jkyros: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/openshift/custom-metrics-autoscaler-operator/main/cma-e2e-aws-ovn 16a638b6bc98e7fe95450abb0b93e8de3b92f5b8 link unknown /pj-rehearse pull-ci-openshift-custom-metrics-autoscaler-operator-main-cma-e2e-aws-ovn

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@jkyros
Copy link
Contributor Author

jkyros commented Sep 28, 2023

lets see if it rereads the config now that it's optional:
/test build09-dry

@smg247
Copy link
Member

smg247 commented Sep 28, 2023

/override ci/build-farm/build09-dry

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 28, 2023

@smg247: Overrode contexts on behalf of smg247: ci/build-farm/build09-dry

In response to this:

/override ci/build-farm/build09-dry

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-merge-robot openshift-merge-robot merged commit 4d1430c into openshift:master Sep 28, 2023
16 of 18 checks passed
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 28, 2023

@jkyros: Updated the image-mirror-mappings configmap in namespace ci at cluster app.ci using the following files:

  • key mapping_origin_4_15 using file core-services/image-mirroring/openshift/mapping_origin_4_15

In response to this:

Neither keda nor the custom-metrics-autoscaler-operator had e2e tests, so we need to add some, but we need to do keda first -- I can't rehearse them both together.

This:

  • adds image builds for the keda components (the controller, the metrics adapter, and the webhook) that we can use both in the keda and in the CMA tests

  • coupled with the hacks in PODAUTO-21: Adjust e2e tests so they can run in OpenShift / OpenShif tCI kedacore-keda#15, will let us run the keda e2e suite here in openshift CI.

  • currently uses the upstream tools container in a Dockerfile literal to run the tests, but we hope to either get rid of that or mirror it/build it in CI eventually

  • currently skips the scalers/cpu and internals/global_custom_ca_test because they do not work in OpenShift for some reason, but we will figure that out and re-add them

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jkyros jkyros mentioned this pull request Nov 9, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bparees bparees bparees left review comments

@joelsmith joelsmith Awaiting requested review from joelsmith

@rphillips rphillips Awaiting requested review from rphillips

Assignees

@joelsmith joelsmith

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. rehearsals-ack Signifies that rehearsal jobs have been acknowledged
Projects
None yet
Milestone
No milestone
6 participants
@jkyros @openshift-ci-robot @joelsmith @bparees @smg247 @openshift-merge-robot