HOSTEDCP-1429: hypershift-operator: add a controller for t-shirt sizing #3686

stevekuznetsov · 2024-03-05T15:15:17Z

openshift-ci-robot · 2024-03-05T15:15:22Z

@stevekuznetsov: This pull request references HOSTEDCP-1429 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

/assign @csrwng

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

netlify · 2024-03-05T15:16:35Z

✅ Deploy Preview for hypershift-docs ready!

Name	Link
🔨 Latest commit	`4ff8a15`
🔍 Latest deploy log	https://app.netlify.com/sites/hypershift-docs/deploys/65e8ce700742430008f3d2ac
😎 Deploy Preview	https://deploy-preview-3686--hypershift-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

stevekuznetsov · 2024-03-05T17:05:01Z

Looks like I need to write that integration test :D

stevekuznetsov · 2024-03-05T21:43:09Z

// TODO: pause controller when reconciliation is paused

stevekuznetsov · 2024-03-06T20:14:12Z

@csrwng integration test:

$ GO_TEST_FLAGS='-run TestHostedSizingController' PATH="${PATH}:$(realpath ./bin)" WORK_DIR=/tmp/integration PULL_SECRET=~/secrets/ocm-pull-secret ./test/integration/run.sh test
Running test...
2024-03-06T13:13:07-07:00	INFO	running tests
=== RUN   TestHostedSizingController
    hosted_cluster_sizing_test.go:33: setting the cluster sizing configuration
    hosted_cluster_sizing_test.go:78: adding a small node count (1) to the hosted control plane hosted-clusters-2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8
    hosted_cluster_sizing_test.go:184: expecting hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 to transition to small size
    hosted_cluster_sizing_test.go:210: waiting for hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 to have status condition ClusterSizeComputed=True: small and match validation func
    hosted_cluster_sizing_test.go:224: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 observed at RV 2147 after 9.721094ms
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeTransitionPending=True: TransitionDelayNotElapsed (HostedClusters must wait at least 1h0m0s to increase in size after the last transition.)
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeComputed=True: small (The HostedCluster has transitioned to a new t-shirt size.)
    hosted_cluster_sizing_test.go:254: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 validation: hostedCluster.metadata.labels[io.openshift.hypershift/hosted-cluster-size]=small
    hosted_cluster_sizing_test.go:91: adding a large node count (50) to the hosted control plane hosted-clusters-2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8
    hosted_cluster_sizing_test.go:101: expecting hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 to delay transition
    hosted_cluster_sizing_test.go:210: waiting for hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 to have status condition ClusterSizeTransitionPending=True: TransitionDelayNotElapsed
    hosted_cluster_sizing_test.go:224: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 observed at RV 2149 after 9.176652ms
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeTransitionPending=True: TransitionDelayNotElapsed (HostedClusters must wait at least 1h0m0s to increase in size after the last transition.)
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeComputed=True: small (The HostedCluster has transitioned to a new t-shirt size.)
    hosted_cluster_sizing_test.go:107: configure the cluster sizing configuration to have miniscule delays
    hosted_cluster_sizing_test.go:184: expecting hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 to transition to medium size
    hosted_cluster_sizing_test.go:210: waiting for hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 to have status condition ClusterSizeComputed=True: medium and match validation func
    hosted_cluster_sizing_test.go:224: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 observed at RV 2149 after 2.248304ms
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeTransitionPending=True: TransitionDelayNotElapsed (HostedClusters must wait at least 1h0m0s to increase in size after the last transition.)
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeComputed=True: small (The HostedCluster has transitioned to a new t-shirt size.)
    hosted_cluster_sizing_test.go:224: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 observed at RV 2153 after 116.688634ms
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeComputed=True: medium (The HostedCluster has transitioned to a new t-shirt size.)
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeTransitionPending=False: ClusterSizeTransitioned (The HostedCluster has transitioned to a new t-shirt size.)
    hosted_cluster_sizing_test.go:254: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 validation: hostedCluster.metadata.labels[io.openshift.hypershift/hosted-cluster-size]=medium
    hosted_cluster_sizing_test.go:128: configure the cluster sizing configuration to have no concurrency
    hosted_cluster_sizing_test.go:148: adding a huge node count (500) to the hosted control plane hosted-clusters-2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8
    hosted_cluster_sizing_test.go:158: expecting hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 to delay transition for concurrency
    hosted_cluster_sizing_test.go:210: waiting for hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 to have status condition ClusterSizeTransitionPending=True: ConcurrencyLimitReached
    hosted_cluster_sizing_test.go:224: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 observed at RV 2153 after 2.245286ms
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeComputed=True: medium (The HostedCluster has transitioned to a new t-shirt size.)
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeTransitionPending=False: ClusterSizeTransitioned (The HostedCluster has transitioned to a new t-shirt size.)
    hosted_cluster_sizing_test.go:224: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 observed at RV 2157 after 100.333364ms
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeTransitionPending=True: TransitionDelayNotElapsed (HostedClusters must wait at least 1s to increase in size after the last transition.)
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeComputed=True: medium (The HostedCluster has transitioned to a new t-shirt size.)
    hosted_cluster_sizing_test.go:224: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 observed at RV 2159 after 513.554479ms
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeTransitionPending=True: ConcurrencyLimitReached (1 HostedClusters have already transitioned sizes in the last 1h0m0s, more time must elapse before the next transition.)
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeComputed=True: medium (The HostedCluster has transitioned to a new t-shirt size.)
    hosted_cluster_sizing_test.go:164: configure the cluster sizing configuration to have allow lots of concurrency
    hosted_cluster_sizing_test.go:184: expecting hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 to transition to large size
    hosted_cluster_sizing_test.go:210: waiting for hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 to have status condition ClusterSizeComputed=True: large and match validation func
    hosted_cluster_sizing_test.go:224: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 observed at RV 2159 after 2.33001ms
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeTransitionPending=True: ConcurrencyLimitReached (1 HostedClusters have already transitioned sizes in the last 1h0m0s, more time must elapse before the next transition.)
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeComputed=True: medium (The HostedCluster has transitioned to a new t-shirt size.)
    hosted_cluster_sizing_test.go:224: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 observed at RV 2163 after 104.618861ms
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeComputed=True: large (The HostedCluster has transitioned to a new t-shirt size.)
    hosted_cluster_sizing_test.go:236: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 status: ClusterSizeTransitionPending=False: ClusterSizeTransitioned (The HostedCluster has transitioned to a new t-shirt size.)
    hosted_cluster_sizing_test.go:254: hosted cluster hosted-clusters/2sbjc6akxi6ochgq0u1y0u6gah0ica9ph1s668m92fw8 validation: hostedCluster.metadata.labels[io.openshift.hypershift/hosted-cluster-size]=large
    run.go:79: cleaning up...
--- PASS: TestHostedSizingController (0.98s)
PASS
ok  	github.com/openshift/hypershift/test/integration	0.998s

cmd/install/install.go

There are only two places in k8s where ordering matters for what you're applying to the server: namespaces and CRDs. When we declare that we need CRDs, we also need to know that they are established before they can be used. New options to the install command allow us to wait on this explicitly, instead of implcitly, and output separate files during render to allow third parties to do so as well. Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

csrwng

nits

Dockerfile

api/scheduling/v1alpha1/clustersizingconfiguration_types.go

csrwng · 2024-03-11T19:53:17Z

/approve

openshift-ci · 2024-03-11T19:53:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csrwng, stevekuznetsov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [csrwng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hypershift-operator/controllers/hostedclustersizing/hostedclustersizing_controller.go

csrwng · 2024-03-12T01:10:42Z

/hold
@stevekuznetsov in manually testing this, I don't see the delay taking effect on scale up/down. I changed the config after the hypershift operator was already running, so not sure if the config is stale or it's something else.

stevekuznetsov · 2024-03-12T16:24:03Z

 Error: failed to create infra: cannot tag NAT gateway EIP: InvalidElasticIpID.NotFound: The elasticIp ID 'eipalloc-02b301c4c6f314eec' does not exist
	status code: 400, request id: c6d39334-9eac-4b65-a468-1971e77a3cfc
failed to create infra: cannot tag NAT gateway EIP: InvalidElasticIpID.NotFound: The elasticIp ID 'eipalloc-02b301c4c6f314eec' does not exist
	status code: 400, request id: c6d39334-9eac-4b65-a468-1971e77a3cfc

/retest

csrwng · 2024-03-13T01:35:32Z

/hold cancel

csrwng · 2024-03-13T01:55:17Z

@stevekuznetsov one thing that occurred to me is that we should skip reconcile of a hostedcluster that either has a deletionTimestamp or is paused. Other than that, lgtm

hypershift-operator/controllers/hostedclustersizing/hostedclustersizing_controller.go

csrwng · 2024-03-13T15:20:36Z

/lgtm

openshift-ci-robot · 2024-03-13T15:26:43Z

/retest-required

Remaining retests: 0 against base HEAD 8fc4d27 and 2 for PR HEAD d6f9feb in total

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

csrwng · 2024-03-13T17:16:43Z

/lgtm
/retest-required

openshift-ci-robot · 2024-03-13T18:36:26Z

/retest-required

Remaining retests: 0 against base HEAD 374e6c0 and 2 for PR HEAD c17b642 in total

openshift-ci · 2024-03-13T20:21:33Z

@stevekuznetsov: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-ibmcloud-iks	`c17b642`	link	false	`/test e2e-ibmcloud-iks`
ci/prow/e2e-ibmcloud-roks	`c17b642`	link	false	`/test e2e-ibmcloud-roks`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 5, 2024

openshift-ci bot assigned csrwng Mar 5, 2024

openshift-ci bot added the do-not-merge/needs-area label Mar 5, 2024

openshift-ci bot requested review from csrwng and sjenning March 5, 2024 15:17

stevekuznetsov force-pushed the skuznets/t-shirt-sizing branch from 35c2fba to e5626c5 Compare March 5, 2024 17:20

openshift-ci bot added the area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release label Mar 5, 2024

stevekuznetsov force-pushed the skuznets/t-shirt-sizing branch from e5626c5 to 31df990 Compare March 5, 2024 20:01

stevekuznetsov force-pushed the skuznets/t-shirt-sizing branch 4 times, most recently from 5c4377f to 4ff8a15 Compare March 6, 2024 20:13

openshift-ci bot added the area/testing Indicates the PR includes changes for e2e testing label Mar 6, 2024

stevekuznetsov force-pushed the skuznets/t-shirt-sizing branch from 4ff8a15 to 01c4be5 Compare March 6, 2024 20:17

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 6, 2024

stevekuznetsov force-pushed the skuznets/t-shirt-sizing branch from 01c4be5 to 08c9faf Compare March 6, 2024 22:22

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 6, 2024

stevekuznetsov force-pushed the skuznets/t-shirt-sizing branch from 08c9faf to 3cca91b Compare March 7, 2024 02:52

csrwng reviewed Mar 7, 2024

View reviewed changes

cmd/install/install.go Outdated Show resolved Hide resolved

csrwng reviewed Mar 11, 2024

View reviewed changes

Dockerfile Show resolved Hide resolved

api/scheduling/v1alpha1/clustersizingconfiguration_types.go Outdated Show resolved Hide resolved

api/scheduling/v1alpha1/clustersizingconfiguration_types.go Outdated Show resolved Hide resolved

stevekuznetsov force-pushed the skuznets/t-shirt-sizing branch from 3cca91b to ed0ab72 Compare March 11, 2024 18:53

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 11, 2024

csrwng reviewed Mar 11, 2024

View reviewed changes

hypershift-operator/controllers/hostedclustersizing/hostedclustersizing_controller.go Outdated Show resolved Hide resolved

stevekuznetsov force-pushed the skuznets/t-shirt-sizing branch from ed0ab72 to 451f123 Compare March 11, 2024 23:17

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 12, 2024

stevekuznetsov force-pushed the skuznets/t-shirt-sizing branch from 451f123 to 25fc012 Compare March 12, 2024 13:56

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 13, 2024

stevekuznetsov force-pushed the skuznets/t-shirt-sizing branch from 25fc012 to 341d4c6 Compare March 13, 2024 14:13

csrwng reviewed Mar 13, 2024

View reviewed changes

hypershift-operator/controllers/hostedclustersizing/hostedclustersizing_controller.go Outdated Show resolved Hide resolved

stevekuznetsov force-pushed the skuznets/t-shirt-sizing branch from 341d4c6 to d6f9feb Compare March 13, 2024 15:04

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Mar 13, 2024

hypershift-operator: add a controller for t-shirt sizing

c17b642

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

stevekuznetsov force-pushed the skuznets/t-shirt-sizing branch from d6f9feb to c17b642 Compare March 13, 2024 15:50

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Mar 13, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Mar 13, 2024

stevekuznetsov mentioned this pull request Mar 13, 2024

HOSTEDCP-1481,HOSTEDCP-1429: hypershift-operator: wire up placeholder pods for sizes #3749

Merged

openshift-merge-bot bot merged commit be38396 into openshift:main Mar 14, 2024
12 of 14 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HOSTEDCP-1429: hypershift-operator: add a controller for t-shirt sizing #3686

HOSTEDCP-1429: hypershift-operator: add a controller for t-shirt sizing #3686

stevekuznetsov commented Mar 5, 2024

openshift-ci-robot commented Mar 5, 2024 •

edited by openshift-ci bot

netlify bot commented Mar 5, 2024 •

edited

stevekuznetsov commented Mar 5, 2024

stevekuznetsov commented Mar 5, 2024

stevekuznetsov commented Mar 6, 2024

csrwng left a comment

csrwng commented Mar 11, 2024

openshift-ci bot commented Mar 11, 2024

csrwng commented Mar 12, 2024

stevekuznetsov commented Mar 12, 2024

csrwng commented Mar 13, 2024

csrwng commented Mar 13, 2024 •

edited

csrwng commented Mar 13, 2024

openshift-ci-robot commented Mar 13, 2024

csrwng commented Mar 13, 2024

openshift-ci-robot commented Mar 13, 2024

openshift-ci bot commented Mar 13, 2024

HOSTEDCP-1429: hypershift-operator: add a controller for t-shirt sizing #3686

HOSTEDCP-1429: hypershift-operator: add a controller for t-shirt sizing #3686

Conversation

stevekuznetsov commented Mar 5, 2024

openshift-ci-robot commented Mar 5, 2024 • edited by openshift-ci bot

netlify bot commented Mar 5, 2024 • edited

✅ Deploy Preview for hypershift-docs ready!

stevekuznetsov commented Mar 5, 2024

stevekuznetsov commented Mar 5, 2024

stevekuznetsov commented Mar 6, 2024

csrwng left a comment

Choose a reason for hiding this comment

csrwng commented Mar 11, 2024

openshift-ci bot commented Mar 11, 2024

csrwng commented Mar 12, 2024

stevekuznetsov commented Mar 12, 2024

csrwng commented Mar 13, 2024

csrwng commented Mar 13, 2024 • edited

csrwng commented Mar 13, 2024

openshift-ci-robot commented Mar 13, 2024

csrwng commented Mar 13, 2024

openshift-ci-robot commented Mar 13, 2024

openshift-ci bot commented Mar 13, 2024

openshift-ci-robot commented Mar 5, 2024 •

edited by openshift-ci bot

netlify bot commented Mar 5, 2024 •

edited

csrwng commented Mar 13, 2024 •

edited