Bug 2032566: Azure: Add Support for Azure Stack Hub #686

patrickdillon · 2021-12-13T16:23:43Z

Azure Stack is on-prem Azure, so the ARM endpoint, which is known in advance for public Azure or Government Azure, must be provided through configuration. This commit adds support for Azure Stack by checking the infrastructure object to determine the cloud and in the case of Azure Stack, uses the provided ARMEndpoint.

I think this is a bug because we released UPI in 4.9, and this should be supported in UPI. I will open a BZ for backporting in a moment.

Without this PR, Azure Stack is failing like this:

2021-12-12T20:13:36.690Z	ERROR	operator.init.controller-runtime.manager.controller.dns_controller	controller/controller.go:253	Reconciler error	{"name": "default-wildcard", "namespace": "openshift-ingress-operator", "error": "failed to create DNS provider: failed to create Azure DNS manager: could not determine cloud environment: open : no such file or directory"}

Also, at the moment, the installer does not create a privateDNSZone in the DNS manifest for Azure Stack (because Azure Stack does not have private DNS zones). Does the cluster-ingress-operator require BOTH a private and public zone in order to add *.apps?

patrickdillon · 2021-12-13T16:25:23Z

/hold
I am still testing but would like to rely on cluster bot for image building.

Azure Stack CI should be done soon openshift/release#24394 but I doubt in time for this PR.

patrickdillon · 2021-12-13T16:34:21Z

/test verify

Installing yq into '_output/tools/bin/yq-2.4.0'
mkdir -p '_output/tools/bin/'
curl -s -f -L https://github.com/mikefarah/yq/releases/download/2.4.0/yq_linux_amd64 -o '_output/tools/bin/yq-2.4.0'
chmod +x '_output/tools/bin/yq-2.4.0';
Installing yaml-patch into '_output/tools/bin/yaml-patch-v0.0.10'
mkdir -p '_output/tools/bin/'
curl -s -f -L https://github.com/krishicks/yaml-patch/releases/download/v0.0.10/yaml_patch_linux -o '_output/tools/bin/yaml-patch-v0.0.10'
make: *** [vendor/github.com/openshift/build-machinery-go/make/targets/openshift/operator/../yaml-patch.mk:18: ensure-yaml-patch] Error 22
{"component":"entrypoint","error":"wrapped process failed: exit status 2","file":"prow/entrypoint/run.go:80","func":"k8s.io/test-infra/prow/entrypoint.Options.Run","level":"error","msg":"Error executing test process","severity":"error","time":"2021-12-13T16:27:09Z"}

Doesn't look related to this changeset...

patrickdillon · 2021-12-13T16:49:16Z

curl -s -f -L https://github.com/krishicks/yaml-patch/releases/download/v0.0.10/yaml_patch_linux -o '_output/tools/bin/yaml-patch-v0.0.10'
This repo does not seem to exist: https://github.com/krishicks/yaml-patch

patrickdillon · 2021-12-13T16:50:57Z

openshift/build-machinery-go#56

Azure Stack is on-prem Azure, so the ARM endpoint, which is known in advance for public Azure or Government Azure, must be provided through configuration. This commit adds support for Azure Stack by checking the infrastructure object to determine the cloud and in the case of Azure Stack, uses the provided ARMEndpoint.

patrickdillon · 2021-12-13T18:47:40Z

I made it past the initial error in testing but I hit this issue and have updated accordingly.

patrickdillon · 2021-12-13T18:47:47Z

/test ?

openshift-ci · 2021-12-13T18:47:48Z

@patrickdillon: The following commands are available to trigger required jobs:

/test e2e-aws
/test e2e-aws-operator
/test e2e-gcp-serial
/test e2e-upgrade
/test images
/test unit
/test verify

The following commands are available to trigger optional jobs:

/test e2e-aws-single-node
/test e2e-azure
/test e2e-azure-operator
/test e2e-gcp-operator

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-cluster-ingress-operator-master-e2e-aws
pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator
pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-single-node
pull-ci-openshift-cluster-ingress-operator-master-e2e-gcp-serial
pull-ci-openshift-cluster-ingress-operator-master-e2e-upgrade
pull-ci-openshift-cluster-ingress-operator-master-images
pull-ci-openshift-cluster-ingress-operator-master-unit
pull-ci-openshift-cluster-ingress-operator-master-verify

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

patrickdillon · 2021-12-13T18:49:19Z

/test e2e-azure
/test e2e-azure-operator

Azure SDK Profile ensures compatibility with Azure Stack.

patrickdillon · 2021-12-13T21:02:25Z

/test e2e-azure
/test e2e-azure-operator

patrickdillon · 2021-12-13T21:54:55Z

/hold cancel

INFO Waiting up to 10m0s (until 9:56PM) for the openshift-console route to be created... 
DEBUG Route found in openshift-console namespace: console 
DEBUG OpenShift console route is admitted          
INFO Install complete!

patrickdillon · 2021-12-13T21:55:32Z

/cc @Miciah
/cc @staebler

patrickdillon · 2021-12-14T00:25:48Z

/test e2e-azure
/test e2e-azure-operator

staebler

Functionally, these changes look correct to me.

patrickdillon · 2021-12-14T03:18:34Z

/test e2e-azure
/test e2e-azure-operator

patrickdillon · 2021-12-14T11:23:49Z

/test verify

patrickdillon · 2021-12-14T14:24:04Z

verify test was supposedly fixed in openshift/api-machinery-go#58 but my retest failed. Just force pushed with no changes to see if that refreshes the ci namespace or something

patrickdillon · 2021-12-14T14:24:17Z

/test e2e-azure
/test e2e-azure-operator

patrickdillon · 2021-12-14T15:04:33Z

verify test was supposedly fixed in openshift/api-machinery-go#58 but my retest failed. Just force pushed with no changes to see if that refreshes the ci namespace or something

Oh, of course it wouldn't fix it until it is vendored in. Doh. #687 will take care of it.

/test e2e-azure
/test e2e-azure-operator

These passed earlier. Flaking now, methinks.

patrickdillon · 2021-12-14T15:55:46Z

These are build failures:

error: unable to parse image registry.build01.ci.openshift.org/ci-op-q7129nws/stable@sha256:005401f403d05c38167c5c5f4074d5d38a1e1fb0e08b8c4b8deaff13b22b5aab: cannot retrieve image configuration for manifest sha256:005401f403d05c38167c5c5f4074d5d38a1e1fb0e08b8c4b8deaff13b22b5aab: received unexpected HTTP status: 500 Internal Server Error
{"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"prow/entrypoint/run.go:80","func":"k8s.io/test-infra/prow/entrypoint.Options.Run","level":"error","msg":"Error executing test process","severity":"error","time":"2021-12-14T15:12:37Z"}

patrickdillon · 2021-12-14T17:58:20Z

/retitle Bug 2032566: Azure: Add Support for Azure Stack Hub

openshift-ci · 2021-12-14T17:58:27Z

@patrickdillon: This pull request references Bugzilla bug 2032566, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.10.0) matches configured target release for branch (4.10.0)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @lihongan

In response to this:

Bug 2032566: Azure: Add Support for Azure Stack Hub

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

patrickdillon · 2021-12-14T17:58:32Z

/bugzilla refresh

openshift-ci · 2021-12-14T17:58:35Z

@patrickdillon: This pull request references Bugzilla bug 2032566, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.10.0) matches configured target release for branch (4.10.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @lihongan

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Miciah · 2021-12-14T18:39:07Z

Also, at the moment, the installer does not create a privateDNSZone in the DNS manifest for Azure Stack (because Azure Stack does not have private DNS zones). Does the cluster-ingress-operator require BOTH a private and public zone in order to add *.apps?

The ingress operator itself doesn't require that both private and public zones be configured; the operator will publish records in the zones if they're specified, and it will not complain if a zone is not specified.

On some platforms (such as AWS), it is necessary to configure records in both the public zone and the private zone because otherwise a wildcard DNS record (such as *.mycluster.com) in the private zone can take priority over the ingress wildcard DNS record (*.apps.mycluster.com) if the latter only exists in the public zone. If Azure Stack Hub doesn't have private zones at all, then this shouldn't be an issue there.

Miciah · 2021-12-14T18:42:11Z

If e2e-azure-operator passes,
/lgtm
(Looking forward to having a CI job for Azure Stack Hub.)

openshift-ci · 2021-12-14T18:42:42Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Miciah, patrickdillon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Miciah]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Miciah · 2021-12-14T19:01:39Z

Seeing TestConfigurableRouteNoConsumingUserNoRBAC fail a lot, on multiple PRs. See search.ci.
/retest

patrickdillon · 2021-12-14T21:03:58Z

/bugzilla refresh

openshift-ci · 2021-12-14T21:04:03Z

@patrickdillon: This pull request references Bugzilla bug 2032566, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.10.0) matches configured target release for branch (4.10.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @lihongan

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-12-14T21:13:53Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-12-14T21:38:33Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-12-14T22:25:50Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-12-14T23:01:54Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-12-14T23:38:54Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

Miciah · 2021-12-14T23:43:14Z

TestUpdateDefaultIngressController and TestDefaultIngressCertificate failed.
/test e2e-azure-operator

openshift-ci · 2021-12-15T00:31:01Z

@patrickdillon: All pull requests linked via external trackers have merged:

openshift/cluster-ingress-operator#686

Bugzilla bug 2032566 has been moved to the MODIFIED state.

In response to this:

Bug 2032566: Azure: Add Support for Azure Stack Hub

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-12-15T00:31:27Z

@patrickdillon: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-cherrypick-robot · 2021-12-15T00:31:44Z

@patrickdillon: new pull request created: #690

In response to this:

/retitle Bug 2032566: Azure: Add Support for Azure Stack Hub

/cherry-pick release-4.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot requested review from ironcladlou and knobunc December 13, 2021 16:25

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 13, 2021

patrickdillon force-pushed the azurestack branch from 995fcfa to 5c5fe85 Compare December 13, 2021 18:45

Use Azure SDK Profile for DNS Client

04b05c2

Azure SDK Profile ensures compatibility with Azure Stack.

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 13, 2021

openshift-ci bot requested review from Miciah and staebler December 13, 2021 21:55

staebler reviewed Dec 14, 2021

View reviewed changes

vendor: azure-sdk-for-go profile

5786874

patrickdillon force-pushed the azurestack branch from 5061b6d to 5786874 Compare December 14, 2021 14:23

openshift-ci bot changed the title ~~Azure: Add Support for Azure Stack Hub~~ Bug 2032566: Azure: Add Support for Azure Stack Hub Dec 14, 2021

openshift-ci bot added the bugzilla/severity-unspecified Referenced Bugzilla bug's severity is unspecified for the PR. label Dec 14, 2021

openshift-ci bot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Dec 14, 2021

openshift-ci bot requested a review from lihongan December 14, 2021 17:58

openshift-ci bot assigned Miciah Dec 14, 2021

openshift-ci bot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Dec 14, 2021

openshift-ci bot added bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. and removed bugzilla/severity-unspecified Referenced Bugzilla bug's severity is unspecified for the PR. labels Dec 14, 2021

openshift-merge-robot merged commit 43c49a0 into openshift:master Dec 15, 2021

openshift-cherrypick-robot mentioned this pull request Dec 15, 2021

[release-4.9] Bug 2032677: Azure: Add Support for Azure Stack Hub #690

Merged

Bug 2032566: Azure: Add Support for Azure Stack Hub #686

Bug 2032566: Azure: Add Support for Azure Stack Hub #686

Conversation

patrickdillon commented Dec 13, 2021 • edited Loading

patrickdillon commented Dec 13, 2021

patrickdillon commented Dec 13, 2021

patrickdillon commented Dec 13, 2021

patrickdillon commented Dec 13, 2021

patrickdillon commented Dec 13, 2021

patrickdillon commented Dec 13, 2021

openshift-ci bot commented Dec 13, 2021

patrickdillon commented Dec 13, 2021

patrickdillon commented Dec 13, 2021

patrickdillon commented Dec 13, 2021

patrickdillon commented Dec 13, 2021

patrickdillon commented Dec 14, 2021

staebler left a comment

Choose a reason for hiding this comment

patrickdillon commented Dec 14, 2021

patrickdillon commented Dec 14, 2021

patrickdillon commented Dec 14, 2021

patrickdillon commented Dec 14, 2021

patrickdillon commented Dec 14, 2021

patrickdillon commented Dec 14, 2021

patrickdillon commented Dec 14, 2021

openshift-ci bot commented Dec 14, 2021

patrickdillon commented Dec 14, 2021

openshift-ci bot commented Dec 14, 2021

Miciah commented Dec 14, 2021 • edited Loading

Miciah commented Dec 14, 2021

openshift-ci bot commented Dec 14, 2021

Miciah commented Dec 14, 2021

patrickdillon commented Dec 14, 2021

openshift-ci bot commented Dec 14, 2021

openshift-bot commented Dec 14, 2021

openshift-bot commented Dec 14, 2021

openshift-bot commented Dec 14, 2021

openshift-bot commented Dec 14, 2021

openshift-bot commented Dec 14, 2021

Miciah commented Dec 14, 2021

openshift-ci bot commented Dec 15, 2021

openshift-ci bot commented Dec 15, 2021

openshift-cherrypick-robot commented Dec 15, 2021

patrickdillon commented Dec 13, 2021 •

edited

Loading

Miciah commented Dec 14, 2021 •

edited

Loading