Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] SDN-3901: CNO 4.14 kubernetes 1.27 #1806

Conversation

flavio-fernandes
Copy link
Contributor

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 8, 2023
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented May 8, 2023

@flavio-fernandes: This pull request references SDN-3901 which is a valid jira issue.

In response to this:

Jira: https://issues.redhat.com/browse/SDN-3901

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 8, 2023
@openshift-ci openshift-ci bot requested review from dougbtv and trozet May 8, 2023 20:34
@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 8, 2023

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: flavio-fernandes
Once this PR has been reviewed and has the lgtm label, please assign danwinship for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 10, 2023
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 17, 2023
@flavio-fernandes
Copy link
Contributor Author

flavio-fernandes commented May 17, 2023

I'm in a pickle. If I try building after upgrading to 1.27 I hit the following failure:

# sigs.k8s.io/controller-runtime/pkg/cache
vendor/sigs.k8s.io/controller-runtime/pkg/cache/multi_namespace_cache.go:308:9: cannot use handles (variable of type map[string]"k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration) as "k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration value in return statement: map[string]"k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration does not implement "k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration (missing method HasSynced)
vendor/sigs.k8s.io/controller-runtime/pkg/cache/multi_namespace_cache.go:321:9: cannot use handles (variable of type map[string]"k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration) as "k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration value in return statement: map[string]"k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration does not implement "k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration (missing method HasSynced)
vendor/sigs.k8s.io/controller-runtime/pkg/cache/multi_namespace_cache.go:326:17: impossible type assertion: h.(map[string]toolscache.ResourceEventHandlerRegistration)
        map[string]"k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration does not implement "k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration (missing method HasSynced)
make: *** [vendor/github.com/openshift/build-machinery-go/make/targets/golang/build.mk:16: build] Error 1

I can see that this particular issue has been addressed here:
kubernetes-sigs/controller-runtime#2223

However, by pulling in that fix, I break sigs.k8s.io/cluster-api as follows:

go build -mod=vendor -trimpath -ldflags "-X github.com/openshift/cluster-network-operator/pkg/version.versionFromGit="v0.0.0-unknown-08d8ed444" -X github.com/openshift/cluster-network-operator/pkg/version.commitFromGit="08d8ed444" -X github.c
om/openshift/cluster-network-operator/pkg/version.gitTreeState="clean" -X github.com/openshift/cluster-network-operator/pkg/version.buildDate="2023-05-17T20:42:36Z" " github.com/openshift/cluster-network-operator/cmd/cluster-network-operator
# sigs.k8s.io/cluster-api/api/v1beta1
vendor/sigs.k8s.io/cluster-api/api/v1beta1/machine_webhook.go:45:27: cannot use &Machine{} (value of type *Machine) as admission.Validator value in variable declaration: *Machine does not implement admission.Validator (wrong type for method V
alidateCreate)
                have ValidateCreate() error
                want ValidateCreate() (admission.Warnings, error)
vendor/sigs.k8s.io/cluster-api/api/v1beta1/machinedeployment_webhook.go:47:27: cannot use &MachineDeployment{} (value of type *MachineDeployment) as admission.Validator value in variable declaration: *MachineDeployment does not implement admi
ssion.Validator (wrong type for method ValidateCreate)
                have ValidateCreate() error
                want ValidateCreate() (admission.Warnings, error)
vendor/sigs.k8s.io/cluster-api/api/v1beta1/machinehealthcheck_webhook.go:63:27: cannot use &MachineHealthCheck{} (value of type *MachineHealthCheck) as admission.Validator value in variable declaration: *MachineHealthCheck does not implement
admission.Validator (wrong type for method ValidateCreate)
                have ValidateCreate() error
                want ValidateCreate() (admission.Warnings, error)
vendor/sigs.k8s.io/cluster-api/api/v1beta1/machineset_webhook.go:45:27: cannot use &MachineSet{} (value of type *MachineSet) as admission.Validator value in variable declaration: *MachineSet does not implement admission.Validator (wrong type
for method ValidateCreate)
                have ValidateCreate() error
                want ValidateCreate() (admission.Warnings, error)
vendor/sigs.k8s.io/cluster-api/api/v1beta1/machineset_types.go:184:83: not enough arguments in call to metav1validation.ValidateLabelSelector
        have (*"k8s.io/apimachinery/pkg/apis/meta/v1".LabelSelector, *"k8s.io/apimachinery/pkg/util/validation/field".Path)
        want (*"k8s.io/apimachinery/pkg/apis/meta/v1".LabelSelector, "k8s.io/apimachinery/pkg/apis/meta/v1/validation".LabelSelectorValidationOptions, *"k8s.io/apimachinery/pkg/util/validation/field".Path)
make: *** [vendor/github.com/openshift/build-machinery-go/make/targets/golang/build.mk:16: build] Error 1

After upgrading go get -u sigs.k8s.io/cluster-api I still hit the same issue. Also tried go get sigs.k8s.io/cluster-api@main
without success:

❯ make build
fatal: No names found, cannot describe anything.
fatal: No names found, cannot describe anything.
fatal: No names found, cannot describe anything.
go build -mod=vendor -trimpath -ldflags "-X github.com/openshift/cluster-network-operator/pkg/version.versionFromGit="v0.0.0-unknown-129262162" -X github.com/openshift/cluster-network-operator/pkg/version.commitFromGit="129262162" -X github.com/openshift/cluster-network-operator/pkg/version.gitTreeState="clean" -X github.com/openshift/cluster-network-operator/pkg/version.buildDate="2023-05-17T21:33:02Z" " github.com/openshift/cluster-network-operator/cmd/cluster-network-check-endpoints
go build -mod=vendor -trimpath -ldflags "-X github.com/openshift/cluster-network-operator/pkg/version.versionFromGit="v0.0.0-unknown-129262162" -X github.com/openshift/cluster-network-operator/pkg/version.commitFromGit="129262162" -X github.com/openshift/cluster-network-operator/pkg/version.gitTreeState="clean" -X github.com/openshift/cluster-network-operator/pkg/version.buildDate="2023-05-17T21:33:03Z" " github.com/openshift/cluster-network-operator/cmd/cluster-network-check-target
go build -mod=vendor -trimpath -ldflags "-X github.com/openshift/cluster-network-operator/pkg/version.versionFromGit="v0.0.0-unknown-129262162" -X github.com/openshift/cluster-network-operator/pkg/version.commitFromGit="129262162" -X github.com/openshift/cluster-network-operator/pkg/version.gitTreeState="clean" -X github.com/openshift/cluster-network-operator/pkg/version.buildDate="2023-05-17T21:33:05Z" " github.com/openshift/cluster-network-operator/cmd/cluster-network-operator
# sigs.k8s.io/cluster-api/api/v1beta1
vendor/sigs.k8s.io/cluster-api/api/v1beta1/machine_webhook.go:45:27: cannot use &Machine{} (value of type *Machine) as admission.Validator value in variable declaration: *Machine does not implement admission.Validator (wrong type for method ValidateCreate)
                have ValidateCreate() error
                want ValidateCreate() (admission.Warnings, error)
vendor/sigs.k8s.io/cluster-api/api/v1beta1/machinedeployment_webhook.go:56:27: cannot use &MachineDeployment{} (value of type *MachineDeployment) as admission.Validator value in variable declaration: *MachineDeployment does not implement admission.Validator (wrong type for method ValidateCreate)
                have ValidateCreate() error
                want ValidateCreate() (admission.Warnings, error)
vendor/sigs.k8s.io/cluster-api/api/v1beta1/machinedeployment_webhook.go:61:16: assignment mismatch: 2 variables but admission.NewDecoder returns 1 value
vendor/sigs.k8s.io/cluster-api/api/v1beta1/machinehealthcheck_webhook.go:63:27: cannot use &MachineHealthCheck{} (value of type *MachineHealthCheck) as admission.Validator value in variable declaration: *MachineHealthCheck does not implement admission.Validator (wrong type for method ValidateCreate)
                have ValidateCreate() error
                want ValidateCreate() (admission.Warnings, error)
vendor/sigs.k8s.io/cluster-api/api/v1beta1/machineset_webhook.go:45:27: cannot use &MachineSet{} (value of type *MachineSet) as admission.Validator value in variable declaration: *MachineSet does not implement admission.Validator (wrong type for method ValidateCreate)
                have ValidateCreate() error
                want ValidateCreate() (admission.Warnings, error)
make: *** [vendor/github.com/openshift/build-machinery-go/make/targets/golang/build.mk:16: build] Error 1

@flavio-fernandes
Copy link
Contributor Author

With @martinkennelly's help, we identified kubernetes-sigs/cluster-api#8007 for capi, where the pickle issue described above gets resolved. To be revisited!

@flavio-fernandes
Copy link
Contributor Author

Well, one step fwd, and two steps... sideways. ;)

This is where I'm stuck now:

❯ make build
make: Warning: File 'vendor/github.com/openshift/build-machinery-go/make/targets/openshift/operator/../../../lib/tmp.mk' has modification time 3.4 s in the future
fatal: No names found, cannot describe anything.
fatal: No names found, cannot describe anything.
fatal: No names found, cannot describe anything.
go build -mod=vendor -trimpath -ldflags "-X github.com/openshift/cluster-network-operator/pkg/version.versionFromGit="v0.0.0-unknown-f68e72f6c" -X github.com/openshift/cluster-network-operator/pkg/version.commitFromGit="f68e72f6c" -X github.com/openshift/cluster-network-operator/pkg/version.gitTreeState="clean" -X github.com/openshift/cluster-network-operator/pkg/version.buildDate="2023-05-26T21:08:49Z" " github.com/openshift/cluster-network-operator/cmd/cluster-network-check-endpoints
# k8s.io/apiserver/pkg/server/routes
vendor/k8s.io/apiserver/pkg/server/routes/openapi.go:46:8: openAPIVersionedService.RegisterOpenAPIVersionedService("/openapi/v2", mux) (no value) used as value
make: *** [vendor/github.com/openshift/build-machinery-go/make/targets/golang/build.mk:16: build] Error 1

go mod edit -go=1.20

removed k8s.io/apimachinery from replace section
changed k8s.io/client-go to v0.27.2 in replace section

added k8s.io/kube-openapi v0.0.0-20230515203736-54b630e78af5 in replace section to addresss
issue with openAPIVersionedService.RegisterOpenAPIVersionedService compile

go get -u k8s.io/api ; \
go get -u k8s.io/apiextensions-apiserver ; \
go mod tidy

Signed-off-by: Flavio Fernandes <flaviof@redhat.com>
Signed-off-by: Flavio Fernandes <flaviof@redhat.com>
@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 31, 2023

@flavio-fernandes: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-network-mtu-migration-sdn-ipv4 edbb050 link false /test e2e-network-mtu-migration-sdn-ipv4
ci/prow/e2e-azure-ovn edbb050 link false /test e2e-azure-ovn
ci/prow/e2e-vsphere-ovn edbb050 link false /test e2e-vsphere-ovn
ci/prow/e2e-ovn-ipsec-step-registry edbb050 link false /test e2e-ovn-ipsec-step-registry
ci/prow/e2e-aws-ovn-serial edbb050 link false /test e2e-aws-ovn-serial
ci/prow/e2e-ovn-hybrid-step-registry edbb050 link false /test e2e-ovn-hybrid-step-registry
ci/prow/e2e-hypershift-ovn edbb050 link true /test e2e-hypershift-ovn
ci/prow/e2e-aws-sdn-multi edbb050 link true /test e2e-aws-sdn-multi
ci/prow/e2e-gcp-ovn edbb050 link true /test e2e-gcp-ovn
ci/prow/e2e-aws-ovn-network-migration edbb050 link true /test e2e-aws-ovn-network-migration
ci/prow/e2e-aws-sdn-network-migration-rollback edbb050 link true /test e2e-aws-sdn-network-migration-rollback
ci/prow/e2e-gcp-ovn-upgrade edbb050 link false /test e2e-gcp-ovn-upgrade
ci/prow/e2e-aws-sdn-upgrade edbb050 link false /test e2e-aws-sdn-upgrade
ci/prow/e2e-vsphere-ovn-windows edbb050 link true /test e2e-vsphere-ovn-windows
ci/prow/lint edbb050 link true /test lint
ci/prow/e2e-aws-ovn-windows edbb050 link true /test e2e-aws-ovn-windows
ci/prow/e2e-network-mtu-migration-ovn-ipv6 edbb050 link false /test e2e-network-mtu-migration-ovn-ipv6
ci/prow/e2e-openstack-sdn edbb050 link false /test e2e-openstack-sdn
ci/prow/e2e-metal-ipi-ovn-ipv6-ipsec edbb050 link false /test e2e-metal-ipi-ovn-ipv6-ipsec
ci/prow/e2e-vsphere-ovn-dualstack edbb050 link false /test e2e-vsphere-ovn-dualstack
ci/prow/e2e-network-mtu-migration-ovn-ipv4 edbb050 link false /test e2e-network-mtu-migration-ovn-ipv4
ci/prow/e2e-aws-ovn-single-node edbb050 link false /test e2e-aws-ovn-single-node
ci/prow/e2e-ovn-step-registry edbb050 link false /test e2e-ovn-step-registry
ci/prow/e2e-aws-sdn-network-reverse-migration edbb050 link true /test e2e-aws-sdn-network-reverse-migration
ci/prow/e2e-gcp-sdn edbb050 link true /test e2e-gcp-sdn
ci/prow/e2e-openstack-ovn edbb050 link false /test e2e-openstack-ovn
ci/prow/images edbb050 link true /test images
ci/prow/e2e-metal-ipi-ovn-ipv6 edbb050 link true /test e2e-metal-ipi-ovn-ipv6
ci/prow/unit edbb050 link true /test unit
ci/prow/verify edbb050 link true /test verify

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@flavio-fernandes
Copy link
Contributor Author

Another step forward and another pitfall. In order to get past the issue with openAPIVersionedService.RegisterOpenAPIVersionedService("/openapi/v2", mux) mentioned in
#1806 (comment) the following
version of k8s.io/kube-openapi is added to replace section v0.0.0-20230515203736-54b630e78af5
Ref: https://pkg.go.dev/k8s.io/kube-openapi@v0.0.0-20230515203736-54b630e78af5/pkg/handler

But now build fails in the following place (see below) and this is the version of k8s.io/client-go that
has the implementation of ResourceEventHandlerRegistration that does not expects missing method HasSynced: v0.27.0-alpha.0

I located the latest 1.27 version of k8s.io/client-go that uses ResourceEventHandlerRegistration without the missing method HasSynced and tried using it:
https://pkg.go.dev/k8s.io/client-go@v0.27.0-alpha.0/tools/cache#ResourceEventHandlerRegistration

diff --git a/go.mod b/go.mod
index 2bc495864..cf0dee439 100644
--- a/go.mod
+++ b/go.mod
@@ -152,7 +152,7 @@ require (

 replace (
        github.com/openshift/hypershift/api => github.com/openshift/hypershift v0.0.0-20220323152148-c356b8b72d66
-       k8s.io/client-go => k8s.io/client-go v0.27.2
+       k8s.io/client-go => k8s.io/client-go v0.27.0-alpha.0
        k8s.io/kube-openapi => k8s.io/kube-openapi v0.0.0-20230515203736-54b630e78af5
        // for Hypershift
        kubevirt.io/containerized-data-importer-api => github.com/kubevirt/containerized-data-importer-api v1.41.1-0.20211201033752-05520fb9f18d

But could not get past mod tidy:

❯ go mod tidy
go: finding module for package k8s.io/client-go/tools/cache/synctrack
go: finding module for package k8s.io/api/resource/v1alpha1
github.com/openshift/cluster-network-operator/pkg/client imports
        k8s.io/client-go/informers imports
        k8s.io/api/resource/v1alpha1: module k8s.io/api@latest found (v0.27.2), but does not contain package k8s.io/api/resource/v1alpha1
github.com/openshift/cluster-network-operator/cmd/cluster-network-operator imports
        github.com/openshift/library-go/pkg/controller/controllercmd imports
        k8s.io/apiserver/pkg/server imports
        k8s.io/apiserver/pkg/admission/plugin/validatingadmissionpolicy imports
        k8s.io/apiserver/pkg/admission/plugin/validatingadmissionpolicy/internal/generic imports
        k8s.io/client-go/tools/cache/synctrack: module k8s.io/client-go@latest found (v11.0.0+incompatible, replaced by k8s.io/client-go@v0.27.0-alpha.0), but does not contain package k8s.io/client-go/tools/cache/synctrack
❯ make build
fatal: No names found, cannot describe anything.
fatal: No names found, cannot describe anything.
fatal: No names found, cannot describe anything.
go build -mod=vendor -trimpath -ldflags "-X github.com/openshift/cluster-network-operator/pkg/version.versionFromGit="v0.0.0-unknown-edbb05081" -X github.com/openshift/cluster-network-operator/pkg/version.commitFromGit="edbb05081" -X github.com/openshift/cluster-network-operator/pkg/version.gitTreeState="clean" -X github.com/openshift/cluster-network-operator/pkg/version.buildDate="2023-05-31T21:41:15Z" " github.com/openshift/cluster-network-operator/cmd/cluster-network-check-endpoints
go build -mod=vendor -trimpath -ldflags "-X github.com/openshift/cluster-network-operator/pkg/version.versionFromGit="v0.0.0-unknown-edbb05081" -X github.com/openshift/cluster-network-operator/pkg/version.commitFromGit="edbb05081" -X github.com/openshift/cluster-network-operator/pkg/version.gitTreeState="clean" -X github.com/openshift/cluster-network-operator/pkg/version.buildDate="2023-05-31T21:41:16Z" " github.com/openshift/cluster-network-operator/cmd/cluster-network-check-target
go build -mod=vendor -trimpath -ldflags "-X github.com/openshift/cluster-network-operator/pkg/version.versionFromGit="v0.0.0-unknown-edbb05081" -X github.com/openshift/cluster-network-operator/pkg/version.commitFromGit="edbb05081" -X github.com/openshift/cluster-network-operator/pkg/version.gitTreeState="clean" -X github.com/openshift/cluster-network-operator/pkg/version.buildDate="2023-05-31T21:41:18Z" " github.com/openshift/cluster-network-operator/cmd/cluster-network-operator
# sigs.k8s.io/controller-runtime/pkg/cache
vendor/sigs.k8s.io/controller-runtime/pkg/cache/multi_namespace_cache.go:308:9: cannot use handles (variable of type map[string]"k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration) as "k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration value in return statement: map[string]"k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration does not implement "k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration (missing method HasSynced)
vendor/sigs.k8s.io/controller-runtime/pkg/cache/multi_namespace_cache.go:321:9: cannot use handles (variable of type map[string]"k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration) as "k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration value in return statement: map[string]"k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration does not implement "k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration (missing method HasSynced)
vendor/sigs.k8s.io/controller-runtime/pkg/cache/multi_namespace_cache.go:326:17: impossible type assertion: h.(map[string]toolscache.ResourceEventHandlerRegistration)
        map[string]"k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration does not implement "k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration (missing method HasSynced)
make: *** [vendor/github.com/openshift/build-machinery-go/make/targets/golang/build.mk:16: build] Error 1

@flavio-fernandes
Copy link
Contributor Author

In order to get past

map[string]"k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration does not implement "k8s.io/client-go/tools/cache".ResourceEventHandlerRegistration (missing method HasSynced)

I tried using a replace for client-go: k8s.io/client-go v0.27.0-alpha.0 but that is not working:

❯ go mod tidy
go: warning: "all" matched no packages

@flavio-fernandes flavio-fernandes marked this pull request as draft June 7, 2023 20:57
@flavio-fernandes
Copy link
Contributor Author

Abandoning this PR. Will resume this work in #1826

@openshift-merge-robot
Copy link
Contributor

Fix included in accepted release 4.14.0-0.nightly-2023-09-11-201102

@openshift-merge-robot
Copy link
Contributor

Fix included in accepted release 4.14.0-0.nightly-2023-09-12-024050

@openshift-merge-robot
Copy link
Contributor

Fix included in accepted release 4.14.0-0.nightly-2023-09-15-101929

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants