Deprecate azure and gcp in-tree auth plugins #102181

enj · 2021-05-20T16:03:27Z

With the client-go credential plugin functionality going GA in 1.22,
it is now time to deprecate these legacy integrations.

Signed-off-by: Monis Khan mok@vmware.com

/kind cleanup
/kind deprecation
/sig auth
/priority important-soon
/triage accepted
/milestone v1.22
@kubernetes/sig-auth-pr-reviews
/assign @ritazh @mikedanese @cjcullen @ankeesler @liggitt

The in-tree azure and gcp auth plugins have been deprecated.  The https://github.com/Azure/kubelogin and gcloud commands serve as out-of-tree replacements via the kubectl/client-go credential plugin mechanism.

- [KEP]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/541-external-credential-providers

mikedanese · 2021-05-20T22:31:46Z

staging/src/k8s.io/client-go/plugin/pkg/client/auth/gcp/gcp.go

@@ -114,6 +114,9 @@ type gcpAuthProvider struct {
 }

 func newGCPAuthProvider(_ string, gcpConfig map[string]string, persister restclient.AuthProviderConfigPersister) (restclient.AuthProvider, error) {
+	klog.Warningf(`WARNING: in-tree gcp auth plugin is now deprecated.
+Please use the gcloud kubectl/client-go credential plugin instead.`)


We should hold off on this until we have an alternative. This is not currently actionable. Can we add this to release notes instead?

I would prefer to have the warning where an end-user would actually see it. Perhaps GCP can make a documentation page that we can link to? Then you can update the documentation page with details when it becomes actionable.

Also GCP has had years of runway to migrate away from this. This should be no surprise. It is time (technically in August).

Moved to a .V(1) to limit unactionable warning.

Opened #103499 to track moving this back to a Warningf by 1.24 at the latest.

enj · 2021-05-21T14:24:52Z

staging/src/k8s.io/client-go/plugin/pkg/client/auth/azure/azure.go

+		klog.Warningf(`WARNING: in-tree azure auth plugin is now deprecated.
+Please use the https://github.com/Azure/kubelogin kubectl/client-go credential plugin instead.`)


@ritazh are you happy with this verbiage?

enj · 2021-05-21T14:25:14Z

staging/src/k8s.io/client-go/plugin/pkg/client/auth/gcp/gcp.go

+		klog.V(1).Infof(`WARNING: in-tree gcp auth plugin is now deprecated.
+Please use the gcloud kubectl/client-go credential plugin instead.`)


@mikedanese are you happy with this verbiage?

I'll defer to mike on content, but I'd suggest a single line if possible

also, in other deprecation warnings, we've included the "from" version and the targeted removal version, e.g.

WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use ___ instead

I'll defer to mike on content, but I'd suggest a single line if possible

The docs link forced me to use multiple lines to keep the length sane.

Updated to include the from and target versions.

enj · 2021-05-21T14:29:25Z

/hold

(until PR to make client-go exec plugin officially GA is merged)

sambdavidson · 2021-06-07T22:44:22Z

staging/src/k8s.io/client-go/plugin/pkg/client/auth/gcp/gcp.go

 func newGCPAuthProvider(_ string, gcpConfig map[string]string, persister restclient.AuthProviderConfigPersister) (restclient.AuthProvider, error) {
+	// deprecated in v1.22, remove in v1.25
+	// this should be updated to use klog.Warningf in v1.24 to more actively warn consumers


Is there going to be some kind of process in place to ensure this happens? Otherwise is there anything programatic we could do to query the version at runtime and update the error?

Also it seems slightly risky to explicitly communicate the exact version it is being disabled. What if there is a lot of pushback at depreciation and its replacement and we need a few extra release cycles to hammer out those concerns?

Is there going to be some kind of process in place to ensure this happens?

I'd expect a milestoned issue

Also it seems slightly risky to explicitly communicate the exact version it is being disabled. What if there is a lot of pushback at depreciation and its replacement and we need a few extra release cycles to hammer out those concerns?

This won't merge until the client-go exec plugin GA merges. That is the target for any issues or concerns. Once that is released as GA, setting a definite removal version for the other auth plugins is reasonable.

Opened #103498 to track removal in 1.25. #102890 is tracking GA of client-go credential plugins.

sambdavidson · 2021-06-07T22:46:59Z

staging/src/k8s.io/client-go/plugin/pkg/client/auth/azure/azure.go

+	// deprecated in v1.22, remove in v1.25
+	warnOnce.Do(func() {
+		klog.Warningf(`WARNING: in-tree azure auth plugin is now deprecated.
+Please use the https://github.com/Azure/kubelogin kubectl/client-go credential plugin instead.`)


As a casual/novice Kubectl user kubectl/client-go doesn't seem very actionable to me (as opposed to Azure's login thing getting an entire URL). Is it a directory? Or package? Or Repo? I would prefer a more actionable link where I can learn how to use kubectl/client-go.

suggestions for alternative links are welcome

Added a link to the credential plugin docs.

enj · 2021-07-06T13:23:15Z

/retest

liggitt · 2021-07-06T16:02:53Z

staging/src/k8s.io/client-go/plugin/pkg/client/auth/azure/azure.go

+		klog.Warningf(`WARNING: the azure auth plugin is deprecated in v1.22+, unavailable in v1.25+; use https://github.com/Azure/kubelogin client-go credential plugin instead.
+
+To learn more about this feature, consult the documentation available at:
+      https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins`)


can we shrink to two lines instead of four?

WARNING: the azure auth plugin is deprecated in v1.22+, unavailable in v1.25+; use https://github.com/Azure/kubelogin instead. To learn more, consult https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins

liggitt · 2021-07-06T16:04:53Z

staging/src/k8s.io/client-go/plugin/pkg/client/auth/azure/azure.go

 func newAzureAuthProvider(_ string, cfg map[string]string, persister restclient.AuthProviderConfigPersister) (restclient.AuthProvider, error) {
+	// deprecated in v1.22, remove in v1.25
+	warnOnce.Do(func() {
+		klog.Warningf(`WARNING: the azure auth plugin is deprecated in v1.22+, unavailable in v1.25+; use https://github.com/Azure/kubelogin client-go credential plugin instead.


should this get the same V(1) treatment and move to a Warningf in 1.24?

Pinged @ritazh @aramase @weinong @feiskyer @karataliu @andyzhangx

+1 on what @liggitt pointed out.

Fixed. Opened #103525 to track.

looks good.

enj · 2021-07-06T19:28:12Z

/hold

(until PR to make client-go exec plugin officially GA is merged)

/hold cancel

#102890 has merged.

With the client-go credential plugin functionality going GA in 1.22, it is now time to deprecate these legacy integrations. Signed-off-by: Monis Khan <mok@vmware.com>

k8s-ci-robot · 2021-07-06T22:05:21Z

@enj: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-aks-engine-azure-disk-vmas	6bc9c858e9b5d71c6c18c1e25808be5ca38bd081	link	`/test pull-kubernetes-e2e-aks-engine-azure-disk-vmas`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

enj · 2021-07-06T22:20:35Z

/retest

liggitt · 2021-07-06T23:20:10Z

/lgtm
/approve

k8s-ci-robot · 2021-07-06T23:20:17Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enj, liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~staging/src/k8s.io/client-go/plugin/pkg/client/auth/OWNERS~~ [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

howardjohn · 2021-07-08T15:01:21Z

As a user reading the release notes, I am still not quite sure what I am expected to do to authenticate with GCP now. I hope it will not require the 1gb+ (and 10x slower in our tests) gcloud command just to connect to a GCP cluster?

EronWright · 2021-07-29T21:45:02Z

@howardjohn I observe that the (now deprecated) gcp provider has two modes, one of which doesn't depend on the gcloud tool (it directly uses a go library). I believe it varies based on whether the cmd-path option is set.

enj · 2021-08-07T03:27:56Z

As a user reading the release notes, I am still not quite sure what I am expected to do to authenticate with GCP now. I hope it will not require the 1gb+ (and 10x slower in our tests) gcloud command just to connect to a GCP cluster?

I suggest reaching out to GKE support. GKE will need to provide some binary to fetch credentials. Whether that is the gcloud binary or some new binary is entirely up to them. The same is true for all Kubernetes distributions.

howardjohn · 2021-08-07T14:00:55Z

So when I distribute my k8s controller (which needs to authenticate with all platforms) and stick it on a distroless image, it's not going to work anymore? I need to somehow find a magic binary for gke (and aks, eks,....), get them in the docker image, manage the versions, etc?

I very much hope I misunderstand this change - this seems like a very hostile change for users if not.

enj · 2021-08-08T00:48:25Z

So when I distribute my k8s controller (which needs to authenticate with all platforms) and stick it on a distroless image, it's not going to work anymore? I need to somehow find a magic binary for gke (and aks, eks,....), get them in the docker image, manage the versions, etc?

No, your controller would run as a Kubernetes service account via the in cluster config credentials provided uniformally across all clusters by the Kubelet. This is the standard way all controllers obtain an identity on Kubernetes. The various controller frameworks do this automatically.

I very much hope I misunderstand this change - this seems like a very hostile change for users if not.

This change pertains to human users obtaining proprietary credentials that are specific to their Kubernetes distribution. We are replacing the use of a proprietary SDK with a proper extension point.

zulh-civo · 2022-07-05T02:12:11Z

@enj how can I hide this warning when using kubectl?

enj · 2022-07-05T12:22:07Z

@enj how can I hide this warning when using kubectl?

Follow the directions to migrate to the replacement client-go credential plugin.

zulh-civo · 2022-07-06T00:35:54Z

Follow the directions to migrate to the replacement client-go credential plugin.

I did. But it's still showing. My GKE cluster is on 1.21.11. My kubectl is on 1.24.2.

k8s-ci-robot assigned ankeesler May 20, 2021

k8s-ci-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label May 20, 2021

k8s-ci-robot assigned cjcullen, liggitt and mikedanese May 20, 2021

k8s-ci-robot added the sig/auth Categorizes an issue or PR as relevant to SIG Auth. label May 20, 2021

k8s-ci-robot assigned ritazh May 20, 2021

k8s-ci-robot added this to the v1.22 milestone May 20, 2021

k8s-ci-robot requested review from cjcullen and sttts May 20, 2021 16:03

k8s-ci-robot added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label May 20, 2021

enj added this to Needs Triage PRs in SIG Auth Old May 20, 2021

enj moved this from Needs Triage PRs to In Review (v1.22) in SIG Auth Old May 20, 2021

mikedanese reviewed May 20, 2021

View reviewed changes

enj force-pushed the enj/i/deprecate_gcp_azure branch from c2bf4dd to 6bc9c85 Compare May 21, 2021 14:23

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels May 21, 2021

enj commented May 21, 2021

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 21, 2021

sambdavidson reviewed Jun 7, 2021

View reviewed changes

enj mentioned this pull request Jul 6, 2021

Update gcp auth plugin deprecation with actionable message #103499

Closed

liggitt reviewed Jul 6, 2021

View reviewed changes

enj force-pushed the enj/i/deprecate_gcp_azure branch from 74e5ca1 to 0c72fef Compare July 6, 2021 19:15

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 6, 2021

Deprecate azure and gcp in-tree auth plugins

6bfaeaf

With the client-go credential plugin functionality going GA in 1.22, it is now time to deprecate these legacy integrations. Signed-off-by: Monis Khan <mok@vmware.com>

enj force-pushed the enj/i/deprecate_gcp_azure branch from 0c72fef to 6bfaeaf Compare July 6, 2021 21:18

enj mentioned this pull request Jul 6, 2021

Update azure auth plugin deprecation to warning #103525

Closed

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 6, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 6, 2021

k8s-ci-robot merged commit 60475ee into kubernetes:master Jul 7, 2021

SIG Auth Old automation moved this from In Review (v1.22) to Closed / Done Jul 7, 2021

costinm mentioned this pull request Jul 6, 2022

Natively support GCR/ECR/ACR authentication istio/istio#39487

Closed

Freyert mentioned this pull request Aug 20, 2022

GCP auth plugin is deprecated linkerd/linkerd2#9218

Closed

embik mentioned this pull request Nov 28, 2022

kubermatic-installer convert-kubeconfig failing on gcp kubermatic/kubermatic#11458

Closed

jacobtomlinson mentioned this pull request Jul 25, 2023

Support auth without refresh tokens dask/dask-kubernetes#775

Closed

		klog.Warningf(`WARNING: in-tree azure auth plugin is now deprecated.
		Please use the https://github.com/Azure/kubelogin kubectl/client-go credential plugin instead.`)

		klog.V(1).Infof(`WARNING: in-tree gcp auth plugin is now deprecated.
		Please use the gcloud kubectl/client-go credential plugin instead.`)

Deprecate azure and gcp in-tree auth plugins #102181

Deprecate azure and gcp in-tree auth plugins #102181

Conversation

enj commented May 20, 2021 • edited Loading

Choose a reason for hiding this comment

enj May 21, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

enj commented May 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

enj commented Jul 6, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

enj commented Jul 6, 2021

k8s-ci-robot commented Jul 6, 2021 • edited Loading

enj commented Jul 6, 2021

liggitt commented Jul 6, 2021

k8s-ci-robot commented Jul 6, 2021

howardjohn commented Jul 8, 2021

EronWright commented Jul 29, 2021 • edited Loading

enj commented Aug 7, 2021

howardjohn commented Aug 7, 2021

enj commented Aug 8, 2021

zulh-civo commented Jul 5, 2022

enj commented Jul 5, 2022

zulh-civo commented Jul 6, 2022

enj commented May 20, 2021 •

edited

Loading

enj May 21, 2021 •

edited

Loading

k8s-ci-robot commented Jul 6, 2021 •

edited

Loading

EronWright commented Jul 29, 2021 •

edited

Loading