Use of anonymous kubeclient fails on restart/run

[midN]

Bug report

[Description of the bug]
It looks like compared to deploy, restart and run both use Kubeclient instead of kubectl command execution.
And Kubeclient called from kubernetes-deploy uses an anonymous user to query namespaces.
It fails miserably in protected clusters that disable any access to anonymous users.

Expected behavior: [What you expected to happen]
Use my "authenticated" kubectl to query namespaces

Actual behavior: [What actually happened]
Uses Kubernetes API without any form of authentication and fails

Proposal:
Replace kubeclient usage with kubectl calls and let kubectl handle the authentication.
Using kubeclient for authentication and any API calls might result in too much extra work.
Not all clusters use simple username/password or certificate authentication.

AWS EKS Uses custom built script/wrapper to query IAM user/role for example and i suspect kubeclient currently does not handle it.

[midN]

So it looks like you guys added google wrapper for the kubeclient to auth against google properly.
AWS uses custom exec command to get the script.

I'll do a PR if our team decides to go forward with kubernetes-deploy and add support for EKS

Easily fixable locally with 3 extra lines to google friendly helper

        elsif user.dig('exec')
          args = [user.dig('exec', 'command')] + user.dig('exec','args')
          out, err, st = Open3.capture3(*args)
          { bearer_token: JSON.parse(out)['status']['token'] }

[KnVerey]

Thanks for the bug report, @midN . Note that there's also an active PR for EKS support on the kubeclient side, which we'll be eager to pick up in kubernetes-deploy once it is released: ManageIQ/kubeclient#363.

[midN]

@KnVerey Nice! i'll not do a PR then, will wait for kubeclient to be merged and meanwhile use our fork with a small addition to make it usable for us :)

[benlangfeld]

What is the status of this issue? An updated version of kubeclient with the support for EKS which @KnVerey mentioned was released in December and kubernetes-deploy includes it since #418 (currently unreleased). Is that sufficient to resolve this issue?

[KnVerey]

@benlangfeld as far as we know there's nothing else to do, but I don't have an easy way to test it. I'm going to close this issue now that the change has been released though.

@midN we've just released version 0.25.0, which contains the kubeclient bump. When you have a chance, can you please try that out and reopen this if it didn't fix the problem for you as expected?

[lucasmazza]

looks like we are facing this kubectl vs kubeclient issue with v0.25.0 + google cloud. Kubernetes-deploy works as expected but any kubernetes-run calls fail with:

[INFO][2019-03-07 14:57:42 -0300]
[INFO][2019-03-07 14:57:42 -0300]	-------------------------------------Phase 1: Initializing task-------------------------------------
[INFO][2019-03-07 14:57:42 -0300]	Validating configuration
Traceback (most recent call last):
	12: from /Users/lucas/.rbenv/versions/2.5.0/bin/kubernetes-run:23:in `<main>'
	11: from /Users/lucas/.rbenv/versions/2.5.0/bin/kubernetes-run:23:in `load'
	10: from /Users/lucas/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/kubernetes-deploy-0.25.0/exe/kubernetes-run:37:in `<top (required)>'
	 9: from /Users/lucas/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/kubernetes-deploy-0.25.0/lib/kubernetes-deploy/runner_task.rb:23:in `run'
	 8: from /Users/lucas/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/kubernetes-deploy-0.25.0/lib/kubernetes-deploy/runner_task.rb:34:in `run!'
	 7: from /Users/lucas/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/kubernetes-deploy-0.25.0/lib/kubernetes-deploy/runner_task.rb:123:in `validate_configuration'
	 6: from /Users/lucas/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/kubernetes-deploy-0.25.0/lib/kubernetes-deploy/runner_task.rb:202:in `kubeclient'
	 5: from /Users/lucas/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/kubernetes-deploy-0.25.0/lib/kubernetes-deploy/kubeclient_builder.rb:17:in `build_v1_kubeclient'
	 4: from /Users/lucas/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/kubernetes-deploy-0.25.0/lib/kubernetes-deploy/kubeclient_builder.rb:94:in `_build_kubeclient'
	 3: from /Users/lucas/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/kubeclient-4.3.0/lib/kubeclient/config.rb:47:in `context'
	 2: from /Users/lucas/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/kubernetes-deploy-0.25.0/lib/kubernetes-deploy/kubeclient_builder/kube_config.rb:14:in `fetch_user_auth_options'
	 1: from /Users/lucas/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/kubeclient-4.3.0/lib/kubeclient/google_application_default_credentials.rb:20:in `token'
/Users/lucas/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/googleauth-0.6.7/lib/googleauth/application_default.rb:64:in `get_application_default': Could not load the default credentials. Browse to (RuntimeError)
https://developers.google.com/accounts/docs/application-default-credentials
for more information

[KnVerey]

@lucasmazza are you sure GOOGLE_APPLICATION_CREDENTIALS is properly set in your environment? That looks like the error you'd get if it genuinely was not properly configured. I know kubernetes-run works with GCP in general because Shopify uses it with GCP clusters.

[lucasmazza]

@KnVerey we aren't using GOOGLE_APPLICATION_CREDENTIALS on our setup, but our deployments are working fine, while kubernetes-run doesn't.

I'm far from an expert on the multiple ways kubectl/kubeclient can authenticate with my google account but I wonder if would be possible to get kubernetes-run working with the same authentication method that kubernetes-deploy uses?

[dturn]

@lucasmazza I'm surprised to hear that kubernetes-deploy is working but not kubernetes-run . They both use the same code from kubeclient_builder.rb. I guess it's possible that kubernetes-deploy makes a call to something (kubectl?) that set's up something that for the ruby library.

Without more information I'm not sure how to help. I'm also don't know what extra information would help us here.

Everything above is wrong, please ignore.

It actually looks like kubernetes-deploy uses kubectl while runner and restart use kubeclient.

[timothysmith0609]

we aren't using GOOGLE_APPLICATION_CREDENTIALS on our setup, but our deployments are working fine, while kubernetes-run doesn't.

@lucasmazza Where are your credentials stored, if not in the env var? Your failure is happening inside the googleauth gem, which tries the env var, well-known path, and system default (in that order), raising the error you observe if none of them work/exist.

[lucasmazza]

@timothysmith0609 sorry for the delayed response, but I was part time digging into this. We are using gcloud container clusters get-credentials to authenticate against our clusters, which sets the auth provider to be gcp but uses a different approach to do the auth: it reads the token from ~/.kube/config and/or shells out to gcloud config config-helper to get a new one. The comment on ManageIQ/kubeclient#210 (comment) explains it.

kubectl itself has support for this, so that is why our deploys are working but not runner/restart. Adding support for this on kubeclient seems to be a good way to fix this 🤔

[benlangfeld]

@lucasmazza That sounds like #456

[lucasmazza]

@benlangfeld it's a possible issue that tokens might not be refreshed when they might need to be, but on our case kubeclient is not authenticating as expected, regardless of the token freshness.