New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

spin-clouddriver gcloud command not found #2900

Closed
HirokiSakonju opened this Issue Jun 13, 2018 · 15 comments

Comments

Projects
None yet
5 participants
@HirokiSakonju
Copy link

HirokiSakonju commented Jun 13, 2018

Issue Summary:

I deploy spinnaker, but spin-clouddriver is Readiness probe failed.
I checked the log with kubectl logs pod-name-n spinnaker and an internal server error has occurred.

Cloud Provider(s):

Kubernetes

Environment:

Halyard:
on GCE(ubuntu-1604-xenial-v20180522)

$ hal -v
1.2.0-20180509113540

spinnaker:
on GKE(1.10.4-gke.0)

$ hal config version edit --version 1.7.6
$ hal config provider kubernetes account add spinnaker \
    --provider-version v2 \
    --context $(kubectl config current-context)

Feature Area:

clouddriver

Description:

The status of pod is below and the status of spin-clouddriver is not normal.

$ kubectl get pod -n spinnaker
NAME                                READY     STATUS    RESTARTS   AGE
spin-clouddriver-8585c8cc88-pnsqb   0/1       Running   0          23m
spin-deck-64c77858bc-j2bjf          1/1       Running   0          23m
spin-echo-5b778876dd-5kfzx          1/1       Running   0          23m
spin-front50-65c677f697-fv5xx       1/1       Running   0          23m
spin-gate-7655445d46-9jlb7          1/1       Running   0          23m
spin-orca-b88b864d6-c479t           1/1       Running   0          22m
spin-redis-6d49c9c5b9-x4vkw         1/1       Running   0          23m
spin-rosco-7d8cc5bb6f-5gwj6         1/1       Running   0          22m

Steps to Reproduce:

  1. install halyard on GCE(ubuntu-1604-xenial-v20180522)
  2. hal config provider kubernetes enable
  3.  --provider-version v2 \
     --context $(kubectl config current-context)
    
  4. hal config features edit --artifacts true
  5. hal config deploy edit --type distributed --account-name spinnaker
  6. storage edit
     --json-path $SERVICE_ACCOUNT_DEST \
     --bucket $BUCKET_NAME
    
  7. hal config version edit --version 1.7.6
  8. hal deploy apply

Additional Details:

2018-06-13 02:03:48.418 ERROR 1 --- [0.0-7002-exec-3] c.n.s.k.w.e.GenericExceptionHandlers     : Internal Server Error

java.lang.RuntimeException: com.netflix.spinnaker.clouddriver.kubernetes.v2.op.job.KubectlJobExecutor$KubectlException: Failed to read [namespace] from : Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=

	at com.netflix.spinnaker.clouddriver.kubernetes.v2.security.KubernetesV2Credentials.getDeclaredNamespaces(KubernetesV2Credentials.java:311) ~[clouddriver-kubernetes-2.22.0-SNAPSHOT.jar:2.22.0-SNAPSHOT]
	at com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesCredentials$getDeclaredNamespaces.call(Unknown Source) ~[na:na]
	at com.netflix.spinnaker.clouddriver.kubernetes.health.KubernetesHealthIndicator.checkHealth(KubernetesHealthIndicator.groovy:65) ~[clouddriver-kubernetes-2.22.0-SNAPSHOT.jar:2.22.0-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65) ~[spring-context-4.3.11.RELEASE.jar:4.3.11.RELEASE]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-4.3.11.RELEASE.jar:4.3.11.RELEASE]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_151]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_151]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_151]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_151]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]
Caused by: com.netflix.spinnaker.clouddriver.kubernetes.v2.op.job.KubectlJobExecutor$KubectlException: Failed to read [namespace] from : Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=

@lwander

This comment has been minimized.

Copy link
Member

lwander commented Jun 13, 2018

Does your pod have gcloud installed at that location? Also, does that pod have access to read resources from k8s using the IAM roles assigned to it? If not, you should use a service account: https://www.spinnaker.io/setup/install/providers/kubernetes-v2/#optional-create-a-kubernetes-service-account

@HirokiSakonju

This comment has been minimized.

Copy link
Author

HirokiSakonju commented Jun 14, 2018

@lwander Thanks for the reply!

I could not access the pod.
I have access to the pod by checking the document presented and the following issues.
helm/charts#5483

However, it was Readiness probe failed.

As you can see in the log, I checked for /usr/lib/google-cloud-sdk/bin/gcloud but none.

I confirmed the path of the command and it was below.

$ which gcloud
/opt/google-cloud-sdk/bin/gcloud
@lwander

This comment has been minimized.

Copy link
Member

lwander commented Jun 14, 2018

Yeah those paths look different -- if you follow the above linked steps to create a kubernetes service account & redeploy these errors should go away. Note, this is done outside of the pod, no access to the pod is needed.

@HirokiSakonju

This comment has been minimized.

Copy link
Author

HirokiSakonju commented Jun 14, 2018

I'm sorry I didn't make it clear enough.
I can access pod.
But since RBAC was enabled, it was not authorized to read the kubernetes resource from pod.
So I added permissions.

Once I added the permissions, I could read the resources of kubernetes as follows.

$ kubectl exec -it spin-clouddriver-6fcbf7cf7-s8b25 -n spinnaker kubectl get pod
NAME                               READY     STATUS    RESTARTS   AGE
spin-clouddriver-6fcbf7cf7-s8b25   0/1       Running   0          3m
spin-deck-64c77858bc-sqvhv         1/1       Running   0          3m
spin-echo-5b778876dd-bkx9x         1/1       Running   0          3m
spin-front50-65c677f697-mgk7r      1/1       Running   0          3m
spin-gate-7655445d46-zsx74         1/1       Running   0          3m
spin-orca-b88b864d6-mdmkx          1/1       Running   0          3m
spin-redis-6d49c9c5b9-c6wjj        1/1       Running   0          4m
spin-rosco-8687b9db56-knbdj        1/1       Running   0          3m

However, Readiness probe failed: is still ongoing.
After doing hal deploy clean, I executedhal deploy apply again, but it did not change.

@lwander

This comment has been minimized.

Copy link
Member

lwander commented Jun 14, 2018

This is because gcloud configured in your kubeconfig doesn't exist in the pod. If you either change the gcloud path, or use a service account, Spinnaker will be able to read your cluster.

@HirokiSakonju

This comment has been minimized.

Copy link
Author

HirokiSakonju commented Jun 16, 2018

I ran the hal config provider kubernetes account delete at halyard and performed hal config provider kubernetes account add again, the pod started up normally.
Thank you for supporting me!

$ kubectl get pod -n spinnaker
NAME                                READY     STATUS    RESTARTS   AGE
spin-clouddriver-5b7d5ffcd6-d6w4g   1/1       Running   0          8m
spin-deck-65c9f5dd96-6z2vk          1/1       Running   0          1d
spin-echo-5b778876dd-fhcmw          1/1       Running   0          1d
spin-front50-65c677f697-cp9nj       1/1       Running   0          1d
spin-gate-7655445d46-x7rwk          1/1       Running   0          1d
spin-orca-b88b864d6-w6ps2           1/1       Running   0          1d
spin-redis-6d49c9c5b9-9wq7n         1/1       Running   0          1d
spin-rosco-54dc7f49f6-qp7xn         1/1       Running   0          8m
@HirokiSakonju

This comment has been minimized.

Copy link
Author

HirokiSakonju commented Jun 18, 2018

Re-adding provider kubernetes account prevented errors from occurring, but after about 30 minutes the same error ( Readiness probe failed) occurred again.

This time, I have two accounts registered.

  • GKE Cluster running spinnaker
  • External GKE Cluster where spinnaker is not running

After adding a GKE Cluster where spinnaker is not running, it works for a while for a while, but Readiness probe failed occurs after 30 minutes.

As I mentioned earlier, re-registering an external GKE Cluster account will work for a while, but it will not be usable.

@hammett

This comment has been minimized.

Copy link

hammett commented Jun 20, 2018

I'm getting the exact same issue, but only on the spin-gate pod.

Readiness probe failed: wget: can't connect to remote host (127.0.0.1): Connection refused

However, looking at the logs and even if I connect (hal deploy connect) and access the endpoints, it works.

I've tried hal deploy clean and then redeploy, no changes.

image: gcr.io/spinnaker-marketplace/gate:0.12.0-20180521101558

curl -v http://localhost:8084/health gives me a {status: "UP"}. Anything I'm missing?

@hammett

This comment has been minimized.

Copy link

hammett commented Jun 20, 2018

Well, I've deleted the probe from the yaml so I could continue. All is "good"

@HirokiSakonju

This comment has been minimized.

Copy link
Author

HirokiSakonju commented Jul 22, 2018

By upgrading to the following, clouddriver started to run.

$ hal -v
1.5.0-20180713143726
$ hal config version
+ Get current deployment
  Success
+ Get Spinnaker version
  Success
1.8.1

Although it was operating normally for about 30 minutes, CLUSTERS and LOAD BALANCERS were not displayed.
I checked the log of clouddriver and found that an error occurred on the way.

2018-07-22 11:04:28.362  INFO 1 --- [ecutionAction-2] s.c.k.v.c.a.KubernetesCacheDataConverter : external-cluster/KubernetesUnregisteredCustomResourceCachingAgent[1/1]: grouping Policy has 1 entries and 3 relationships
2018-07-22 11:04:28.362  INFO 1 --- [ecutionAction-2] s.c.k.v.c.a.KubernetesCacheDataConverter : external-cluster/KubernetesUnregisteredCustomResourceCachingAgent[1/1]: grouping kubernetes has 1 entries and 3 relationships
2018-07-22 11:04:28.362  INFO 1 --- [ecutionAction-2] s.c.k.v.c.a.KubernetesCacheDataConverter : external-cluster/KubernetesUnregisteredCustomResourceCachingAgent[1/1]: grouping stdio has 1 entries and 3 relationships
2018-07-22 11:04:28.362  INFO 1 --- [ecutionAction-2] s.c.k.v.c.a.KubernetesCacheDataConverter : external-cluster/KubernetesUnregisteredCustomResourceCachingAgent[1/1]: grouping kubernetesenv has 1 entries and 3 relationships
2018-07-22 11:04:28.362  INFO 1 --- [ecutionAction-2] s.c.k.v.c.a.KubernetesCacheDataConverter : external-cluster/KubernetesUnregisteredCustomResourceCachingAgent[1/1]: grouping MeshPolicy has 1 entries and 2 relationships
2018-07-22 11:04:28.363  INFO 1 --- [ecutionAction-2] s.c.k.v.c.a.KubernetesCacheDataConverter : external-cluster/KubernetesUnregisteredCustomResourceCachingAgent[1/1]: grouping rule has 5 entries and 15 relationships
2018-07-22 11:04:28.363  INFO 1 --- [ecutionAction-2] s.c.k.v.c.a.KubernetesCacheDataConverter : external-cluster/KubernetesUnregisteredCustomResourceCachingAgent[1/1]: grouping DestinationRule has 4 entries and 12 relationships
2018-07-22 11:04:28.363  INFO 1 --- [ecutionAction-2] s.c.k.v.c.a.KubernetesCacheDataConverter : external-cluster/KubernetesUnregisteredCustomResourceCachingAgent[1/1]: grouping artifact has 1 entries and 24 relationships
2018-07-22 11:04:28.363  INFO 1 --- [ecutionAction-2] s.c.k.v.c.a.KubernetesCacheDataConverter : external-cluster/KubernetesUnregisteredCustomResourceCachingAgent[1/1]: grouping attributemanifest has 2 entries and 6 relationships
2018-07-22 11:04:28.363  INFO 1 --- [ecutionAction-2] s.c.k.v.c.a.KubernetesCacheDataConverter : external-cluster/KubernetesUnregisteredCustomResourceCachingAgent[1/1]: grouping metric has 6 entries and 18 relationships
2018-07-22 11:04:28.363  INFO 1 --- [ecutionAction-2] s.c.k.v.c.a.KubernetesCacheDataConverter : external-cluster/KubernetesUnregisteredCustomResourceCachingAgent[1/1]: grouping logentry has 1 entries and 3 relationships
2018-07-22 11:04:28.363  INFO 1 --- [ecutionAction-2] s.c.k.v.c.a.KubernetesCacheDataConverter : external-cluster/KubernetesUnregisteredCustomResourceCachingAgent[1/1]: grouping prometheus has 1 entries and 3 relationships
2018-07-22 11:04:28.363  INFO 1 --- [ecutionAction-2] s.c.k.v.c.a.KubernetesCacheDataConverter : external-cluster/KubernetesUnregisteredCustomResourceCachingAgent[1/1]: grouping applications has 20 entries and 24 relationships
2018-07-22 11:04:28.388  INFO 1 --- [ecutionAction-2] c.n.s.c.cache.LoggingInstrumentation     : kubernetes:external-cluster/KubernetesUnregisteredCustomResourceCachingAgent[1/1] completed in 14.739s
2018-07-22 11:04:32.649  INFO 1 --- [ecutionAction-2] c.n.s.c.k.v.c.a.KubernetesV2CachingAgent : spinnaker/KubernetesNamespaceCachingAgent[1/1] is starting
2018-07-22 11:04:34.052  INFO 1 --- [ecutionAction-2] s.c.k.v.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesNamespaceCachingAgent[1/1]: grouping artifact has 1 entries and 4 relationships
2018-07-22 11:04:34.052  INFO 1 --- [ecutionAction-2] s.c.k.v.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesNamespaceCachingAgent[1/1]: grouping namespace has 4 entries and 8 relationships
2018-07-22 11:04:34.052  INFO 1 --- [ecutionAction-2] s.c.k.v.c.a.KubernetesCacheDataConverter : spinnaker/KubernetesNamespaceCachingAgent[1/1]: grouping applications has 3 entries and 4 relationships
2018-07-22 11:04:34.055  INFO 1 --- [ecutionAction-2] c.n.s.c.cache.LoggingInstrumentation     : kubernetes:spinnaker/KubernetesNamespaceCachingAgent[1/1] completed in 1.406s
2018-07-22 11:04:34.648  INFO 1 --- [ecutionAction-2] k.v.c.a.KubernetesV2OnDemandCachingAgent : spinnaker/KubernetesUnregisteredCustomResourceCachingAgent[1/1] is starting
2018-07-22 11:04:38.158  INFO 1 --- [ecutionAction-2] c.n.s.c.cache.LoggingInstrumentation     : kubernetes:spinnaker/KubernetesUnregisteredCustomResourceCachingAgent[1/1] completed in 3.509s
2018-07-22 11:04:38.649  INFO 1 --- [ecutionAction-2] k.v.c.a.KubernetesV2OnDemandCachingAgent : external-cluster/KubernetesCoreCachingAgent[1/1] is starting
2018-07-22 11:04:39.354  WARN 1 --- [ecutionAction-2] c.n.s.c.k.v.c.a.KubernetesV2CachingAgent : Failed to read kind [clusterRole, clusterRoleBinding, configMap, controllerRevision, customResourceDefinition, cronJob, daemonSet, deployment, event, horizontalpodautoscaler, ingress, job, pod, replicaSet, role, roleBinding, namespace, networkPolicy, persistentVolume, persistentVolumeClaim, secret, service, serviceAccount, statefulSet] from namespace default: Failed to read [clusterRole, clusterRoleBinding, configMap, controllerRevision, customResourceDefinition, cronJob, daemonSet, deployment, event, horizontalpodautoscaler, ingress, job, pod, replicaSet, role, roleBinding, namespace, networkPolicy, persistentVolume, persistentVolumeClaim, secret, service, serviceAccount, statefulSet] from default: Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=
Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=

2018-07-22 11:04:41.356  WARN 1 --- [ecutionAction-3] c.n.s.c.cache.LoggingInstrumentation     : kubernetes:external-cluster/KubernetesNamespaceCachingAgent[1/1] completed with one or more failures

com.netflix.spinnaker.clouddriver.kubernetes.v2.op.job.KubectlJobExecutor$KubectlException: Failed to read [namespace] from : Unable to connect to the server: error executing access token command "/usr/lib/google-cloud-sdk/bin/gcloud config config-helper --format=json": err=fork/exec /usr/lib/google-cloud-sdk/bin/gcloud: no such file or directory output= stderr=

	at com.netflix.spinnaker.clouddriver.kubernetes.v2.op.job.KubectlJobExecutor.list(KubectlJobExecutor.java:352) ~[clouddriver-kubernetes-2.54.0-SNAPSHOT.jar:2.54.0-SNAPSHOT]
	at com.netflix.spinnaker.clouddriver.kubernetes.v2.security.KubernetesV2Credentials.lambda$list$3(KubernetesV2Credentials.java:348) ~[clouddriver-kubernetes-2.54.0-SNAPSHOT.jar:2.54.0-SNAPSHOT]
	at com.netflix.spinnaker.clouddriver.kubernetes.v2.security.KubernetesV2Credentials.runAndRecordMetrics(KubernetesV2Credentials.java:406) ~[clouddriver-kubernetes-2.54.0-SNAPSHOT.jar:2.54.0-SNAPSHOT]
	at com.netflix.spinnaker.clouddriver.kubernetes.v2.security.KubernetesV2Credentials.runAndRecordMetrics(KubernetesV2Credentials.java:397) ~[clouddriver-kubernetes-2.54.0-SNAPSHOT.jar:2.54.0-SNAPSHOT]
	at com.netflix.spinnaker.clouddriver.kubernetes.v2.security.KubernetesV2Credentials.list(KubernetesV2Credentials.java:348) ~[clouddriver-kubernetes-2.54.0-SNAPSHOT.jar:2.54.0-SNAPSHOT]
	at com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.agent.KubernetesNamespaceCachingAgent.loadPrimaryResourceList(KubernetesNamespaceCachingAgent.java:61) ~[clouddriver-kubernetes-2.54.0-SNAPSHOT.jar:2.54.0-SNAPSHOT]
	at com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.agent.KubernetesV2CachingAgent.loadData(KubernetesV2CachingAgent.java:121) ~[clouddriver-kubernetes-2.54.0-SNAPSHOT.jar:2.54.0-SNAPSHOT]
	at com.netflix.spinnaker.cats.agent.CachingAgent$CacheExecution.executeAgentWithoutStore(CachingAgent.java:80) ~[cats-core-2.54.0-SNAPSHOT.jar:2.54.0-SNAPSHOT]
	at com.netflix.spinnaker.cats.agent.CachingAgent$CacheExecution.executeAgent(CachingAgent.java:73) ~[cats-core-2.54.0-SNAPSHOT.jar:2.54.0-SNAPSHOT]
	at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler$AgentExecutionAction.execute(ClusteredAgentScheduler.java:299) ~[cats-redis-2.54.0-SNAPSHOT.jar:2.54.0-SNAPSHOT]
	at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler$AgentJob.run(ClusteredAgentScheduler.java:273) ~[cats-redis-2.54.0-SNAPSHOT.jar:2.54.0-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_171]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
@lwander

This comment has been minimized.

Copy link
Member

lwander commented Jul 23, 2018

gcloud is installed to a different path than /usr/lib/google-cloud-sdk/bin/gcloud on the clouddriver container. Just change the path from /usr/lib/google-cloud-sdk/bin/gcloud to gcloud in your ~/.kube/config and you'll be good to go.

@spinnakerbot

This comment has been minimized.

Copy link

spinnakerbot commented Sep 6, 2018

This issue hasn't been updated in 45 days, so we are tagging it as 'stale'. If you want to remove this label, comment:

@spinnakerbot remove-label stale

@spinnakerbot spinnakerbot added the stale label Sep 6, 2018

@spinnakerbot

This comment has been minimized.

Copy link

spinnakerbot commented Oct 21, 2018

This issue is tagged as 'stale' and hasn't been updated in 45 days, so we are tagging it as 'to-be-closed'. It will be closed in 45 days unless updates are made. If you want to remove this label, comment:

@spinnakerbot remove-label to-be-closed

@somejavadev

This comment has been minimized.

Copy link

somejavadev commented Nov 10, 2018

Ran into the exact same issue. The gcloud command: gcloud beta container clusters get-credentials spinnaker --region europe-west3 --project xxx causes the gcloud path to change within the ~/.kube/config file. To work around this I changed the path of the gcloud binary to just gcloud as @lwander mentioned.

@spinnakerbot

This comment has been minimized.

Copy link

spinnakerbot commented Dec 25, 2018

This issue is tagged as 'to-be-closed' and hasn't been updated in 45 days, so we are closing it. You can always reopen this issue if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment