Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Google Cloud: Error 403: Request had insufficient authentication scopes #1020

Closed
AndrewDryga opened this issue May 10, 2019 · 1 comment

Comments

Projects
None yet
1 participant
@AndrewDryga
Copy link

commented May 10, 2019

We are getting errors on a newly created GKE cluster (same configuration works on older clusters):

time="2019-05-10T17:23:13Z" level=info msg="config: {Master: KubeConfig: RequestTimeout:30s IstioIngressGatewayServices:[istio-system/istio-ingressgateway] Sources:[ingress] Namespace: AnnotationFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false Compatibility: PublishInternal:false PublishHostIP:false ConnectorSourceServer:localhost:8080 Provider:google GoogleProject: DomainFilter:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType: AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: CloudflareProxied:false CloudflareZonesPerPage:50 RcodezeroTXTEncrypt:false InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:default TXTPrefix: Interval:1m0s Once:false DryRun:false LogFormat:text MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false}"
ERROR: logging before flag.Parse: W0510 17:23:13.197232       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2019-05-10T17:23:13Z" level=info msg="Created Kubernetes client https://10.4.0.1:443"
time="2019-05-10T17:23:14Z" level=info msg="Google project auto-detected: app-staging"
time="2019-05-10T17:23:14Z" level=error msg="googleapi: Error 403: Forbidden, forbidden"
ERROR: logging before flag.Parse: E0510 17:23:43.214165       1 round_trippers.go:291] CancelRequest not implemented
ERROR: logging before flag.Parse: E0510 17:23:43.214430       1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
ERROR: logging before flag.Parse: E0510 17:24:13.215004       1 round_trippers.go:291] CancelRequest not implemented
ERROR: logging before flag.Parse: E0510 17:24:13.215387       1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
time="2019-05-10T17:24:14Z" level=error msg="googleapi: Error 403: Forbidden, forbidden"
ERROR: logging before flag.Parse: E0510 17:24:43.216704       1 round_trippers.go:291] CancelRequest not implemented
ERROR: logging before flag.Parse: E0510 17:24:43.217118       1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)

This I have tried to do:

  1. Making sure correct service account and service account key is used (with DNS Administrator role) many many times.
  2. Adding both https://www.googleapis.com/auth/devstorage.read_write, https://www.googleapis.com/auth/cloud-platform oAuth scopes to GKE cluster.
  3. Explicitly and implicitly setting google project ID when external-dns is started.
  4. Recreating managed DNS zone.
  5. Making sure dns.googleapis.com is enabled on the project
  6. Updated external-dns to v0.5.13.

Some desperate things I did too:

  1. Adding project/owner permissions to service account used by external-dns;
  2. Adding DNS Administrator to Compute Engine and Kubernetes Engine default service account
  3. Recreating service account and furnishing new key (3 times).
  4. SSH'ed into a container and made sure the correct key is written in file system.
  5. Killing and re-creating external-dns pod/deployment.
  6. Disabling and then re-enabling dns.googleapis.com api.

dns.googleapis.com api metrics show 100% error rate on this call: cloud.dns.api.v1.ManagedZonesService.List.

If I invalidate service account key I get: time="2019-05-10T17:43:57Z" level=error msg="Get https://www.googleapis.com/dns/v1/projects/hammer-staging/managedZones?alt=json&prettyPrint=false: oauth2: cannot fetch token: 400 Bad Request\nResponse: {\n \"error\": \"invalid_grant\",\n \"error_description\": \"Invalid JWT Signature.\"\n}", when remove service account I get time="2019-05-10T17:47:18Z" level=error msg="googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.\nMore details:\nReason: authError, Message: Invalid Credentials\n" that tells that request works to some point but then fails.

Maybe you have any other ideas that I should try?

@AndrewDryga

This comment has been minimized.

Copy link
Author

commented May 10, 2019

Solved!

I'm laughing. I deleted cloud-dns service account and recreated it as cloud-dns2 with exactly same steps, furnished key for it and it works! Then I did it reverse changes, deleted cloud-dns2 and created a new cloud-dns and it stopped working again. Then I did one more round and created cloud-dns2 and it works again.

The problem was in the name (for whatever reason).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.