GKE ingress with https load balancer and IAP/security policy enabled

[rllin-fathom]

I have an application that uses GKE Ingress for a load balancer. Recently GKE started supporting declaring IAP support via BackendConfig. I followed the documentation at [1] and [2]. However, now, GKE seems to hang while creating my Ingress.

Below is the yaml for my service, ingress and backendconfig.

kubectl -n randall-test-1 get svc,ing,backendconfig -o yaml

apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      beta.cloud.google.com/backend-config: '{"default": "airflow-backend-config"}'
      service.alpha.kubernetes.io/app-protocols: '{"web":"HTTPS"}'
    creationTimestamp: 2018-09-10T19:23:13Z
    name: airflow
    namespace: randall-test-1
    resourceVersion: "2155724"
    selfLink: /api/v1/namespaces/randall-test-1/services/airflow
    uid: X-X-X-X-X
  spec:
    clusterIP: X.X.X.X
    externalTrafficPolicy: Cluster
    ports:
    - name: web
      nodePort: 30099
      port: 8080
      protocol: TCP
      targetPort: web
    selector:
      app: airflow
    sessionAffinity: None
    type: NodePort
  status:
    loadBalancer: {}
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.allow-http: "false"
    creationTimestamp: 2018-09-10T19:23:13Z
    generation: 1
    name: airflow
    namespace: randall-test-1
    resourceVersion: "2155721"
    selfLink: /apis/extensions/v1beta1/namespaces/randall-test-1/ingresses/airflow
    uid: X-X-X-X-X
  spec:
    backend:
      serviceName: airflow
      servicePort: 8080
    tls:
    - secretName: tls
  status:
    loadBalancer: {}
- apiVersion: cloud.google.com/v1beta1
  kind: BackendConfig
  metadata:
    clusterName: ""
    creationTimestamp: 2018-09-10T19:23:13Z
    generation: 1
    name: airflow-backend-config
    namespace: randall-test-1
    resourceVersion: "2155728"
    selfLink: /apis/cloud.google.com/v1beta1/namespaces/randall-test-1/backendconfigs/airflow-backend-config
    uid: X-X-X-X-X
  spec:
    iap:
      enabled: true
      oauthclientCredentials:
        secretName: oauth2
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

The hang gives me no insight.

Name:             airflow
Namespace:        randall-test-1
Address:
Default backend:  airflow:8080 (X.X.X.X:8080)
TLS:
  tls terminates
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     *     airflow:8080 (X.X.X.X:8080)
Annotations:
Events:
  Type    Reason  Age   From                     Message
  ----    ------  ----  ----                     -------
  Normal  ADD     6m    loadbalancer-controller  randall-test-1/airflow

However, in GKE console, I just get Creating ingress as a status for > 20 mins with no resolution. I also check my Load Balancers in console and see nothing.

Any ideas what is happening or what else I can check?

I also tried to do this with just securityPolicy which is supposed to link the Load Balancer with a Cloud Armor policy. This also doesn't work with a similar hang.

[1] https://cloud.google.com/iap/docs/enabling-kubernetes-howto
[2] https://cloud.google.com/kubernetes-engine/docs/concepts/backendconfig

[MrHohn]

@rllin-fathom What is your GKE master version? Does creating ingress with the same setup but without BackendConfig work?

[rllin-fathom]

@MrHohn
my gke master version is 1.10.6-gke.2. And yes creating the same set up without the BackendConfig works

[MrHohn]

@rllin-fathom Given the situation, I think the best move would be looking at ingress controller's logs at the moment. Would you mind emailing me your cluster information {cluster_location, project_name, cluster_name} and I can take a look?

[rllin-fathom]

@MrHohn done, thanks!

[MrHohn]

Posting an update here, after inspecting the logs, it turned out ingress controller is getting permission denied (due to an internal bug on GCP) while updating IAP configuration on backend service, which leads to the hang of ingress creation.

For those who hit this same issue, a temporary workaround is to set [1] the default GKE service account (Kubernetes Engine Service Agent) as an Owner. This should eventually be revoked once that internal bug is fixed.

[1] https://cloud.google.com/iam/docs/granting-changing-revoking-access

[aviresonai]

We are seeing the same behavior (status remain "Creating ingress" in gke version 1.10.6-gke.2), yet we tried the suggested workaround and it did not work for us.

Our Ingress object contain a list of rules (with a backend for each ) - something like
spec:
rules:

• host:
  http:
  paths:
  • backend:
    serviceName:
    servicePort: www
    I tried to edit it and remove the backend, managed to save but service state remains "Creating ingress" forever

[MrHohn]

@aviresonai Did you enable IAP on Ingress via BackendConfig as well? Did creating ingress without IAP work?

[aviresonai]

I think the issue we are having is discussed at #471

[rramkumar1]

@rllin-fathom Closing this bug since the original issue looks to be resolved.

/close

[k8s-ci-robot]

@rramkumar1: Closing this issue.

In response to this:

@rllin-fathom Closing this bug since the original issue looks to be resolved.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.