How to configure Galapagos to operate in the Google Kubernetes Engine with Load Balancer and TLS?

[Rob3681]

I'm currently trying to configure Galapagos and Keycloak so that they can run in the Google Kubernetes Engine. Most of it is done for it. However, there are strange behaviors, probably due to the fact that a load balancer is running before the application on which TLS is terminated. This means that communication takes place between client and load balancer via HTTPS, between load balancer and Galapagos/Keycloak via HTTP.

For this I have added an ingress (IP address, certificate and other settings of course also exist):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: galapagos-ingress
  annotations:
    kubernetes.io/ingress.class: "gce
    kubernetes.io/ingress.global-static-ip-name: "my-own-ip"
    kubernetes.io/ingress.allow-http: "false"
    networking.gke.io/managed-certificates: "my-own-certificate"
spec:
  rules:
  - host: galapagos.testdomain8739.de
    http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: galapagos-service
            port: 
              number: 8080
  - host: keycloak.galapagos.testdomain8739.de
    http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: keycloak-service
            port: 
              number: 8080

In order not to be redirected to HTTP after a request to Galapagos, I added the following values in a .properties file, which also work:

server.forward-headers-strategy=NATIVE
server.tomcat.use-relative-redirects=true

However, there is still a problem. When I access https://galapagos.testdomain8739.de, I am redirected to https://galapagos.testdomain8739.de/app. However, there I get the following message:

Whitelabel Error Page
This application has no explicit mapping for /error, so you are seeing this as a fallback.

Mon Oct 11 15:06:47 GMT 2021
There was an unexpected error (type=Not Found, status=404).
No message available

The same page appears when I am redirected to https://galapagos.testdomain8739.de/sso/login. A log entry is not generated. But if I go to https://galapagos.testdomain8739.de/test, for example, I get the same page, but also a log entry: "WARN 1 --- [nio-8080-exec-4] o.s.web.servlet.PageNotFound : No mapping for GET /test".

I don't get this error page when I navigate to https://galapagos.testdomain8739.de/actuator. There I get the real content.

What configuration do I have to make so that I can operate Galapagos behind a load balancer / reverse proxy on which TLS is terminated? Has anyone already had experience with it?

[albrechtflo-hg]

Hi @Rob3681,

Our very own production setup of Galapagos is quite similar - but it is not a GKE cluster, but a self-hosted Kubernetes Cluster. There, we have an Ingress terminating the TLS connection and forwarding the traffic to the Galapagos Service, which is the "load balancer" for the pods.

Do you use Galapagos 1.9.0 from DockerHub, or do you use a custom build? If the latter, how did you build it? From the errors, it more looks as if the complete Frontend part is missing from your deployment.

---

For reference, here are our K8S configs (from our 1.9.0 deployment; later deployments are now on Google Cloud Run, but a new official release will follow in the next two weeks):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: galapagos-test-ingress
spec:
  tls:
    - hosts:
        - galapagos-test.somewhere.internal.de
      secretName: tls-secret
  rules:
    - host: galapagos-test.somewhere.internal.de
      http:
        paths:
          - backend:
              serviceName: galapagos-test-service
              servicePort: 80

apiVersion: v1
kind: Service
metadata:
  name: galapagos-test-service
spec:
  selector:
    app: galapagos-test
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

apiVersion: apps/v1
kind: Deployment
metadata:
  name: galapagos-test-deployment
  labels:
    app: galapagos-test
spec:
  replicas: 2
  revisionHistoryLimit: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: galapagos-test
  template:
    spec:
      containers:
      - name: galapagos-test-container
        image: hermesgermany/galapagos:1.9.0
  ...

[Rob3681]

Oh, that would explain a lot. 🙈

I built a Docker image myself with mvn compile jib:dockerBuild. How can I include the frontend part if this is not done automatically?

[albrechtflo-hg]

Almost ;-) As stated in the README, you'll need the Maven package stage, so try using

mvn package jib:dockerBuild

(We use mvn package jib:build, together with jib.to.image and jib.to.tags, to directly deploy to our Docker registry)

[Rob3681]

It works. Thanks.

However, I have the next problem straight away. Are further special settings required for communication with the Keycloak instance? redirectUris and auth-server-url are correctly defined. However, I get the following error:

ERROR 1 --- [nio-8080-exec-8] o.k.a.rotation.AdapterTokenVerifier : Didn't find publicKey for kid: 92kgdHkaMEauMb9ELpidksjWp2jmZosaFsDXZzNtSPk

The public key can be found at https://keycloak.galapagos.testdomain8739.de/auth/realms/galapagos/protocol/openid-connect/certs. HTTP traffic is not permitted.

I then temporarily added the public key as realm-public-key. Then this error disappeared, but a new one popped up:

WARN 1 --- [nio-8080-exec-1] o.k.adapters.RequestAuthenticator : SSL is required to authenticate. Remote address XX.XX.XXX.XXX is secure: false, SSL required for: EXTERNAL .

In both cases, after logging in, I got the message "Kommunikation mit dem Backend schlug fehl." displayed in the frontend.

[albrechtflo-hg]

Hmm, Keycloak setup in a GKE cluster and e.g. behind Google Frontend can be tricky. For us, we needed to allow HTTP traffic on the Keycloak end, and ensure that the Google Frontend does the TLS termination (and communicates with Keycloak via HTTP only). We didn't manage to enable "Google Frontend to Keycloak Backend" Communication via HTTPS (which doesn't really add to security, after all). Only make sure that your public endpoint for Keycloak enforces HTTPS.

Also make sure that you have PROXY_ADDRESS_FORWARDING=true if you run Keycloak from a Docker image. There are some - partly outdated, partly bad - docs on the web on Keycloak behind a reverse proxy, which may be more or less applicable for your case.