airgapped: unable to reach the dashboard through public-url

[NohaIhab]

Bug Description

Airgapped Charmed Kubeflow

Getting a 403 Forbidden when trying to curl the public-url. Looking at the logs in oidc-gatekeeper:

level=error msg="OIDC provider setup failed, retrying in 10 seconds: Get http://10.64.140.43.nip.io/dex/.well-known/openid-configuration: dial tcp: lookup 10.64.140.43.nip.io on 10.152.183.10:53: server misbehaving"

oidc is not able to reach dex through the public-url.

However, when setting the public-url to the in-cluster service IP of dex, the dashboard is reachable. For example by doing:

juju config dex-auth public-url=http://10.152.183.208:5556 # the dex ClusterIP service
juju config oidc-gatekeeper public-url=http://10.152.183.208:5556 # the dex ClusterIP service

I can reach the dashboard with:

curl -v http://10.64.140.43.nip.io
*   Trying 10.64.140.43:80...
* Connected to 10.64.140.43.nip.io (10.64.140.43) port 80 (#0)
> GET / HTTP/1.1
> Host: 10.64.140.43.nip.io
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< content-type: text/html; charset=utf-8
< location: http://10.152.183.208:5556/dex/auth?client_id=authservice-oidc&redirect_uri=%2Fauthservice%2Foidc%2Fcallback&response_type=code&scope=openid+profile+email+groups&state=MTY5NDc4NzU2NXxOd3dBTkUxRlEwbEpWVWhYV1U1S1FUWldVMHhJU1ZrM1JGaE1VRFV5TWtkU1JVZE9NMDFNV1ZGQk4weEZTVlJVVWxKUU5GaFNUVUU9fO55va4u97gB9qMixE8VS5LsjZBxPsD79Jl5Nw21n2yN
< set-cookie: oidc_state_csrf=MTY5NDc4NzU2NXxOd3dBTkUxRlEwbEpWVWhYV1U1S1FUWldVMHhJU1ZrM1JGaE1VRFV5TWtkU1JVZE9NMDFNV1ZGQk4weEZTVlJVVWxKUU5GaFNUVUU9fO55va4u97gB9qMixE8VS5LsjZBxPsD79Jl5Nw21n2yN; Path=/; Expires=Fri, 06 Mar 2054 15:12:54 GMT; Max-Age=1200000000000
< date: Fri, 15 Sep 2023 14:19:25 GMT
< content-length: 367
< x-envoy-upstream-service-time: 9
< server: istio-envoy
< 
<a href="http://10.152.183.208:5556/dex/auth?client_id=authservice-oidc&amp;redirect_uri=%2Fauthservice%2Foidc%2Fcallback&amp;response_type=code&amp;scope=openid+profile+email+groups&amp;state=MTY5NDc4NzU2NXxOd3dBTkUxRlEwbEpWVWhYV1U1S1FUWldVMHhJU1ZrM1JGaE1VRFV5TWtkU1JVZE9NMDFNV1ZGQk4weEZTVlJVVWxKUU5GaFNUVUU9fO55va4u97gB9qMixE8VS5LsjZBxPsD79Jl5Nw21n2yN">Found</a>.

* Connection #0 to host 10.64.140.43.nip.io left intact

Based on the above, the issue is potentially with dns resolution. To confirm this, we can exec into a pod in the cluster and try to curl the url.

To Reproduce

1. follow the instructions in Extend tests and automation scripts for airgapped installations #682
2. deploy the air-gapped bundle in https://github.com/canonical/bundle-kubeflow/tree/main/releases/latest/edge
3. configure access to the dashboard:

juju config dex-auth public-url=http://10.64.140.43.nip.io
http://10.64.140.43.nip.io
juju config dex-auth static-username=admin
juju config dex-auth static-password=admin

4. try to curl the public-url

curl -v http://10.64.140.43.nip.io

Environment

microk8s 1.24/stable
juju 2.9/stable

Relevant Log Output

curl -v http://10.64.140.43.nip.io
*   Trying 10.64.140.43:80...
* Connected to 10.64.140.43.nip.io (10.64.140.43) port 80 (#0)
> GET / HTTP/1.1
> Host: 10.64.140.43.nip.io
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< date: Fri, 15 Sep 2023 14:13:23 GMT
< server: istio-envoy
< content-length: 0
< 
* Connection #0 to host 10.64.140.43.nip.io left intact

[NohaIhab]

from inside the oidc-authservice container, the url is unreachable with:

wget 10.64.140.43.nip.io/dex/.well-known/openid-configuration
wget: bad address '10.64.140.43.nip.io

looking into the microk8s coredns container, I can see timeouts to resolve the url:

[ERROR] plugin/errors: 2 10.64.140.43.nip.io.lxd. AAAA: read udp 10.1.205.129:38375->8.8.4.4:53: i/o timeout

it's clear that it is indeed a dns resolution problem, microk8s is using the google dns by default 8.8.4.4, seen in the log above. This dns is unreachable from air-gapped.

[NohaIhab]

The DNS add-on of microk8s should be configured to the host's local nameserver, this is possible by running:

microk8s enable dns:$(resolvectl status | grep "Current DNS Server" | awk '{print $NF}')

Tested and verified this fix.

We should add a note about this in our air-gapped documentation: to correctly set the dns in your kubernetes cluster.
cc: @ColmBhandal

[kimwnasptd]

Nice finding! I'll include this to my PR to we can close this issue. But before doing so, @NohaIhab does the above change make CoreDNS to only use the new DNS server or it will still use 8.8.4.4, plus the new server?

If it replaces 8.8.4.4 then could you see if this also resolves canonical/knative-operators#151?

[NohaIhab]

@kimwnasptd it will replace 8.8.4.4., sure ill test that too.

[ColmBhandal]

@NohaIhab I updated the doc per your note: https://discourse.charmhub.io/t/install-charmed-kubeflow-in-an-air-gapped-environment/11733#heading--microk8s-dns

[kimwnasptd]

Closing this, since we now configure the dns accordingly in #682