Resources can't sync except when proxied

[solidDoWant]

Have a really oddball problem. Running coredns outside of a k3s cluster, with k8s_gateway and it's kubeconfig pointing at the k3s cluster. Here's my config:

Corefile:

(default_config) {
  bind lo0 vtnet1 vtnet1_vlan100 vtnet1_vlan300 vtnet1_vlan400 vtnet1_vlan500
  root /usr/local/etc/coredns
  errors
  debug
  log
  prometheus :9153
}

import ./config.d/*

. {
  import default_config

  import ./root_config.d/*

  auto {
    directory ./zones (.*) {1}
  }

  forward . tls://1.1.1.1 tls://1.0.0.1 {
    tls_servername cloudflare-dns.com
  }
}

k8s_gateway config:

k8s_gateway echozulu.games {
    ttl 30
    kubeconfig /usr/local/etc/coredns/kubeconfig
    fallthrough
}

kubeconfig (secrets redacted):

---
apiVersion: v1
kind: Config
clusters:
  - name: home
    cluster:
      insecure-skip-tls-verify: true
#      proxy-url: "https://10.3.2.4:8080"
      server: "https://10.1.1.1:6443"
contexts:
  - name: home
    context:
      cluster: home
      user: home-dns
users:
  - name: home-dns
    user:
      token: "ey......"
current-context: home

With the config above, k8s_gateway fails to sync with the controller, logging [ERROR] plugin/errors: 2 traefik.echozulu.games. A: plugin/k8s_gateway: Could not sync required resources. Full log:

[WARNING] An external plugin (/go/pkg/mod/github.com/ori-edge/k8s_gateway@v0.1.8/setup.go line 56) is using the deprecated function Normalize. This will be removed in a future versions of CoreDNS. The plugin should be updated to use OriginsFromArgsOrServerBlock or NormalizeExact instead.
[INFO] plugin/k8s_gateway: Starting k8s_gateway controller
[INFO] plugin/auto: Inserting zone `10.in-addr.arpa.' from: /usr/local/etc/coredns/zones/10.in-addr.arpa
[INFO] plugin/auto: Inserting zone `echozulu.local.' from: /usr/local/etc/coredns/zones/echozulu.local
.:53 on 10.4.0.1
.:53 on ::1
.:53 on 127.0.0.1
.:53 on 10.1.0.1
.:53 on 10.0.0.1
.:53 on 10.2.0.1
.:53 on 10.3.0.1
CoreDNS-1.8.5
freebsd/amd64, go1.17.1, 8ddb631
[DEBUG] plugin/k8s_gateway: Computed Index Keys [traefik.echozulu.games traefik]
[INFO] 10.3.2.4:56958 - 52816 "A IN traefik.echozulu.games. udp 40 false 512" - - 0 0.00033038s
[ERROR] plugin/errors: 2 traefik.echozulu.games. A: plugin/k8s_gateway: Could not sync required resources

The really weird part is that when I setup a HTTPS proxy like mitmproxy on another computer, and uncomment the proxy line in the kubeconfig, everything works as expected:

[WARNING] An external plugin (/go/pkg/mod/github.com/ori-edge/k8s_gateway@v0.1.8/setup.go line 56) is using the deprecated function Normalize. This will be removed in a future versions of CoreDNS. The plugin should be updated to use OriginsFromArgsOrServerBlock or NormalizeExact instead.
[INFO] plugin/k8s_gateway: Starting k8s_gateway controller
[INFO] plugin/auto: Inserting zone `10.in-addr.arpa.' from: /usr/local/etc/coredns/zones/10.in-addr.arpa
[INFO] plugin/auto: Inserting zone `echozulu.local.' from: /usr/local/etc/coredns/zones/echozulu.local
.:53 on 10.2.0.1
.:53 on 10.3.0.1
.:53 on 10.4.0.1
.:53 on ::1
.:53 on 127.0.0.1
.:53 on 10.1.0.1
.:53 on 10.0.0.1
CoreDNS-1.8.5
freebsd/amd64, go1.17.1, 8ddb631
[DEBUG] plugin/k8s_gateway: Adding index traefik.networking for service traefik
[DEBUG] plugin/k8s_gateway: Adding index hajimari.echozulu.games for ingress hajimari
[DEBUG] plugin/k8s_gateway: Adding index traefik.echozulu.games for ingress traefik-dashboard
[DEBUG] plugin/k8s_gateway: Adding index rook.echozulu.games for ingress rook-ceph-mgr-dashboard
[INFO] plugin/k8s_gateway: Synced all required resources
[DEBUG] plugin/k8s_gateway: Computed Index Keys [traefik.echozulu.games traefik]
[DEBUG] plugin/k8s_gateway: Found 1 matching Ingress objects
[DEBUG] plugin/k8s_gateway: Computed response addresses [10.44.0.1]
[INFO] 10.3.2.4:59050 - 7981 "A IN traefik.echozulu.games. udp 40 false 512" NOERROR qr,aa,rd 78 0.0005981s

What am I doing wrong here?

[networkop]

this usually indicates some sort of connectivity problem between coredns and the API server. See similar issue #38
This plugin does not make its own connection and relies on client-go for all interactions with the API server.
Have you tried running anything else with this kubeconfig, e.g. upstream coredns?
Also try and capture the tcpdump of the packet exchange to see if there are any silent drops or connection resets.

[solidDoWant]

#38 was my first guess as to the cause of the issue but I can confirm that the kubeconfig works. Here's a test from another host where kubectl is installed:

kubectl --kubeconfig ../../working/kubeconfig.yaml get services -A
NAMESPACE       NAME                              TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                    AGE
calico-system   calico-kube-controllers-metrics   ClusterIP      10.43.230.49    <none>        9094/TCP                                                   6d15h
...

I also ran a packet capture and I can see that there are no silent drops/resets, and that there is bidirectional application data. Unfortunately as far as I know I cannot decrypt the TLS session after it's been captured even with the private key here due to the kube API using elliptic curve encryption. And when I tried proxying the traffic to see the requests being made, the problem went away... See here for a packet capture from tcpdump: k8s_gateway cap.zip.

Is there any additional logging that I can turn on?

[networkop]

you can also check the logs on the API server to see if there's anything obvious there.