ExternalDNS does not support K8s 1.22

[robbiezhang]

What would you like to be added:
Support k8s 1.22

Why is this needed:
k8s 1.22 stops to support a couple deprecated APIs, including the Ingress in extensions/v1beta1
https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122

I haven't verified but suspect that external-dns will stop working since it uses IngressInformer from k8s.io/client-go/informers/extensions/v1beta1

external-dns/source/ingress.go

Line 33 in c3a28ec

extinformers "k8s.io/client-go/informers/extensions/v1beta1"

[spirosoik]

It's true but still it's an RC

[maybe-sybr]

I can confirm that using a 1.22 k8s server breaks with the following output:

sh-5.1$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"archive", BuildDate:"2021-03-30T00:00:00Z", GoVersion:"go1.16", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.0-beta.0", GitCommit:"a3f24e8459465495738af1b9cc6c3db80696e3c1", GitTreeState:"clean", BuildDate:"2021-06-22T21:00:26Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
sh-5.1$ kubectl get pods -A | grep external-dns
kube-system   external-dns-5ffddbb8fc-l6ztj                     0/1     CrashLoopBackOff   3 (11s ago)   5m10s
sh-5.1$ kubectl logs -n kube-system   external-dns-5ffddbb8fc-l6ztj
time="2021-08-04T01:45:38Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s ContourLoadBalancerService:heptio-contour/contour GlooNamespace:gloo-system SkipperRouteGroupVersion:zalando.org/v1 Sources:[service ingress] Namespace: AnnotationFilter: LabelFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:coredns GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s DomainFilter:[] ExcludeDomains:[] RegexDomainFilter: RegexDomainExclusion: ZoneNameFilter:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType: AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: BluecatConfigFile:/etc/kubernetes/bluecat.json CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: AkamaiEdgercPath: AkamaiEdgercSection: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:upsert-only Registry:txt TXTOwnerID:default TXTPrefix: TXTSuffix: Interval:1m0s MinEventSyncInterval:5s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:debug TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136GSSTSIG:false RFC2136KerberosRealm: RFC2136KerberosUsername: RFC2136KerberosPassword: RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A CNAME] GoDaddyAPIKey: GoDaddySecretKey: GoDaddyTTL:0 GoDaddyOTE:false}"
time="2021-08-04T01:45:38Z" level=info msg="Instantiating new Kubernetes client"
time="2021-08-04T01:45:38Z" level=debug msg="apiServerURL: "
time="2021-08-04T01:45:38Z" level=debug msg="kubeConfig: "
time="2021-08-04T01:45:38Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2021-08-04T01:45:38Z" level=info msg="Created Kubernetes client https://10.0.0.1:443"
time="2021-08-04T01:46:39Z" level=fatal msg="failed to sync cache: timed out waiting for the condition"

Deployed using a YAML doc from `helm template` using the bitnami chart (click to expand):

---
# Source: external-dns/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
  namespace: kube-system
  labels: 
    app.kubernetes.io/name: external-dns
    helm.sh/chart: external-dns-5.2.3
    app.kubernetes.io/instance: external-dns
    app.kubernetes.io/managed-by: Helm
automountServiceAccountToken: true
---
# Source: external-dns/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
  labels: 
    app.kubernetes.io/name: external-dns
    helm.sh/chart: external-dns-5.2.3
    app.kubernetes.io/instance: external-dns
    app.kubernetes.io/managed-by: Helm
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - pods
      - nodes
      - endpoints
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - "networking.k8s.io"
      - getambassador.io
    resources:
      - ingresses
      - hosts
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - route.openshift.io
    resources:
      - routes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - networking.istio.io
    resources:
      - gateways
      - virtualservices
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - zalando.org
    resources:
      - routegroups
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - zalando.org
    resources:
      - routegroups/status
    verbs:
      - patch
      - update
  - apiGroups:
      - projectcontour.io
    resources:
      - httpproxies
    verbs:
      - get
      - watch
      - list
  - apiGroups:
      - gloo.solo.io
      - gateway.solo.io
    resources:
      - proxies
      - virtualservices
    verbs:
      - get
      - list
      - watch
---
# Source: external-dns/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns
  labels: 
    app.kubernetes.io/name: external-dns
    helm.sh/chart: external-dns-5.2.3
    app.kubernetes.io/instance: external-dns
    app.kubernetes.io/managed-by: Helm
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
  - kind: ServiceAccount
    name: external-dns
    namespace: kube-system
---
# Source: external-dns/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: external-dns
  namespace: kube-system
  labels: 
    app.kubernetes.io/name: external-dns
    helm.sh/chart: external-dns-5.2.3
    app.kubernetes.io/instance: external-dns
    app.kubernetes.io/managed-by: Helm
spec:
  ports:
    - name: http
      port: 7979
      protocol: TCP
      targetPort: http
  selector: 
    app.kubernetes.io/name: external-dns
    app.kubernetes.io/instance: external-dns
  type: ClusterIP
---
# Source: external-dns/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
  namespace: kube-system
  labels: 
    app.kubernetes.io/name: external-dns
    helm.sh/chart: external-dns-5.2.3
    app.kubernetes.io/instance: external-dns
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels: 
      app.kubernetes.io/name: external-dns
      app.kubernetes.io/instance: external-dns
  template:
    metadata:
      labels: 
        app.kubernetes.io/name: external-dns
        helm.sh/chart: external-dns-5.2.3
        app.kubernetes.io/instance: external-dns
        app.kubernetes.io/managed-by: Helm
      annotations:
    spec:
      
      securityContext: 
        fsGroup: 1001
        runAsUser: 1001
      affinity:
        podAffinity:
          
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: external-dns
                    app.kubernetes.io/instance: external-dns
                namespaces:
                  - "kube-system"
                topologyKey: kubernetes.io/hostname
              weight: 1
        nodeAffinity:
          
      serviceAccountName: external-dns
      containers:
        - name: external-dns
          image: "docker.io/bitnami/external-dns:0.8.0-debian-10-r73"
          imagePullPolicy: "IfNotPresent"
          args:
            # Generic arguments
            - --log-level=debug
            - --log-format=text
            - --policy=upsert-only
            - --provider=coredns
            - --registry=txt
            - --interval=1m
            - --source=service
            - --source=ingress
          env:
            # CoreDNS environment variables
            - name: ETCD_URLS
              value: "http://10.4.2.251:2379"
          ports:
            - name: http
              containerPort: 7979
          readinessProbe:
            failureThreshold: 6
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          livenessProbe:
            failureThreshold: 2
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          resources:
            limits: {}
            requests: {}
          volumeMounts:
      volumes:

Given that the 1.22 release is imminent (ie. today) [0], it'd be great if this could be fixed. I'm also keeping an eye on some misbehaviours over in kubernetes/ingress-nginx and they've made the choice to drop support for < 1.19 in their 1.0.0 ingress controller release. I imagine a similar choice will need to be thought about here.

[0] https://github.com/kubernetes/sig-release/tree/master/releases/release-1.22

[maybe-sybr]

ping @seanmalloy @njuettner @Raffo, looks like you are active maintainers here.

There's potentially relevant previous discussion at #1931 which was closed since k3s was possibly removing stuff early. Not sure if that misbehaviour was related to this ingress informer issue though. A handful of other changes preparing for 1.22 have been merged in #2001, #2012 , #2120 . Any chance of getting this one sorted and cutting a 0.9 or 1.0 release?

[Raffo]

I am not sure that this is related to the new API version actually. The error in your example seems permission related 🤔

[scholzj]

@Raffo I run into the same problem as well with 0.8.0 (didn't try with 0.9.0 yet since the container image was not available). The same RBAC configuration seemed to work fine for me on 1.21 but has shown this error on 1.22.

[alainvanhoof]

After Upgrading a working 1.21.2 cluster to 1.22 external-dns started showing the above metioned error and stopped working. It looks like the source/ingress.go that is trying to get info from the resource is still requesting v1beta1 API while only v1 is available
( extinformers "k8s.io/client-go/informers/extensions/v1beta1" )

[maybe-sybr]

I am not sure that this is related to the new API version actually. The error in your example seems permission related thinking

Any tips on how I could get some further information to help diagnose the issue, @Raffo? I've got the source but have been a bti reluctant to start trying to build it with extra debugging output. The current usage of the v1beta1 ingress interface is definitely going to cause issues, but it's entirely possible there are other things getting in the way as well.

My current premise is that this code is to blame:

external-dns/source/ingress.go

Lines 99 to 104 in c88486e

	err = poll(time.Second, 60*time.Second, func() (bool, error) {
	return ingressInformer.Informer().HasSynced(), nil
	})
	if err != nil {
	return nil, fmt.Errorf("failed to sync cache: %v", err)
	}

where:

external-dns/source/ingress.go

Line 85 in c88486e

ingressInformer := informerFactory.Extensions().V1beta1().Ingresses()

However, that error message text is use by all source implementations (grep -r 'failed to sync cache' shows a bunch of files under source/). For reference, I'm using both the ingress and service sources for my external-dns deployment. That's the default for the bitnami chart and I'm not overriding it:

https://github.com/bitnami/charts/blob/34f27c7d8dc3fd19ef88f04159ab4b92381a78b5/bitnami/external-dns/values.yaml#L61-L67

[rjosephwright]

I did a quick fix for this at rjosephwright@52823b4. It's not perfect as it has introduced a failure in the tests for the gloo proxy source, which I didn't have time to troubleshoot. I did verify that I am not seeing the "failed to sync cache" issue with this change.

[Raffo]

@rjosephwright I am working on a general bump of dependencies and will look into this as well, thank you for providing more information. It will take a bit to be fixed (ExternalDNS isn't really my daily job 😅 ) so for now I recommend avoiding 1.22.

I will rename this issue and pin it for maximum visibility.

[agilgur5]

The error in your example seems permission related

For reference, this same error seems to get printed out when a CRD doesn't exist. #961 (comment) suggested this was the case when you add Istio sources but Istio CRDs don't exist and my team has been able to confirm this is the case. (mentioned this in #2100 (comment) as well)

So if that error is for either permissions or resource non-existence issues, that would make sense for this issue as well since Ingress under extensions/v1beta1 no longer exists in k8s v1.22

[mattmichal]

Related to this is lack of support for the ingressClassName field mentioned in #1792. The annotation that is used today for finding eligible ingress resources has been deprecated since 1.18.

[mstrohl]

Hello, thanks to @rjosephwright i built an image without tests because i've seen that my source (OVH) still working with all dependencies errors. thanks @Raffo for the time you spend because there are a lot of impacts. For those who can't wait to try an image that coul possibly work you can try in the repository smartdeploy/external-dns:0.0.1. It has been build from the @rjosephwright commit.

[masterkain]

@mstrohl I had some success with smartdeploy/external-dns:0.0.1 image,
I only have a slight spamming of

W0911 12:20:13.950956 1 transport.go:260] Unable to cancel request for *instrumented_http.Transport

but beside that I was able to update my records on cloudflare, thanks

[alainvanhoof]

Build the current PR #2281 which indeed works for 1.22.1 and PowerDNS but has the same warning
W0911 12:20:13.950956 1 transport.go:260] Unable to cancel request for *instrumented_http.Transport
as smartdeploy/external-dns:0.0.1

[maybe-sybr]

Thanks for getting #2281 merged, @Raffo and @andrewstuart

[mstrohl]

Updating pod with this image working well : docker pull smartdeploy/external-dns:latest (made from #2281 commit on master)

[ghost]

FYI for others in similar situation: Ingresses running *v1beta1 (including those currently supported <=1.21, but are deprecated) are no longer looked for by external-dns due to this backwards incompatible change.

I found this while spending some time troubleshooting an update of external-dns image to 0.10.0, and there was no notices, docs, or comments on the PR that it only supports v1.19+ compatible Ingresses from now on.

So in other words, the release including this change, v0.10.0+, has a MINIMUM required K8s version of v1.19 where Ingresses are used (including having to upgrade Ingresses accordingly to 1.19+ apiVersion) due to a requirement to use networking.k8s.io/v1 (which is only available v1.19+) instead of networking.k8s.io/v1beta1 which is supported <= v1.18 but not served as of v1.22.

[larivierec]

The minimum required k8s version wasn't bumped in docs?
It should probably be mentioned somewhere

[tim-sendible]

@tr-mdaize It's actually worse than this, if you want to declaratively create your ingresses, then this change doesn't work for 1.19+ anyway: see #2386