Route53 txt-prefix does not create registry heritage records

[masterkain]

I had to make a root CNAME (ALIAS) co exists with a TXT record for external domain verification.

Given the TXT record was already taken by external-dns I tried to reach for the --txt-prefix flag.

What happened:

I deploy external dns with Helm, so I changed txtPrefix: k8s, deployed and verified that indeed the external dns image starts up with this new parameter.

I went on, deleted the current CNAME and TXT and waited for external dns to sync again.

external dns created the CNAME correctly, however there's no TXT record as I expected.

In all the existing entries I have this as TXT content:

"heritage=external-dns,external-dns/owner=my-kubernetes,external-dns/resource=ingress/prod/my-prod-blog-wordpress"

but those are not being created anymore.

With --txt-prefix

I wiped existing records before this test, there were no related records leftover in console.

│ time="2020-02-11T09:50:45Z" level=info msg="Instantiating new Kubernetes client"
│ time="2020-02-11T09:50:45Z" level=info msg="Using inCluster-config based on serviceaccount-token"
│ time="2020-02-11T09:50:45Z" level=info msg="Created Kubernetes client https://10.100.0.1:443"
│ time="2020-02-11T09:50:47Z" level=info msg="All records are already up to date, there are no changes for the matching hosted zones"
│ time="2020-02-11T09:51:47Z" level=info msg="Desired change: CREATE my.com A [Id: /hostedzone/Z1KM8PR0TJR5XX]"
│ time="2020-02-11T09:51:47Z" level=info msg="1 record(s) in zone my.com. [Id: /hostedzone/Z1KM8PR0TJR5XX] were successfully updated"

Without --txt-prefix

Wiped the record again (only CNAME from previous creation) and this time without specifying a --txt-prefix the registry record TXT gets created.

│ time="2020-02-11T10:00:46Z" level=info msg="Instantiating new Kubernetes client"
│ time="2020-02-11T10:00:46Z" level=info msg="Using inCluster-config based on serviceaccount-token"
│ time="2020-02-11T10:00:46Z" level=info msg="Created Kubernetes client https://10.100.0.1:443"
│ time="2020-02-11T10:00:48Z" level=info msg="All records are already up to date, there are no changes for the matching hosted zones"
│ time="2020-02-11T10:01:49Z" level=info msg="Desired change: CREATE my.com A [Id: /hostedzone/Z1KM8PR0TJR5XX]"
│ time="2020-02-11T10:01:49Z" level=info msg="Desired change: CREATE my.com TXT [Id: /hostedzone/Z1KM8PR0TJR5XX]"
│ time="2020-02-11T10:01:49Z" level=info msg="2 record(s) in zone my.com. [Id: /hostedzone/Z1KM8PR0TJR5XX] were successfully updated"

spec:
  releaseName: external-dns
  chart:
    repository: https://charts.bitnami.com/bitnami
    name: external-dns
    version: 2.16.1
  values:
    policy: upsert-only
    replicas: 1
    sources: [ingress]
    provider: aws
    aws:
      zoneType: public
      region: eu-west-1
      credentials:
        accessKey: xxx
        secretKey: xxx
    txtOwnerId: my-kubernetes
    # txtPrefix: k8s

[vutny]

I cannot reproduce this issue. Using --txt-prefix=prefix- parameter with ExternalDNS 0.6.0.

time="2020-04-13T16:36:50Z" level=info msg="Desired change: CREATE test.example.com CNAME [Id: /hostedzone/ZXXXXXXXXXXXXX]"
time="2020-04-13T16:36:50Z" level=info msg="Desired change: CREATE prefix-test.example.com TXT [Id: /hostedzone/ZXXXXXXXXXXXXX]"
time="2020-04-13T16:36:51Z" level=info msg="2 record(s) in zone example.com. [Id: /hostedzone/ZXXXXXXXXXXXXX] were successfully updated"

Got both CNAME and prefixed TXT record in Route53.

[jgreat]

I'm seeing the same behavior.

Bitnami helm chart: external-dns-3.2.1 app version 0.7.2

values.yaml

provider: aws
policy: sync
txtOwnerId: production-u1-a
txtPrefix: externaldns
domainFilters:
  - .com
aws:
  zoneType: public
  evaluateTargetHealth: "true"
sources:
  - ingress

Logs

time="2020-06-17T19:57:20Z" level=info msg="config: {Master: KubeConfig: RequestTimeout:30s IstioIngressGatewayServices:[] ContourLoadBalancerService:heptio-contour/contour SkipperRouteGroupVersion:zalando.org/v1 Sources:[ingress] Namespace: AnnotationFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:aws GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s DomainFilter:[.com] ExcludeDomains:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType:public AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] OVHEndpoint:ovh-eu PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:production-u1-a TXTPrefix:externaldns TXTSuffix: Interval:1m0s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s NS1Endpoint: NS1IgnoreSSL:false TransIPAccountName: TransIPPrivateKeyFile:}"
time="2020-06-17T19:57:20Z" level=info msg="Instantiating new Kubernetes client"
time="2020-06-17T19:57:20Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2020-06-17T19:57:20Z" level=info msg="Created Kubernetes client https://10.43.0.1:443"
time="2020-06-17T19:59:30Z" level=info msg="Desired change: CREATE .com A [Id: /hostedzone/ZX]"
time="2020-06-17T19:59:30Z" level=info msg="1 record(s) in zone .com. [Id: /hostedzone/ZX] were successfully updated"

[lossanarch]

Also seeing this on 0.7.3 using raw yaml install

relevant container config:

       - name: external-dns
          image: "asia.gcr.io/k8s-artifacts-prod/external-dns/external-dns:v0.7.3"
          args:
            - --log-level=info
            - --log-format=text
            - --policy=upsert-only
            - --provider=$(PROVIDER)
            - --txt-owner-id=$(TXT_OWNER_ID)
            - --txt-prefix=external-dns-
            - --registry=txt
            - --interval=1m
            - --source=service
            - --source=ingress
            - --aws-batch-change-size=1000

logs:

external-dns-7dccf489ff-h4tvp external-dns time="2020-08-11T06:24:27Z" level=info msg="Desired change: CREATE <redacted>.com.au A [Id: /hostedzone/<redacted>]"
external-dns-7dccf489ff-h4tvp external-dns time="2020-08-11T06:24:27Z" level=info msg="1 record(s) in zone <redacted>.com.au. [Id: /hostedzone/<redacted>] were successfully updated"

(note the lack of logs for creating the TXT record)

[lossanarch]

Ahh actually i believe this is more useful:

external-dns-6d95f8c8b5-mcph5 external-dns time="2020-08-11T06:37:47Z" level=debug msg="Adding <redacted>.com.au. to zone <redacted>.com.au. [Id: /hostedzone/<zoneid>]"
external-dns-6d95f8c8b5-mcph5 external-dns time="2020-08-11T06:37:47Z" level=debug msg="Skipping record {\n  Action: \"CREATE\",\n  ResourceRecordSet: {\n    Name: \"external-dns-<redacted>.com.au\",\n    ResourceRecords: [{\n        Value: \"\\\"heritage=external-dns,external-dns/owner=<ownerString>,external-dns/resource=ingress/<namespace>/<ingress>\\\"\"\n      }],\n    TTL: 300,\n    Type: \"TXT\"\n  }\n} because no hosted zone matching record DNS Name was detected"

Apologies for all the redaction, if you take all the domains as being the same it should make sense.

So it would appear that once the prefix is added it doesn't match the record it's signalling heritage for anymore. I'll try to take a look at the code tonight and figure out where this is happening.

[lossanarch]

Ok, I believe I've found the issue: without a trailing . in the txt-prefix value, attempts to attach the prefix to a root domain will fail.
So in my case, I'm using --txt-prefix=external-dns- and the resulting txt record for example.com is external-dns-example.com, which throws this error:

DEBU[0046] Skipping record {
  Action: "CREATE",
  ResourceRecordSet: {
    Name: "external-dns-example.com",
    ResourceRecords: [{
        Value: "\"heritage=external-dns,external-dns/owner=foobar\""
      }],
    TTL: 300,
    Type: "TXT"
  }
} because no hosted zone matching record DNS Name was detected

Which is perfectly reasonable, because I have no zone external-dns-example.com in my aws account.

I believe this will also be the issue for the previous two posts as well - both prefixes lack the trailing dot and at least the first is against a root domain. It looks to me like the domain has been stripped from the second example but my guess is it would also be a root.
I think a safe assumption would be that @masterkain 's example would result in k8smy.com.

This also explains why @vutny 's test could not reproduce the issue as it was not against example.com but test.example.com.

Changing to --txt-prefix=external-dns. produced the following:

DEBU[0047] Adding example.com.au. to zone example.com.au. [Id: /hostedzone/<zoneid>]
DEBU[0047] Adding external-dns.example.com.au. to zone example.com.au. [Id: /hostedzone/<zoneid>]
INFO[0047] Desired change: CREATE external-dns.example.com.au TXT [Id: /hostedzone/<zoneid>]
INFO[0047] Desired change: CREATE example.com.au A [Id: /hostedzone/<zoneid>]

So in summary, I think it's functioning as intended and either a documentation update is needed to note the trailing dot requirement or perhaps some validation of the flag value (or both).
It doesn't really make sense to accept a prefix without the trailing dot, as without it any root domain registry record would always fail.

[seanmalloy]

/kind documentation
/kind bug

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[seanmalloy]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nefelim4ag]

/reopen
Same with root domain on GoDaddy, but with TXT Postfix

[k8s-ci-robot]

@nefelim4ag: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen
Same with root domain on GoDaddy, but with TXT Postfix

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.