Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CoreDNS/SkyDNS provider #253

Merged
merged 8 commits into from
Jun 11, 2018
Merged

Conversation

istalker2
Copy link
Contributor

This PR adds support for CoreDNS through its etcd middleware.
Because the middleware is backward compatible with SkyDNS this
commit adds support for SkyDNS as well. In fact, new provider
is available under two names in CLI (coredns and skydns).

All interactions with middleware happen through etcd cluster,
whose location (URIs) is specified via --etcd CLI parameter
by default http://localhost:2379).

The provider translates CoreDNS/DkyDNS SRV records to
A/CNAME + optional TXT endpoints, when reading from etcd and
combines A/CNAME with TXT endpoints back into single SRV record
when writing it back.

Also:

  • adds github.com/coreos/etcd package to glide.yaml and vendor folder
    because it is used by the provider
  • adds "interfaces" packages and moves Registry and Provider interfaces
    declaration there. This is required in order to use registry in
    provider unit tests. Otherwise there going to be circular imports
    because provider UTests would import registry which imports providers
    back.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 27, 2017
@istalker2 istalker2 force-pushed the coredns branch 2 times, most recently from dcf8dd7 to 862679f Compare June 27, 2017 18:35
@mikkeloscar
Copy link
Contributor

adds "interfaces" packages and moves Registry and Provider interfaces
declaration there. This is required in order to use registry in
provider unit tests. Otherwise there going to be circular imports
because provider UTests would import registry which imports providers
back.

This mostly hides the circular dependency, by putting them in the same package. It's still there on a logical level. :)

As the provider doesn't depend on the registry (but the other way around) it should be possible to test it without the registry.

@istalker2
Copy link
Contributor Author

TL;DR; the comment from @mikkeloscar has been addressed, "interfaces" package has gone. Thank you!

The reason why I created interfaces package is to use TXT registry in coredns_test.go. Currently this is impossible due to circular imports. This doesn't discard the fact that registries are tight to providers on logical level, but decouples interface from implementations and breaks circular imports.
The reason, why I wanted TXT registry to be used in tests is because registry by itself creates additional TXT endpoints. Also I wanted the test to be more close to the real code flow so that it use provider Records(), calculate real Plan and append missing TXT records.

However, this makes it to be not a true unit test as it touches several units at once. Also reorganizing code in the same PR with a new feature is not good even though I believe such reorganization is good. So I removed the interfaces package and changed the test

@dshulyak
Copy link

@istalker2 could you please adjust etcd client to work with tls certificates, as in kubernetes/kubernetes#47049 ?

@pigmej
Copy link

pigmej commented Aug 1, 2017

@mikkeloscar could you please take yet another look there? What needs to be addressed to merge that (besides obvious conflict in main.go)? Should it wait for #228, because it seems that development there stalled a bit.

@istalker2

@istalker2
Copy link
Contributor Author

@dshulyak done. Please see 3rd commit message for details

@bcg62
Copy link

bcg62 commented Aug 15, 2017

+1 for this

@hjacobs
Copy link
Contributor

hjacobs commented Aug 17, 2017

What is missing to get this merged?

Copy link

@ideahitme ideahitme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have not looked at it closely, have few comments though:

api etcd.KeysAPI
}

var _ skyDNSClient = etcdClient{}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should probably be in test

}

// Service represents SkyDNS/CoreDNS etcd record
type Service struct {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer it to be SkyDNSService

@@ -0,0 +1,392 @@
/*

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testing coverage needs to be increased, currently it is only ~40% for this file

@istalker2 istalker2 force-pushed the coredns branch 2 times, most recently from a928724 to ea5e710 Compare August 24, 2017 12:00
@linki linki added docs missing kind/feature Categorizes issue or PR as related to a new feature. work-in-progress labels Aug 25, 2017
@k8s-ci-robot k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Aug 31, 2017
@istalker2 istalker2 force-pushed the coredns branch 2 times, most recently from 9078b57 to da74c04 Compare August 31, 2017 16:58
@bcg62
Copy link

bcg62 commented Nov 6, 2017

this still alive?

This commit adds support for CoreDNS through its etcd middleware.
Because the middleware is backward compatible with SkyDNS this
commit adds support for SkyDNS as well. In fact, new provider
is available under two names in CLI (coredns and skydns).

All interactions with middleware happen through etcd cluster,
whose location (URIs) is specified via --etcd CLI parameter
by default http://localhost:2379).

The provider translates CoreDNS/DkyDNS SRV records to
A/CNAME + optional TXT endpoints, when reading from etcd and
combines A/CNAME with TXT endpoints back into single SRV record
when writing it back.

Also adds github.com/coreos/etcd package to glide.yaml and vendor folder
because it is used by the provider
@istalker2
Copy link
Contributor Author

this still alive?
I rebased the commit, yet to find someone, who will work on getting this PR merged since, unfortunately, need to focus on another project at the moment

This commit adds ability to use TLS transport for etcd.
New logic is applied when the etcd URL has https:// scheme.
TLS parameters are passed in the environment variables:

ETCD_CA_FILE - path to CA certificate. If not specified, then
system-provided certificates are used.

ETCD_CERT_FILE - client certificate
ETCD_KEY_FILE - client key file
- either both of none of this two must be specified

ETCD_TLS_SERVER_NAME - expected CN of the certificate. Useful when
URL points to a different domain from that in server certificate

ETCD_TLS_INSECURE - if set to "1" (or "true" or "yes") makes client
bypass server certificate validation.

Also for unification with other providers and rest of connection
settings, etcd URL is no longer specified in the command line, but
rather in ETCD_URLS environment variable (defaults to
http://localhost:2379). More than one comma-separated URL can be
specified. All of the URLs must start with either http:// or https://

Also, now it possible to communicate with etcd through proxy specified
in standard environment variables
@gogeof
Copy link

gogeof commented Nov 30, 2017

Is this finished?
@istalker2 thanks for your hard work, I also need this feature.

@gogeof
Copy link

gogeof commented Nov 30, 2017

@istalker2 Could you add a tutorial of coredns in external-dns/docs/tutorials/ teach us how to use it ?

@istalker2
Copy link
Contributor Author

@gogeof You use it exactly the same way as other externanal-dns providers:

  1. Use "coredns" or "skydns" as a provider name for the CLI parameter
  2. Deploy etcd database that is going to be the backend for both CoreDNS and this provider
  3. Configure CoreDNS to use this etcd etcd instance
  4. Pass etcd URLs to the external-dns through the ETCD_URLS environment variable. See commit message for the last of the 3 commits here for a list of supported environment variables

@gogeof
Copy link

gogeof commented Dec 1, 2017

@istalker2 thanks a lot
I build from your repo, and run external-dns in my env, but get these message, seems not work very well:

# docker logs -f 89b90ac28122
time="2017-12-01T05:40:36Z" level=info msg="config: &{Master: KubeConfig: Sources:[service ingress] Namespace:default FQDNTemplate: Compatibility: PublishInternal:false Provider:coredns GoogleProject: DomainFilter:[example.org.] AWSZoneType: AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: CloudflareProxied:false InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InMemoryZones:[] Policy:sync Registry:txt TXTOwnerID:default.example.org TXTPrefix: Interval:1m0s Once:false DryRun:false LogFormat:text MetricsAddress::7979 LogLevel:debug Debug:false}" 
time="2017-12-01T05:40:36Z" level=info msg="Connected to cluster at https://10.96.0.1:443" 
time="2017-12-01T05:40:36Z" level=debug msg="No endpoints could be generated from service default/cheddar" 
time="2017-12-01T05:40:36Z" level=debug msg="No endpoints could be generated from service default/kubernetes" 
time="2017-12-01T05:40:36Z" level=debug msg="No endpoints could be generated from service default/stilton" 
time="2017-12-01T05:40:36Z" level=debug msg="No endpoints could be generated from service default/wensleydale" 
time="2017-12-01T05:40:36Z" level=debug msg="No endpoints could be generated from ingress default/cheese" 
time="2017-12-01T05:41:36Z" level=debug msg="No endpoints could be generated from service default/cheddar" 
time="2017-12-01T05:41:36Z" level=debug msg="No endpoints could be generated from service default/kubernetes" 
time="2017-12-01T05:41:36Z" level=debug msg="No endpoints could be generated from service default/stilton" 
time="2017-12-01T05:41:36Z" level=debug msg="No endpoints could be generated from service default/wensleydale" 
time="2017-12-01T05:41:36Z" level=debug msg="No endpoints could be generated from ingress default/cheese"

here is my Dockerfile to build external-dns

# builder image
FROM golang:1.8 

WORKDIR /go/src/github.com/kubernetes-incubator/external-dns
COPY . .
RUN make test
RUN make build

# final image
#FROM registry.opensource.zalan.do/stups/alpine:latest
FROM ubuntu:16.04
MAINTAINER Team Teapot @ Zalando SE <team-teapot@zalando.de>

COPY external-dns /bin/external-dns

ENTRYPOINT ["/bin/external-dns"]

here is my external-dns.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: exeternal-dns
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: dev
    spec:
      containers:
      - name: external-dns
        image: index.youruncloud.com/ufleet/external-dns:v0.0.6
        args:
        - --source=service
        - --source=ingress
        - --namespace=default
        - --domain-filter=example.org.
        - --provider=coredns
        - --registry=txt
        - --txt-owner-id=default.example.org
        - --log-level=debug
        env:
        - name: ETCD_URLS
          value: http://192.168.4.106:12379

@istalker2
Copy link
Contributor Author

@Gegeof these warnings have nothing to do with the provider code. Actually, it shows, that provider works as expected. The warnings you see are printed by external-dns sources which collect data that is going to be fed to the provider then. external-dns scans various supported k8s objects marked with special annotation and tries to generate a DNS record out of their metadata. There can be several different approaches for generating such record out of an object. But if none of them succeed you get such warning. You're going to get it with any provider

@gogeof
Copy link

gogeof commented Dec 1, 2017

@istalker2 yes, you're right, I have found the annotations in code:
source/service.go

90		svcEndpoints := sc.endpoints(&svc)
91
92		// process legacy annotations if no endpoints were returned and compatibility mode is enabled.
93		if len(svcEndpoints) == 0 && sc.compatibility != "" {
94			svcEndpoints = legacyEndpointsFromService(&svc, sc.compatibility)
95		}
96
97		// apply template if none of the above is found
98		if len(svcEndpoints) == 0 && sc.fqdnTemplate != nil {
99			svcEndpoints, err = sc.endpointsFromTemplate(&svc)
100			if err != nil {
101				return nil, err
102			}
103		}
104
105		if len(svcEndpoints) == 0 {
106			log.Debugf("No endpoints could be generated from service %s/%s", svc.Namespace, svc.Name)
107			continue
108		}

source/service.go

133 // endpointsFromService extracts the endpoints from a service object
134 func (sc *serviceSource) endpoints(svc *v1.Service) []*endpoint.Endpoint {
135 	var endpoints []*endpoint.Endpoint
136 
137 	// Get the desired hostname of the service from the annotation.
138 	hostnameAnnotation, exists := svc.Annotations[hostnameAnnotationKey]
139 	if !exists {
140 		return nil
141 	}
142 
143 	hostnameList := strings.Split(strings.Replace(hostnameAnnotation, " ", "", -1), ",")
144 	for _, hostname := range hostnameList {
145 		endpoints = append(endpoints, sc.generateEndpoints(svc, hostname)...)
146 	}
147
148	return endpoints
149 }

and find the annotation here:
source/source.go

32 	hostnameAnnotationKey = "external-dns.alpha.kubernetes.io/hostname"

This is my fault, maybe I should learn more how to use external-dns, I will continue. Thanks a lot.

@gogeof
Copy link

gogeof commented Dec 4, 2017

@istalker2 thanks a lot, I use your repo to build external-dns, and after test, it works for me. After your code merge, I will add doc.

@johnmarcou
Copy link

johnmarcou commented Dec 18, 2017

Hello,

Thank you for this pull request. I am testing external-dns with coredns provider, it works really fine.
I am just experimenting a little issue I would like to report though: ingress rules with multiple IPs addresses create a flapping effect on the DNS resolution, returning sometimes the IP list, sometimes a single IP.

EDIT: I read somewhere in the ExternalDNS that mutliple IP are not supported, which make my message as no sense.

Context:

On our on-prem Kubernetes cluster, the Nginx ingress controller is deployed on two nodes (10.24.100.2,3) as entrypoints for the ingress rules. This is a description of one of our ingress rules for example:

Name:             kops-kube-ops-view
Namespace:        kube-system
Address:          10.24.100.2,10.24.100.3
Default backend:  default-http-backend:80 (<none>)
TLS:
  SNI routes kops.company.org.au
Rules:
  Host                                Path  Backends
  ----                                ----  --------
  kops.company.org.au
                                      /   kops-kube-ops-view:80 (10.25.7.68:8080)
Annotations:
Events:  <none>

When external-dns starts, it creates the 2 DNS records properly:

time="2017-12-17T23:44:39Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.2, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:44:39Z" level=info msg="Add/set key /skydns/au/org/company/kops/6074de14 to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""

You can notice:

  • 2f7a904e = 10.24.100.2
  • 6074de14 = 10.24.100.3
# nslookup kops.company.org.au 10.24.0.1
Server:		10.24.0.1
Address:	10.24.0.1#1053

Name:	kops.company.org.au
Address: 10.24.100.2
Name:	kops.company.org.au
Address: 10.24.100.3

The issue

But on the next updates, after --interval time, we are facing two issues:

  • next update apply only on a single etcd item / dns record
  • the updated record doesn't match with its original IP, which result in a duplication of IP
time="2017-12-17T23:44:59Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""

You can notice:

  • 2f7a904e = 10.24.100.3
  • 6074de14 = 10.24.100.3
# nslookup kops.company.org.au 10.24.0.1
Server:		10.24.0.1
Address:	10.24.0.1#1053

Name:	kops.company.org.au
Address: 10.24.100.3

Across multiple updates from external-dns, DNS server returns the first, the second, or both IPs.

Expected

The DNS server should always return all the IP addresses.

Thank you for your help.

Complete logs for information:

time="2017-12-17T23:44:39Z" level=info msg="config: &{Master: KubeConfig: Sources:[ingress] Namespace: FQDNTemplate: Compatibility: PublishInternal:false Provider:coredns GoogleProject: DomainFilter:[company.org.au] AWSZoneType: AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: CloudflareProxied:false InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InMemoryZones:[] Policy:upsert-only Registry:txt TXTOwnerID:default TXTPrefix: Interval:20s Once:false DryRun:false LogFormat:text MetricsAddress::7979 LogLevel:info Debug:false}"
time="2017-12-17T23:44:39Z" level=info msg="Connected to cluster at https://10.24.110.1:443"
time="2017-12-17T23:44:39Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.2, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:44:39Z" level=info msg="Add/set key /skydns/au/org/company/kops/6074de14 to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:44:59Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:45:19Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.2, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:45:39Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:45:59Z" level=info msg="Add/set key /skydns/au/org/company/kops/6074de14 to Host=10.24.100.2, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:46:19Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.2, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:46:39Z" level=info msg="Add/set key /skydns/au/org/company/kops/6074de14 to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:46:59Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:47:19Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.2, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:47:39Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:47:59Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.2, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:48:19Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:48:39Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.2, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:48:59Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:49:19Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.2, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:49:39Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:49:59Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.2, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:50:19Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:50:39Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.2, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:50:59Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:51:19Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.2, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:51:39Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:51:59Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.2, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:52:19Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:52:39Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.2, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:52:59Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:53:19Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.2, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:53:39Z" level=info msg="Add/set key /skydns/au/org/company/kops/6074de14 to Host=10.24.100.2, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:53:59Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:54:19Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.2, Text="heritage=external-dns,external-dns/owner=default""
time="2017-12-17T23:54:39Z" level=info msg="Add/set key /skydns/au/org/company/kops/2f7a904e to Host=10.24.100.3, Text="heritage=external-dns,external-dns/owner=default""

@linki
Copy link
Member

linki commented Feb 12, 2018

Merged-in latest master branch

@AntonioMeireles
Copy link

hi all !

what's current status on this ? any ETA / blockers re: getting this merged ?

Best regards!

@calder
Copy link

calder commented May 16, 2018

Any updates?

@k8s-ci-robot k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels May 24, 2018
@linki
Copy link
Member

linki commented May 24, 2018

I merged in master and fixed the conflicts:

  • combine --provider flag with additonal providers from master (e.g. openstack)
  • minimal fixes to work with multiple targets: ep.Target => ep.Targets[0]
  • changed order of parameters of NewEndpoint: moved targets to be variadic at the end

@njuettner
Copy link
Member

Thanks again for your PR, will be released this week.

@njuettner njuettner merged commit 933b7ff into kubernetes-sigs:master Jun 11, 2018
lou-lan pushed a commit to lou-lan/external-dns that referenced this pull request May 11, 2022
So far, the last column was labelled with `STATUS` and had three options
(installed, available, unavailable). This was hard to read because
installed and available have the same length, and `unavailable` was not
specific.

Now, the column is labelled `INSTALLED` with values
yes/no/unavailable on <PLATFORM>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. docs missing kind/feature Categorizes issue or PR as related to a new feature. new-provider size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. work-in-progress
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet