Clarify libdns API regarding DNS records names being absolute or relative

[delthas]

Hi,

It is unclear whether the DNS record names (libdns.Record.Name):

1. use the standard "dot-suffix" relative/absolute notation (eg if zone FQDN is example.org. then subdomain.example.org. -> subdomain.example.org ; subdomain -> subdomain.example.org)
2. always use absolute paths implicitly (subdomain.example.org -> subdomain.example.org)

I think 1) is better as it directly reflects what a DNS server actually returns, but it seems that currently the API contract is implicitly 2).

Examples:

• the gandi provider has to manually trim the .example.org part to get a relative notation to pass that to Gandi (notice how it expects to receive subdomain.example.org without an ending dot suffix) which uses the standard dot-suffix notation
• the dnspod provider behaves the same (but is buggier because it replaces all instances of the zone in the domain name which is incorrect)
• the cloudflare provider passes the value as-is to the API, but the API docs clearly show that the API uses absolue paths implicitly (see example.org, should be example.org. if using the dot-suffix notation, but here's it's just example.org)
• the digitalocean provider does not work becauses libdns passes implicitly absolute paths without a dot suffix and the provider passes it as is to the API which expects the dot-suffix notation (1))

I suggest to choose which syntax/semantics we want to use for DNS record names, document it clearly and update the various libdns providers as necessary. I think using the standard absolute/relative notation depending on a final dot is better as it better correponds to what a DNS server returns.

[mholt]

Thanks for bringing this up. I think I had noticed this somewhere along the way but lost track of it before I did anything about it because of everything else going on recently.

I can see how we'll need to standardize this... I actually was thinking of (1) when I designed these interfaces. Yet, when I implemented the new DNS solver into CertMagic recently, I used the full domain name (sans trailing dot).

So if I understand correctly, you propose we standardize on relative or absolute depending on the presence of the dot suffix:

• Absolute (FQDN) if ends with .
• Relative (to zone/FQDN) if does NOT end with .

Does that sound about right?

That would mean that providers need to support both, right? I wonder if it would be simpler if we always used only one of those and then the implementation can simply always transform it if necessary for their provider's API.

[crazy-max]

@mholt I think https://github.com/go-acme/lego/tree/master/providers/dns could help there.

[mholt]

@crazy-max In what way? The limitations and difficulties with that code base are the reason why I designed libdns.

[ncw]

If you are a defining a zone file for use with multiple domain names then you want the relative form and the absolute form.

I don't know whether libdns exposes that functionality though, but even if it doesn't the hosting provider may.

[mholt]

@ncw The libdns interfaces take a zone name as input as well as the list of records to manipulate within that zone. The actual implementation is up to whatever the DNS provider APIs allow/require.

[ncw]

If you wanted to fully model what bind does then you would need an extra concept. In bind the zone isn't attached to any particular host name. You do that separately. This is where the difference between relative and absolute addresses are important. Relative addresses have the current host name tacked on, absolute ones don't.

I designed Memset's DNS API to model this: https://www.memset.com/apidocs/methods_dns.html

The Memset DNS API has three main objects

• A zone domain
• A zone record
• A zone
  A zone is composed of a number of zone domains and a number of zone records. The zone domains specify which domains the nameservers will return this zone for, and the zone records specify which records are returned.

However some DNS APIs don't model this extra complexity and just provide a zone with a single hostname and attached records. This works fine of course but makes it harder to just add another domain to the zone.

I guess the libdns interface could be used with either the full or simplified DNS API, the mappings of which domain going to which zone being out of scope.

So to answer the original question, I think you need to allow both relative and absolute names, and using a dot in the end to distinguish seems sensible as that is what bind does.

For some providers you might need to make the names absolute before passing them to the API but some could deal with both relative and absolute names and do different things with them (eg Memset's API)

I wonder if it would be simpler if we always used only one of those and then the implementation can simply always transform it if necessary for their provider's API.

I think that would be a mistake for providers which can host multiple domains from a single zone file as it would remove flexibility.

Apologies for the long winded answer and also apologies if I'm teaching grandma to suck eggs ;-)

[mholt]

Thanks Nick -- we don't need to completely replicate bind's model for DNS records, but we do need something that works with most DNS provider APIs and gives people the flexibility to append, set, delete, and update DNS records.

Ideally we wouldn't have to add more input parameters to the interface methods. Is it OK if we just keep the zone input and then document that the implementation needs to consider whether there is a trailing dot and handle accordingly for the DNS provider?

And don't worry, grandma has nothing to do with this :)

[ncw]

Ideally we wouldn't have to add more input parameters to the interface methods. Is it OK if we just keep the zone input and then document that the implementation needs to consider whether there is a trailing dot and handle accordingly for the DNS provider?

I think that sounds fine.

I note that you didn't define what the zone id actually is anywhere.

I imagine it would be a host name for some providers but might be a uuid for others?

And don't worry, grandma has nothing to do with this :)

😃

[mholt]

I note that you didn't define what the zone id actually is anywhere.

I imagine it would be a host name for some providers but might be a uuid for others?

Host name. We need it to be the same to users, regardless of providers. So probably a domain name, not a provider-specific UUID.

[matthiasng]

So if I understand correctly, you propose we standardize on relative or absolute depending on the presence of the dot suffix:

• Absolute (FQDN) if ends with .
• Relative (to zone/FQDN) if does NOT end with .

Does that sound about right?

That would mean that providers need to support both, right? I wonder if it would be simpler if we always used only one of those and then the implementation can simply always transform it if necessary for their provider's API.

From the view of API design i would vote to use only one of those.

we don't need to completely replicate bind's model for DNS records, but we do need something that works with most DNS provider APIs and gives people the flexibility to append, set, delete, and update DNS records.

Disclaimer: I'm pretty new to all this DNS stuff, so please correct me if i'm wrong or missed something:

• Names relative to the zone sould be fine. I cannot see any downside here.

  zone: example.com
  name: test
  result: test.example.com -> even straightforward for me :D
  

• I'm not sure about absolute names because of the additional zone parameter in the API

  zone: example.com
  name: test.example.com.
  result name: test.example.com -> redundant but also straightforward
  

  but what if i change the zone to example.NET and leave the name to *.example.com. ?
  I cannot see how i could model this with, for example, Hetzner's DNS API.
  Is this even allowed ? If not I dont see any reason why we will loose flexibility if libdns do not support absolute names.

So if we don't loose any flexibility with it, i personally would prefer to support only relative names.

[viscropst]

Names relative in Chinese DNS providers (DNSPOD,Alibaba DNS,etc.) like a standard, if you want to get an acme challenge for domain yx.viscrop.top the complete domain is _acme-challenge.yx.viscrop.top.
If caddyserver put _acme-challenge.yx.viscrop.top to an record name, DNSPOD's API will return the error in Chinese '子域名级数超出限制' translate to English is 'sub-domain levels out of limits',So the record names being relative is right choice,
if some dns providers need an complete domain for record name, contact 'record name' and 'zone' is not a bad idea.
Here is the part of log of caddyserver

2020/11/18 03:27:22.028 ERROR   tls.issuance.acme.acme_client   cleaning up solver      {"identifier": "yx.viscrop.top", "challenge_type": "dns-01", "error": "no memory of presenting a DNS record for yx.viscrop.top (probably OK if presenting failed)"}
2020/11/18 03:27:22.201 ERROR   tls.obtain      will retry      {"error": "[yx.viscrop.top] Obtain: [yx.viscrop.top] solving challenges: presenting for challenge: adding temporary record for zone viscrop.top.: Create record err.Zone:viscrop.top., Name: _acme-challenge.yx.viscrop.top, Value: 8HjC5i8FEUHEIdzzWEGA5ltxzQ07kVW5Dig-y-QW4Vk, Error:could not get domains: 子域名级数超出限制, { TXT _acme-challenge.yx.viscrop.top 8HjC5i8FEUHEIdzzWEGA5ltxzQ07kVW5Dig-y-QW4Vk 0s} (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/16660603/186494357) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 4, "retrying_in": 300, "elapsed": 309.76547885, "max_duration": 2592000}

[mholt]

Thanks for the feedback so far!

It looks like the consensus is to make the record name relative. We still need to know the zone, though, correct?

Can we verify that providers will not lose flexibility with regards to wildcards, for example, if we use relative names?

[viscropst]

Something update of sub-domain's levels,the limits of sub-domain's is depend on your DNS Provider's subscription,because of I'm using the free subscription of DNSPOD so I have only 3 levels of sub-domain. But record Names relative in Chinese DNS providers like a standard is truth.

For domains with wildcards, In the free subscription of Alibaba Cloud DNS and DNSPOD,I will have at least one levels of wildcards domain record to add(likes *.yx.viscrop.top). But some domains likes qn-*.viscrop.top will need an enterprise subscription,but Alibaba Cloud DNS has more freedom to set wildcard domains(in free subscription,it's only have the 5 levels of sub-domain).

[mholt]

Been giving this some thought.

Currently the libdns APIs accept a zone parameter and either accept or return a []Record parameter.

Based on what hasa been said thus far, I believe this API will continue to work, and we need only to clarify that record names are relative to the given zone.

As to what @ncw said:

I guess the libdns interface could be used with either the full or simplified DNS API, the mappings of which domain going to which zone being out of scope.

I think I agree; I want to keep libdns simple, and I don't currently realize any significant loss of functionality with the current APIs. Multiple zones can be manipulated with multiple function calls, and mapping more domains to a zone is not a use case that we've encountered so far with these libraries, so I presume nobody has needed that functionality.

So then, to summarize:

I propose we update the documentation to explicitly state that:

• Record names are relative to the zone
• Zone names OMIT the trailing dot .

Did I get that right? Does that sound good?

[mholt]

It looks like CertMagic, the primary caller/user of libdns functions so far, is getting the DNS record name of the ACME challenge record from the ACME server directly, which is basically the full domain name: _acme-challenge.foo.example.com. So it will simply have to do strings.TrimSuffix(name, "."+zone). (Correct me if I'm wrong.) Thus the record's Name field would become _acme-challenge.foo if the zone is example.com. Make sense?

[mholt]

In preparing CertMagic for this adjustment, then, I ended up with this function, which I will add to the libdns package:

// RelativeName makes fqdn relative to zone. For example, it transforms
// "sub.example.com" to "sub" when the zone is "example.com". It tolerates
// trailing dots but does not require them; i.e. for the prior example,
// the respective input values of "sub.example.com." and "example.com."
// would be equivalent.
func RelativeName(fqdn, zone string) string {
	cleanFQDN := strings.TrimSuffix(fqdn, ".")
	cleanZone := strings.TrimSuffix(zone, ".")
	relative := strings.TrimSuffix(cleanFQDN, cleanZone)
	return strings.Trim(relative, ".")
}

I am writing tests for it now and am not sure how to handle a couple of things.

1. If FQDN is example.com and zone is also example.com (either with or without trailing dot), should we return "" or "@"?
2. If FQDN is example.com but zone is example.net (i.e. assuming zones are always suffixes of the FQDN, the FQDN cannot possibly be in the zone in this case), should we return "example.com" (i.e. return the input)?
3. Are zones always suffixes of the FQDN?
4. Is this the even right implementation altogether?

FWIW, Cloudflare's DNS API seems to accept both relative and absolute record names, equivalently. In my testing, using sub.example.com as the record name in a zone for example.com, and sub as the record name in the same zone of example.com both yielded the same results. When it returns the newly created records, its name always is absolute, even if the input was relative. It seems Cloudflare standardizes on full domain names. Why? Is that good practice? Should we be doing that instead?

[mholt]

There has been no feedback so I will assume that my implementation is correct. Unless there is new feedback soon, I will go ahead and push these changes probably this week or next, at which point changes may be required in existing DNS provider packages.

I will do my best to help update them, but I'd feel better about this with more feedback first.

[matthiasng]

Hey @mholt, sry for the late response.

If FQDN is example.com and zone is also example.com (either with or without trailing dot), should we return "" or "@"?

DOMAIN NAMES - rfc1035 states the following: A free standing @ is used to denote the current origin.
I think in both cases, some providers needs to implement special handling, but i feel an "at" is more convenient for the user. What do you mean ?

2. If FQDN is example.com but zone is example.net (i.e. assuming zones are always suffixes of the FQDN, the FQDN cannot possibly be in the zone in this case), should we return "example.com" (i.e. return the input)?

Consider the following:

p.AppendRecords(context.TODO(), "exampe.com", []libdns.Record{
    {
        Type:  "TXT",
        Name:  libdns.RelativeName("example.com", "exampe.net"),
        Value: "test",
        TTL:   123,
    },
})

This would lead to Record.Name == "example.com" which violates the "relative names only" rule and depending on the provider, we need special handling for this.
I would rather shift this problem to the caller, by having 'RelativeName' return an error.

3. Are zones always suffixes of the FQDN?

A zone is part of the domain namespace and is associated with a particular named domain - https://www.sciencedirect.com/topics/computer-science/domain-namespace
So my conclusion is: yes. :-D

FWIW, Cloudflare's DNS API seems to accept both relative and absolute record names, equivalently. In my testing, using sub.example.com as the record name in a zone for example.com, and sub as the record name in the same zone of example.com both yielded the same results. When it returns the newly created records, its name always is absolute, even if the input was relative. It seems Cloudflare standardizes on full domain names. Why? Is that good practice? Should we be doing that instead?

So this means no need for a RelativeName function and the providers has to handle both cases and always return an absolute name. A'm i right ?

[mholt]

@matthiasng Thanks for the feedback! A bit refreshing to not be working on this in a vacuum. :)

DOMAIN NAMES - rfc1035 states the following: A free standing @ is used to denote the current origin.
I think in both cases, some providers needs to implement special handling, but i feel an "at" is more convenient for the user. What do you mean ?

Yeah, something like that. So which is better to use, the "@" or the empty string? Obviously, individual DNS providers will need to adjust based on what their APIs expect either way.

Help me understand this:

Consider the following:

p.AppendRecords(context.TODO(), "exampe.com", []libdns.Record{
{
Type: "TXT",
Name: libdns.RelativeName("example.com", "exampe.net"),
Value: "test",
TTL: 123,
},
})
This would lead to Record.Name == "example.com" which violates the "relative names only" rule

That is a bit of an add example, since you're passing a zone of example.com (I assume your sample code has typos) to AppendRecords(), but then specifying a zone of example.net in the call to RelativeName(). Wouldn't they be the same?

In what case would a record for example.com be created in a zone for example.net? That seems like an input error, right? If the name is already relative (example.com.example.net -- weird, but that's what your example implies), you wouldn't call RelativeName(), which I believe correctly returns its input in that case.

I would rather shift this problem to the caller, by having 'RelativeName' return an error.

The only problem with this is that I don't think we can know if it's an error. The function is idempotent if it's already a relative name, but there's no way to know if it's outside the zone.

So this means no need for a RelativeName function and the providers has to handle both cases and always return an absolute name. A'm i right ?

Well, I don't know. Records we return from the provider need to be standardized too. Cloudflare happens to return FQDN (absolute) record names, but apparently accepts either FQDN or relative as input. For input, then, the Cloudflare provider wouldn't really need the RelativeName() function. But for output, it would if we decide that libdns APIs work with relative names.

[matthiasng]

That seems like an input error, right?

Maybe i messed up my thoughts a bit. I think it is fine as it is (returning the input), because you are right, my example has just an input error.

The only problem with this is that I don't think we can know if it's an error. The function is idempotent if it's already a relative name, but there's no way to know if it's outside the zone.

The only way i can think of to handle the input error in this case is to distinguish FQDN and relative names by the trailing dots.

Well, I don't know. Records we return from the provider need to be standardized too. Cloudflare happens to return FQDN (absolute) record names, but apparently accepts either FQDN or relative as input. For input, then, the Cloudflare provider wouldn't really need the RelativeName() function. But for output, it would if we decide that libdns APIs work with relative names.

If libdns accepts FQDN's, the zone argument feels a bit redundant but i thinks that's fine.

I also want to add another idea: Support only absolute names an remove the zone arguments.
Positive: the api get's simpler.
Negative: Providers needs to resolve the authoritive zone itself. I know lego is doing the same, so i think it works pretty well, but still bloats up the provider internals.

[mholt]

If libdns accepts FQDN's, the zone argument feels a bit redundant but i thinks that's fine.

Why is that? I'm probably missing something obvious, but some APIs require you to specify the zone you're working with. How can you reasonably extract the zone implicitly from the FQDN (i.e. without extra DNS lookups, API requests, TLD lists, those kinds of things).

Negative: Providers needs to resolve the authoritive zone itself. I know lego is doing the same, so i think it works pretty well, but still bloats up the provider internals.

Ah, yeah, so in libdns case we expect the caller to get this information first. CertMagic -- which is libdns' primary importer right now -- does this DNS lookup similarly to lego. It is a pain for users to configure in extraordinary network environments, though, and doesn't always work otherwise. I figure not all applications will need, or can rely on, automatic DNS lookups, as sometimes they might simply be user input. I figured keeping libdns APIs more bare-bones was important. And this way, each DNS provider doesn't have to do the DNS lookup themselves.

I'm open to moving the DNS lookup code into this repo as an exported function, but it'd be used by importers of libdns providers (like CertMagic), not the providers themselves.

[matthiasng]

Why is that? I'm probably missing something obvious, but some APIs require you to specify the zone you're working with. How can you reasonably extract the zone implicitly from the FQDN (i.e. without extra DNS lookups, API requests, TLD lists, those kinds of things).

I was a bit fixated to DNS Challenges. You are right. Most, if not all, DNS API's requires the fqdn/relative name and the zone.

FWIW, Cloudflare's DNS API seems to accept both relative and absolute record names, equivalently. In my testing, using sub.example.com as the record name in a zone for example.com, and sub as the record name in the same zone of example.com both yielded the same results. When it returns the newly created records, its name always is absolute, even if the input was relative. It seems Cloudflare standardizes on full domain names. Why? Is that good practice? Should we be doing that instead?

I did a quick search through lego's dns provider implementation and i can clearly say the following:
There is no pattern. Half of them are using FQDN's some are using relative names and even some others wants FQDN without a trailing dot.
So after thinking twice, my conclusion is the following: It doesn't really matters, because all the different API's got different requirements.

I also like the idea of always returning a FQDN by now, because it sould be easier for further processing.
Then the input must also support FQDN's, otherwise it wouldn't be possible to create new records, just change the value and put the structure back into an update.

If you decide to support both, as you write in your first comment, FQDN and relative names should definitely be distinguished by a trailing dot.

[lscgzwd]

i think there are two styles:

• the sub. i.e. sub.example.com should be sub
• the full. i.e. sub.example.com should be sub.example.com

and for CNAME, some dns provider need the last point(.) . i.e. sub.example.com CNAME to www.example.com. it should be www.example.com. and some dns provider do not need the last point.

[Xinayder]

From experience with Vultr, if you type relative.domain.com., it won't add the zone, but if you type relative.domain.com it will just show up in the list as relative. With Namecheap I think you can type both the FQDN and the relative one and it will work.

I think we can support both, i.e., if you add a record sub it should be converted to sub.domain.com, and if you type sub.domain.com it gives the same result.

Also, I don't know much about DNS but couldn't we use an asterisk to represent the root zone instead of an empty string or "@"?

[mholt]

Thanks for the discussion. This has been very helpful. I've submitted #28 which I believe should resolve this ambiguity for the most part.

There are still some edge cases left like what to do about trailing dots (can we assume that example.com == example.com.? maybe not?) and @ names. But they are topics for different threads, and we can cross those bridges when we come to them.