Public DNS resolvers lists

[jedisct1]

Having a list of public resolvers to be used with dnscrypt clients is critical.

However, the good old CSV file had quite a few drawbacks.

First, it was centralized. One file, available at a unique URL hardcoded in clients and scripts, maintained by one person. It’s fragile and not sustainable.

In order to address this, dnscrypt-proxy v2 works differently. Users subscribe to one or more “sources”.

A source is a URL returning a list of resolvers, and a public key.

Data from these sources are automatically downloaded, verified, and regularly updated.

So, the OpenNIC organization can autonomously maintain a list of their available resolvers, signed with their own key.

If you run your own private servers, you can list them in a private URL. If you use Kubernetes to spawn the server instances, the source data can be built automatically.

If someone wants to publish a list of resolvers that works well for a given country, or a list of resolvers that block ads, or a list of resolvers responding to non-standard ports, or whatever, they can.

Users just subscribe to the sources they are interested in. Then, they can let the software automatically pick the fastest server in all of the available ones, or explicitly choose a subset of servers from these sources to use.

This doesn’t prevent having some reference page (maybe the dnscrypt-proxy wiki) that lists some of the available/recommended sources.

Which brings us to the second point: what kind of data do these sources return?

The CSV format is a bit unusual for software configuration. But it made sense. After all, the list of resolvers and their properties could be nicely presented as a table.

However:

• CSV is not so easy to edit. Everybody doesn’t use a spreadsheet, and long lines don’t play well with text editors. And CSV is not really normalized (separators and quotes…)
• CSV is not extensible. Columns could be added or removed, if we assume that the first line contains property names, and that software associate individual cells to properties. In practice, doing so is lousy, so most scripts and applications just expect a property to be at a specific column. Adding/removing properties effectively is impossible.
• The CSV file was not great for computers to parse reliably. It was also not great for humans to read, as it contained too much useless information (public keys, IP addresses, protocol version, DNSSEC record), unused information (GPS coordinates), and not enough information (just a couple characters to describe what the server is).

So, I’m looking for suggestions to replace it. Or rather, to add to it, since the legacy CSV format will remain supported as well.

dnscrypt-proxy 2 introduces something called “stamps” (for the lack of a better word). A stamp is a base64 string that contains a protocol identifier (regular non-encrypted DNS, DNSCrypt, DNS-over-HTTP2, …) as well as all the parameters required to connect to a server: IP address, port, public keys, etc.

So, if you want people to use your server, you can just give them a single string to copy&paste.

Back to “what could we replace the CSV file with?”.

I’m looking for suggestions on a better way to publish lists of servers. The new format has to:

• Be easy to parse by scripts
• Be human readable
• Allow freeform text.

It could just be something like:

# example-server-1

This is a DNS server provided by https://example.com, located in India.

It filters out ads and trackers. It doesn’t log anything. It supports DNSSEC, and https://example.com also has cool privacy-oriented free software you should check out.

sdns://unoiwueovqunoeiuqwoienuvioquweo

# example-server-2

Another server in India, but that one doesn’t filter anything. Blablabla.

sdns://weonqviuwenqioevunqwioeuvqwoeqw

We still need some structure to have it parsable (here: the name after the # and the sdns:// stamps on their own line), but everything else can be freeform.

Note that stamps also include information about DNSSEC support and log/nolog, so GUIs, scripts and applications can still apply filters based on that. In fact, stamps include a 64 bit bitfield, so we have 62 bits left to store other properties.

I need your input. You are running servers, using the software, writing software, maintaining websites, your input on this is really badly needed.

What do you think about the general idea? What should we replace CSV files with? I’d be then glad to implement whatever makes everybody happy.

[jedisct1]

/cc @d0wnblog @chantra @cofyc @Noxwizard @alterstep @CHEF-KOCH @bitbeans @Fusl @bitbeans @licaon-kter @simonclausen @GortCodex @jaromil @black-roland @F1ash @pwnsdx @publicarray @xorguy @wulungnet @pysiak @Watanuki-kun @df-cryptostorm

[licaon-kter]

Sounds good to me (providers hosting their own up to date, signed; parsable list)!

[bitbeans]

I like the idea! The format also looks good!
I am looking forward to implementation in Simple DNSCrypt :)

[chantra]

Great move on decentralizing list management!

Re list management, how about something more verbose but both computer and human friendly like yml? There would be way to add new backends as they become available, have defaults for some options...

--- 
dnscrypt: 
  - 
    name: resolver1
    stamp: "dnsc://somebase64here"
    description: "blah blah"
  - 
    ip: "127.0.0.1"
    name: resolver2
    port: 443
    pubkey: abcd
  - 
    ip: "127.0.0.2"
    name: resolver3
    pubkey: abcd
    # port defaults to 443
doh: 
  - 
    ip: "127.0.0.1"
    name: doh1
    # cn defaults to name
  - 
    cn: a.b.c
    ip: "127.0.0.2"
    name: doh2

[jedisct1]

@chantra that looks more error-prone (easier to make a syntax error, which will make the whole list unusable) and less human-readable.

I'm not sure that humans need IPs, CNs, pubkeys, or even to see what protocol is being used. All of these could be in the stamp, even for DoH, so everything else would be for humans.

[jedisct1]

But I like Unbound, too :)

[chantra]

I suppose the stamp is easy to copy/paste, but harder to read. Based on the premise of using stamps, then all you would need is a list of stamps and the rest would be comments so the creator of the list can easily match a stamp with a higher level description of what is in that stamp.

[jedisct1]

Exactly. That's just a proposal, though.

Thinking about it, we probably want a description for the list itself. So, sticking to something Markdownish:

# OpenNIC-Asia

This is a list of OpenNIC servers in Asia, maintained by bob@example.com.

## tkyo-1
A server in Tokio. No logs, just the best sushi in the world.
sdns://...

## tpe-1
A server in Taipei.
Powered by delicious food from street food markets.
sdns://

## bjs-1
A server in Beijing.
sdns://

[ghost]

@jedisct1 is there a command or script I could use to create the base64 sdns strings for public servers?

I'd like to take you up on this offer:

If someone wants to publish a list of resolvers that works well for a given country, or a list of resolvers that block ads, or a list of resolvers responding to non-standard ports, or whatever, they can.

:)

[simonclausen]

I like the idea. But would also like to point in the direction of DNSCrypt/dnscrypt-protocol#1 :)

I think using DNS itself as distribution would be compatible and make sense, especially with the "stamps" concept.

Thoughts?

[jedisct1]

That's also why stamps are a short single string. A yet secret evil plan is to be able to publish them as TXT records. These will have to be signed of course, so it won't happen before dnscrypt-proxy has built-in DNSSEC validation.