Provide fallback semantics for upstream server groups

[paulie-g]

Currently, you can specify more than one upstream server in the google or ietf groups in doh-client.conf. The semantics for choosing one to query are currently random choice. It would be useful to provide an option to configure doh-client to use the upstream servers as an ordered list with fallback semantics. Something like a {google,ietf}_list_semantics = (random,fallback) configuration option would do. An additional fallback_timeout would be great as well, since it would be useful to have it set fairly low, unlike the overall timeout.

The use case for this is that I prefer using cloudflare over google (I distrust them less with query data than google), but I've had hiccups and would like google to be used automagically when that happens.

Ideally, this would include marking upstream servers as down when multiple queries time out in a short window, but in the absence of a full-blown solution like that, the fallback+timeout config options would suffice.

[m13253]

Yes that's cool. I have planned to add server availability measurement since before.

But I currently do not have time to add new features. Do you know any third-party DoH clients who support this feature? (e.g. dnscrypt-dnsproxy)

[paulie-g]

I feel bad submitting a feature request rather than a pull request, but I have 0 experience with golang. Probably best not to butcher your code with unidiomatic kludges.

To be honest, I've no idea - when I looked for a client I found yours, it was neatly packaged in the AUR so I just went with it.

[m13253]

I come up with an easier solution: Write a frontend UDP server, which listens DNS packets, and run two doh-clients, one for CloudFlare, one for Google. The UDP server dispatches DNS requests to CF, wait a second, then Google. In this way we don't need to write more Go code, and we can package the whole thing into a Docker container (or flatpak / snap / whatever).

But the point is, usually a client would retry DNS requests after 1 or 2 second. (This is also why I didn't use a "server availability ranking" algorithm in doh-client, eventually the client requests will be sent to all servers.) It might be difficult to make sure that subsequent requests do not mess up with the fallback mechanism.

[paulie-g]

doh-client already understands dns and knows what the request is, so writing the whole thing from scratch as a proxy in front of it seems like a bad idea - why write a proxy to put in front of a proxy? With that said, since the number of people currently requesting this feature, as far as we know, equals 1, I can certainly understand why you wouldn't want to implement it. Especially since the request is borne of a 45 second time period where cloudflare wasn't responding to me and which happened once. I would've thought that at least the fallback option wouldn't be too onerous, but clearly the time it takes to implement > 0.

There's probably something out there that I can stick in front that supports fallback and might even let me redirect queries to .bit and opennic tlds elsewhere. I'm betting I could get unbound to do it for one if I were a masochist. In any case, if you've decided you'd rather not implement any of this, now or at any time in the future, feel free to close.

[m13253]

doh-client already understands dns and knows what the request is, so writing the whole thing from scratch as a proxy in front of it seems like a bad idea - why write a proxy to put in front of a proxy?

I agree. It's too dirty to layer programs one by one.

With that said, since the number of people currently requesting this feature, as far as we know, equals 1, I can certainly understand why you wouldn't want to implement it.

This is true. I apologize for that.

And another reason is, to make the fallback really work, I need to write code to filter out retry packets from downstream, and implement retry logic by myself, which may additionally require a server health check feature (although many DNS software has it, even Firefox's bulitin DoH has a not fully working one, I currently don't really don't know how to implement the health check because HTTP/2 hangs a connection by stream, not packet.)

In conclusion, the fallback feature may require more code than you would imagine (at least more than words we've been typed). Although it's a good feature, I would might need to allocate my time to my seminar paper for a few months.

In any case, if you've decided you'd rather not implement any of this, now or at any time in the future, feel free to close.

I will do it. But later.

Actually the current retry logic is burdening my public DoH server when the clients have a bad Internet connection. Therefore I would eventually implement health check, integrated retry, server pooling, and fallback algorithm at a same time someday.

And I apologize for any inconvenience. Thank you for understanding.

[paulie-g]

No worries mate and no need for apologies - you're fully within your rights to say 'patch or GTFO' to any feature requests ;) Let's keep this open and tagged with 'feature request' then, so if/when you, I or someone else gets around to it, the interested parties get notified.

I do have a couple of questions though.

Firstly, why are retries problematic in this scenario? What sort of behaviour have you seen? I would assume that retry behaviour is implementation-specific; I certainly can't think of any standard that dictates it and can't think of any configuration that the standard resolver exposes to configure it. Didn't know FF had a dns-over-https implementation, might have to try it and see what it does (not an FF fan though). I'm not sure why the http/2 semantics are problematic - it's just a transport; logic along the lines of 'send request to currently preferred destination, set timer for retry_timeout, if response arrives cancel the timer, if not send request to preferred callback' should work, no? I can see a situation where one particular http/2 stream hangs and an unnecessary fallback is executed, but that's undesirable rather than a flat out showstopper. Maybe I need to read all of the code to see what you're getting at.

What sort of problem are you seeing with scaling your DoH server? Presumably clients with bad connections would result in lots of tcp connections in linger state, but modern linux kernel deal with that quite well and have done since late-2.6. Is it causing a problem on the golang end? Golang has a reputation for scaling well for networking applications and having a good async http implementation. What sort of problems are you seeing? All of those lingering connections should just be one more socket in the epoll fdset which scales independently of the size of the fdset (in the kernel at least, can't speak to golang stdlib). It's not related to this discussion, but I'd be curious to know - go's good-enough performance, scaling characteristics and high-quality implementations of http and friends being available are a big reason go is towards the top of my list of languages to add to my toolbelt, so anything that argues against that would be of immense interest to me.

[m13253]

Hello,

I am sorry for the late reply. I have been kind of busy these days.

Here are two problems I need to address. One is for server quality assessment, the other is for retry tracking.

Let me explain these problems. It's observed in real world scenario.

Firstly, why are retries problematic in this scenario? What sort of behaviour have you seen?

A typical DNS timeout is 10 seconds (2×5), while a typical HTTPS timeout is 30 seconds (10 for DNS, 20 for TCP and TLS handshake). If we leave the timeout as 30 seconds, the fallback mechanism will not work at all because the client gives up. But if we modify the HTTPS timeout to 5 seconds, at some point we might end up with no usable HTTP connection because every request times out.

In fact, one history version of doh-client had an improper HTTP timeout that floods the server with SYN packets when network is unstable, and does not stop even the network condition restored. So I had to leave the timeout at 30 seconds, and to figure out a method to predict whether a server would time out before it actually does.

This is how dnsdist and other software does: They measure the response time with dummy requests and stop using a bad upstream server when it noticed abnormal response time before a downstream request arrives.

What sort of problem are you seeing with scaling your DoH server?

Actually Go is able to handle a huge number of requests. But my DoH server is banned by several authoritative DNS server providers due to massive amplified retry packets from downstream.

                +------+  A  +--------+  B  +----------+
                |Client|---->|OS Cache|---->|doh-client|
                +------+     +--------+     +----+-----+
                                                 |  C
+------------+  E  +------------------+  D  +----v-----+
|Auth. Server|<----|Optional DNS Cache|<----|doh-server|
+------------+     +------------------+     +----------+

There are two situations:

1. When link C is blocked, the request will be duplicated at link B for 5 times (typically). Since doh-client currently does not track which packets are already seen, it will forward them as is to doh-server. But due to link C is in linger state, doh-client would either wait or create new TCP connections, and these new connections may choke very soon after created. Eventually the number of connections at link C is saturated. (The default settings is around 10 or 100 I guess, and I think it's bad if I deliberately lift it to infinity.) No new connections could be created, so new requests will wait, and be duplicated 5 times at link B.

   An example log is like this:
   

   Or if the network condition goes normal. These duplicated requests would hit the authoritative server in a sudden.

2. When link E is blocked, the client will keep retrying, but the DNS cache is retrying too. So the number of try packets will be 25 at link E. On a busy server these packets will soon exceed the QPS limit set by authoritative servers and get banned.

If I just filter the retry packets at link B, the request failure rate will significantly increase because one failed connection at link C will break a lot of requests. Thus a possible solution is again to use server quality measurement.

Therefore there are two problems (three including your feature request), which can be solved together. I plan to do this later this year or early next year, but it requires a big change in the code base.

[paulie-g]

Thanks for the detailed explanation. Everything makes sense. The key, it seems to me, is to decouple incoming request timeouts from upstream timeouts, create a response queue per incoming query and pool upstream connections. A query comes in, a response queue gets created for it (all subsequent identical queries get added to the queue and no additional queries are sent to upstreams), a short timer gets set (independent of upstream timeout) and query is dispatched via primary upstream's pool. If the short timer fires and no response has yet been received, a query is dispatched via fallback upstream pool and timer is reset. If timer fires again and still no response from either upstream, return SERVFAIL to all queries waiting in queue. If a response does come, return that instead. Pools monitor query timeouts to infer overall upstream health and per-connection upstream health and, in the absence of real queries, issue an IN NS . or something like that.

That's if you want to do all the additional functionality. If all you want to do is resolve the query amplification problem, the per query queue would do the job.

I haven't looked at the code in detail, but the former is likely to take a lot of surgery, while the latter might not be too hard, unless I'm missing something.

Incidentally, it looks like I can kludge the fallback functionality with coredns in front of doh-client if I patch doh-client to quickly return SERVFAIL on timeout. I probably won't bother for now, but it's an option if you genuinely want to keep this complexity out of your code.

[m13253]

I haven't looked at the code in detail, but the former is likely to take a lot of surgery, while the latter might not be too hard, unless I'm missing something.

They are actually not too hard, but time consuming. I already have some idea on how to do it (pretty identical as your suggestion).

Incidentally, it looks like I can kludge the fallback functionality with coredns in front of doh-client if I patch doh-client to quickly return SERVFAIL on timeout.

Please check this, you just need to copy line 159-160 to line 135:

dns-over-https/doh-client/ietf.go

Line 135 in c85ef45

// Do not respond, silently fail to prevent caching of SERVFAIL

Not returning SERVFAIL is by my design to fix an issue with macOS and iOS. You can temporarily patch it before I finally implemented the queue.

And again sorry for the inconvenience.

[m13253]

Now we have weighted round robin load balancing algorithm, contributed by Sherlock-Holo.

Would you want to try it and see whether it works as your need?
You can set Google with very low weight and Cloudflare with high weight.

Please tell me whether it works or not. Thank you.

[paulie-g]

I'll test it in the next few days and report back.

[m13253]

Closing because the problem seems workarounded.
If you think the problem still exists, please reopen this issue. :-)