QUERY_DEPTH attempt to add with overflow

[DarrenTsung]

Hi, I'm trying to swap out the default DNS resolver in hyper with trust-dns and I'm seeing this panic when attempting to send out a lot of requests.

Full error:

thread 'Hyper-Client-Pool Executor' panicked at 'attempt to add with overflow', /Users/darrentsung/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-resolver-0.8.2/src/lookup_state.rs:64:30

The code in question:

...
    /// Perform a lookup against this caching client, looking first in the cache for a result
    pub fn lookup(&mut self, query: Query) -> Box<Future<Item = Lookup, Error = ResolveError>> {
        QUERY_DEPTH.with(|c| *c.borrow_mut() += 1);
...

This is reproducible by running the a_ton_of_notifications() test on my fork of hyper-client-pool which we're using to send notifications.

Fork: https://github.com/DarrenTsung/hyper-client-pool/tree/trust-dns.
I'm replacing the DNS resolver with this repo: https://github.com/DarrenTsung/hyper-http-connector/tree/trust-dns-cache-resolver.

Does anyone have any ideas why this might be happening? It might be related to how I implemented trust-dns resolver?

I have a c-ares implementation that works, but it would be great to use trust-dns for this as we use it elsewhere in our notification pipeline!

[bluejekyll]

Which version of TRust-DNS Resolver are you using? I just patched a similar issue recently: #469

Also, I just published new versions of the libraries on crates.io, I'm in the middle of writing up the announcement.

[DarrenTsung]

Using 0.8, I see you pushed up 0.9 to crates.io. I can update the branch and see if it still happens!

[DarrenTsung]

Did an update to 0.9, the panic no longer occurs but I'm seeing a lot of errors and no requests going through:

running 1 test
ERROR 2018-05-18T00:23:16Z: trust_dns_proto::xfer::dns_future: error in Proto: no error specified
ERROR 2018-05-18T00:23:16Z: trust_dns_proto::xfer::dns_future: error in Proto: no error specified
ERROR 2018-05-18T00:23:16Z: trust_dns_proto::xfer::dns_future: error in Proto: no error specified
ERROR 2018-05-18T00:23:16Z: trust_dns_proto::xfer::dns_future: error in Proto: no error specified
ERROR 2018-05-18T00:24:02Z: trust_dns_proto::xfer::dns_future: error in Proto: io error
ERROR 2018-05-18T00:24:02Z: trust_dns_proto::xfer::dns_future: error in Proto: io error
test a_ton_of_notifications ... FAILED

The error messages aren't really saying too much on what's happening.

[bluejekyll]

I'm sorry, I don't have time right now to dig into your issue. But I recently put together an example that might help you: https://github.com/bluejekyll/trust-dns/blob/master/resolver/examples/global_resolver.rs

[DarrenTsung]

👍 Thanks for the example, no worries, I appreciate all the help!

[bluejekyll]

Is this issue resolved?

[DarrenTsung]

No, I haven't taken a look at fixing this as c-ares is working for us right now. The issue can still be reproduced on the trust-dns branch of our repo:

https://github.com/DarrenTsung/hyper-client-pool/tree/trust-dns

[bluejekyll]

It looks like tokio-core is being used there. Not sure if that’s an issue.

[DarrenTsung]

Replacing the tokio-core implementation with the global resolver fixed the issue, thanks for the suggestion!

[bluejekyll]

Awesome!