Caught Real IP LEAK and fallback DNS being used on Very Verbose instead of dnscrypt!

[jfdl1991]

This is a copy paste of the email I just sent. Probably should be on rethink app and not on firestack but I didn't know how to reference the other issue so you'd know my email to check the logs I've sent and the very verbose one that should have got the leaking there. (If you use both mails you should have a long period of time on logs and good screenshots).

Let me know if you need something else. I've been trying to caught this behavior of Dnscrypt and fallback problems and accidentally (and fortunately) happened just after i had set log level to very verbose and caught a real ip leak.
The mail should have the same title as the issue so you can find it easily. ("[Rethink]: Bug report. Caught Real IP LEAK and fallback DNS being used on Very Verbose instead of dnscrypt. !") and all my previous emails describing the erratic behavior on DNSCRYPT and fallback DNS use and my config and some misc bugs.

I hope this is useful for you. Let me know if you need more info.

Here's the copy paste of the mail body:

---

Crash/Issue Report (Logs Attached)

I just wrote a mail about several bugs (and several mails on fallback problems).

While i was doing it something happened and fallback got used and my ip leaked on very verbose logs finally!

As you can see Adguard setting is not getting used and somehow my real ip leaked with Wireguard turned on (wg 2 is Cloudflare warp relay of wg8 that is protonvpn on Romania for brave app and the IP leaked and so did the dns).

Further screenshots to show the change but it should be a continuation on my previous mail.

If you need more info I'll be happy to send it back.

Originally posted by @jfdl1991 in #184

[ignoramous]

From the screenshots, it looks like the domains that are being resolved by "Fallback DNS" are likely "trusted" (allowlisted), either in Configure -> Firewall or at a per-app level (in Configure -> Apps), or the app requesting the name resolution is set up to Bypass DNS & Firewall / Bypass Universal.

This is expected as trusted domains are always resolved by "Fallback DNS" unless *Configure -> DNS -> Use fallback DNS as bypass" is turned OFF.

[jfdl1991]

No.. sorry maybe this needs a lot more explanation. The screenshots where only to show when exactly it did happen. It was a way of putting a sort of timeline so you could check up the logs on that time. And alsl to show you the leaking of the ip on dnsleaktest. My ip shouldn't had been leaked as brave is set to use Wireguard over Wireguard. For Brave browser its set to use Proton VPN Wireguard and also using cloudlflare warp as a relay. My ip should have been masked. First by the warp relay so Cloudflare warp would know my ip but all data would have been tunnelled to protonvpn and encrypted. Then protonvpn shouldn't know my ip, but only the WARP relay IP. Even if protonvpn tunnel had failed the IP shown shoul have been warp relay IP. I've tested this nested Wireguard config and usually it works.

And dns leak and ip leak are different things...
So even if dns wouldn't had leaked to cloudflare resolver which it did happen (even thou my fallback dns is set to rethink servers not to Cloudflare) something weird happened there .
What should have happened it's : from my phone to Cloudflare warp Cloudflare would know my ip and that i was being connected to protonvpn but with all info encrypted (outer layer of the tunnel) then at cloudflare warp end of the data connection that info should be redirected (supposedly by the faster route as that's the function of warp). From there protonvpn should have got a connection from Cloudflare warp ip (proton vpn wouldn't have go know my ip) . Proton would thin that the ip was The cloudflare warp ip edge Endpoint near Romania then the connection should have been encrypted from warp ip to protonvpn server on Romania. From there proton shouldn't know my real IP. Then from warp edge server to protonvpn Romania server all should be encrypted. From protonvpn sever on Romania the connection should have go to dnsleaktest.com server and dnsleaktest.com should have got Protonvpn IP address from that vpn (even if that step would have failed and if the connection would only had been tunnelled by warp vpn, the ip dnsleaktest.com should have be seen as Cloudflare warp ip address).
Then dnsleaktest.com shouldn't have had a way to know my real ip at all but it was shown.

I'm not using "Split DNS" option so cloudlflare shouldn't have resolved my dns query at all. Cloudlflare was not set up as fallback DNS server (my fallback is rethink servers). So dns queries should have go from my real location to cloudflare warp edge input endpoint, then trough Cloudflare warp network to the output cloudlflare edge server nearby Romania and see everything encrypted towards proton vpn server on Romania, as that's the Wireguard relay. Proton server would have seen Cloudflare warp IP (not my real ip). Then decrypt inner tunnel (with 80 less MTU of course) and end up in Romania proton server. Then from there rhe connection from protonvpn sever on Romania with proton VPN IP should have go to dnsleaktest.com server. Dnsleaktest should have thought that my ip was proton's Ip and even if it failed there should have thought the connection came from Cloudflare warp edge server (not from my ip). Then dnsleaktest.com should have sent a bunch of random dns queries. They shouldn't have known my real ip in any stage. Then dns queries made by dnsleaktest should have been routed to protonvpn server on Romania, go to Cloudflare warp server nearby Romania all encrypted and then from there go from warp edge server to my real ip (all double tunneled). Then rethink app should have sent the queries to any given dnscrypt anon relay and depending on the relay routed to the nearby anycast adguard server (datacamp's servers). Then those servers would have shown up on the dnsleaktest.com results as usual and that should have been all.
Dnsleaktest severs should have never ever seen my real ip. Normally the results would be several Adguard dnscrypt target servers ( several would be expected as I'm using different relays on different countries and depending on the relay used, dns queries would have been routed to different adguard anycast servers thinking that the ip where my queries were sent from wherever the anon relays IPs are as usual (and never knowing my ip not from dns part as i use anon relays and not from connection as i use not just one vpn but 2 nested tunnels). Dnsleaktest.com should have never know my real ip and the results shouldn't have shown Cloudflare resolver at all (as i don't have split dns and my fallback dns is rethinkDNS not Cloudflare).

I know this is very confusing but dnsleaktest.com should have never had seen my ip and cloudlflare dns resolver shouldn't have shown as a dns resolvers on the results. I basically put the most complicated setup on everything and the most anonymous one. But in the very verbose logs those anomalies (from several ends) should be shown (whatever that happened). If you manage to unravel what happened I bet you'll be able to patch not just one but several bugs m

(BTW i have 8.x.x.x ; 1.x.x.x and my ISP DNS servers blocked on my firewall rules so it's not like i was allowing cloudlflare dns resolver connections)

Screenshots were sent just to show when fallback strated to being used (easily seen by 🦆) .
Between those times in the screenshots it was all resolved by fallback only and no dnscrypt queries were resolved on every app or domain.
If you need my config or any other detail ill be happy to send them.
I couldn't just send dozens of screenshots of dns logs (but would be useful to be able to send specifically dns logs... And changes on dns protocols and when fallback is being used).

You would really have to check both emails i sent (both onone after another) and the logs on both mails to unravel the whole mystery but probably the last one would be engoug. Screenshots bascialy were jus to show ip leak/dns leak/dns resolver used and timelines on them only.

---

Maybe some filter on app logs would be useful on this. Like dns realted things, wireguard realted thing, etc. And subfilters/categories like dns (odoh, fallback, changes/errors, dnscrypt, doh, cache, dns53 etc). Some kind of checkboxes to filter some specific behavior to be able to send specific things to highlight and delimit the problem). I imagine going through a Very Verbose log is not easy.

Previously you suggested using Gemini/AI to check the logs. However hallucinations are too common and those just fill the nonsense with "probable" things. It would be needed some kind of text document in order to "train" AI so they actually understand whats going on and can process and understand without assuming so much nonsense. A train prompt/documentation for ai to actually be truthful and not making up things could be beneficial both for you as developers and for users that try to help like me (or just want to understand the logs).

I get that very verbose logs can be very confidential and long and the 3h period before reverting to error only logs would be the most private respectful approach but maybe if one could select that time and pre-filter the logs based on categories/events users could send some not so delicate information and send more specific logs about a specific problem. This is so complicated I would not know how to filter it all TBH and I'm comfortable sending you my private info to help you fixing bugs.

I don't want to pressure you ot rush you at all or asphyxiate you as i said before. On the contrary I just want to help you.

If you ask me for more info, settings, or whatever you need I'll be happy to help as much as I'm able to.

Here are some suggestions only and some feedback but the really important part is the leaking part (ip/dns leak).

Ps: if you need help with the translation I'd be happy to help translate to Spanish 👍🏻👍🏻

Maybe I'm all wrong and confused but i think this is worth to take a deep look as probably would help fixing several bugs.

[ignoramous]

even if dns wouldn't had leaked to cloudflare resolver which it did happen (even thou my fallback dns is set to rethink servers not to Cloudflare)

Rethink's zero resolver (in Configure -> Network -> Fallback DNS) is hosted on Cloudflare.

Then dnsleaktest.com shouldn't have had a way to know my real ip at all but it was shown.

Make sure the WireGuard tunnel is Lockdown.

not using "Split DNS" option so cloudlflare shouldn't have resolved my dns query at all.

If you enable Configure -> DNS -> Split DNS, only then (Advanced mode) WireGuard's DNS endpoints are used for DNS. Otherwise it isn't.

Normally the results would be several Adguard dnscrypt target servers

Don't use DNSCrypt in Rethink if you desire no DNS leaks. It has a mandatory "cert refresh" which may fail (and when it does, Rethink uses Configure -> DNS -> Fallback DNS).

Then rethink app should have sent the queries to any given dnscrypt anon relay and depending on the relay routed to the nearby anycast adguard server (datacamp's servers).

From my experience, using DNSCrypt Relays increase the chances of "cert refresh" failures.

dnsleaktest.com should have never had seen my ip

Where did it show your IP? On its homepage? If so, as mentioned above, make sure the WireGuard is in Lockdown.

You would really have to check both emails i sent (both onone after another) and the logs on both mails to unravel the whole mystery but probably the last one would be engoug

OK. Will do and see if I find anything 👍

[ignoramous]

(absent confirmation whether WireGuard was run in Advanced mode with Lockdown turned ON, I'm closing this... please feel free to reopen or ask to reopen).

[jfdl1991]

Sorry , i got sick.
So well i guess zero rethink might explain that. I didn't know sorry.🙇🏻‍♂️

And yeah it was not on lockdown mode😳. I just thought (and probably I'm not the only one) that all apps selected on a given wiregurard would always be routed trough that wiregurard as long as that wiregurard is on; and i thought lockdown mode was to ensure tha the apps only go trough that wiregurard and if that when that wiregurard it's off then the apps just couldn't connect if that wiregurard was not available.
I just didn't imagine that i needed to have lockdown if i wanted to prevent leaks. I'm sorry.🙇🏻‍♂️

And yeah on dnsleaktest webpage my real ip showed. Idk if something wierd happened like if i maybe lost mobile data connectivity for a second or something. I was standing still and with wifi signal available at home but maybe i got the luck to get a network change just the second i was doing the dnsleaktest 🤷🏻‍♂️

As for dnscrypt thanks for explaining 👍🏻👍🏻 I imagined that the app had to reach for certs using fallback dns at the beginning but I didn't know it would be stuck on the cert refresh fail for hours. And indeed after trying a lot, I think Relays make the cert refresh failures more common as you say.

That's too bad cause ODoH servers and proxies are very few and geographically away. After i read this i changed to ODoH and i only have seen fallback 🦆 when app crashes and on rethink app on the logs. But of course I get really slow speed (850-1100ms) due to my location.

It would be really nice if you could put some sort of notification when cert refresh failures happen so dnscrypt users can restart the service manually (or if you can catch them just autorestart the service if cert refresh failures persist for x time) 🥲.
Dnscrypt It's an awesome protocol when used with anonymous relays as good as odoh (IMHO it's even better) with broader server locations distribution and more relays than odoh ... At least untill odoh has broader implementation irl. There are very few odoh proxies and not many servers. Also one can find more dnscrypt target servers that directly have ads/malware filtering and with anycast like Adguard (which is already as a default on rethink). I've only found 1 odoh server that has filtering 😔 in the other side of the world.

For the moment being I'll stay with ODoH even with the slow speed.
Sorry for the inconveniences.

I hope you can still rescue some useful info from those lucky logs. Probably doesn't happen very often but I hope is useful somehow for making your software better.