HTTP(S) connections (without DNS) to certain hosts are blocked (by ISP?) #7055
Comments
|
We should clarify a few basic things at the beginning. When using a VPN such as Wireguard (PiVPN is only an interface on the server for managing clients), all traffic between the VPN client and VPN server is always encrypted. It doesn't matter what it is. Therefore, all requests, including DNS, between your mobile phone and your server are encrypted. Communication between PiHole and Unbound is always unencrypted. However, this does not matter, as this only takes place on the server. DNS requests between Unbound and the upstream DoT server, on the other hand, are encrypted again via TLS protocol. Normally there should be no difference for DNS requests between a mobile phone via VPN or a client in the local network, as all requests first go to the PiHole and then to Unbound. The chain actually looks like this: Client > (VPN) > PiHole > Unbound > Upstream DNS (via DoT) Translated with DeepL.com (free version) |
I understand all that. Thanks for the clarification. So my current VPN traffic should be encrypted. Only the traffics from Pi-hole and Unbound is unencrypted. Then what exactly can I do to test and find out why my query fails to certain websites in my VPN? Yet my traffic in local network works as intended. |
|
Check whether all traffic from the VPN client to the VPN server goes through the tunnel or whether only a split tunnel is active. |
|
WireGuard (clients) have a setting Note that this is only a client-side setting, same as whether all traffic is forced through the VPN, requests to the server-side LAN or the server itself only. So it is all about how the clients are configured. Ah, and don't forget to allow "All origins" in Pi-hole for DNS queries. By default it allows queries only from the LAN/main interface of the server, hence blocks queries form any VPN interface. EDIT: Use |
From my Wireguard client conf Ignore the PrivateKey , PublicKey, and PresharedKey, I changed to hide the actual value. So I definitely have |
It's a full tunnel, which allows access to everything within the network. I can use I checked the public IP address and it's the same as the one from the Local Network, not the coffee shop public IP address or LTE Public IP. |
Hmm, then I do not understand what the issue is, when clients do in fact use that Pi-hole instance as DNS server through the VPN. If those are Linux/UNIX clients, can you check |
|
|
I mean on the client(s). |
RIP. it's an iPhone. |
|
The Termux doesn't work with Any other methods? |
|
From your mobile phone, you should see the DNS queries inside the PiHole log. Do you? Even for the website in question. If this is the case, DNS resolution is working fine. |
|
Yes, especially check whether you see a query log entry for those cases, where a website is blocked, while you expect it to be accessible thanks to the VPN DNS. Ah, and also check whether the related browser has DoH enabled, which would bypass the system DNS and VPN. |
@Joulinar Yes, I can access Pi-hole over the VPN, use the @MichaIng For sure, the DoH, I always check that one off within the browser, on Android 13 and later, inside its Settings, it has Private DNS. I made sure to turn that off first, not even Automatic. |
|
Question was if you see inside PiHole log the DNS request for the website not working. If this is the case, DNS resolution is not your problem. |
@Joulinar Pi-hole logged the query
Screenshot on my Tablet device named
On my PC, with
|
|
yes DNS resolution is working, so it's something else. Like the browser escaping the VPN tunnel |
And exactly like how my browser behaves when my query was unencrypted (without Unbound DoT) using 1.1.1.1 at port 53 so my ISP DNS rejects the query. So that's why I'm stuck here. Wondering if PiVPN is "leaking" the query. But I know it shouldn't be. |
|
It's not a DNS issue as you see the query inside PiHole. As well it is forward to Unbound according your PiHole screen shot. It's not the DNS query escaping but somehow the browser not using to VPN to access the website. |
|
I thought it was a fluke with Firefox. So I installed Vivaldi Browser for Android. I went into Vivaldi Settings > Privacy and security > and disabled I deliberately clicked on a Vivaldi bookmark added that I know for sure is in the blocklist. It blocked it successfully. But I tried with
I think you're right, @Joulinar .
The browser is clearly using the Pi-hole DNS inside the VPN, but the domains blocked by my country is not working inside it. |
|
Can you check whether the IP address And you are sure that Unbound is configured to use DoT? Compare with our docs: https://dietpi.com/docs/software/dns_servers/#__tabbed_2_5 Btw, on your PC |
on PC - LAN on Tablet (Android) - VPN
It's using the same IP address. The Cache is from the first query from PC.
Yes. I was troubleshooting another problem with domain end in
Yup. It works. Same with other 18+ sites and news sites that my countries blocked. |
|
Interesting, the LAN client seems to send a DoH query to Pi-hole, which is forwarded to Unbound, if I understand the "HTTPS" query type correctly? However, so the A type query is answered as intended, works for the LAN client, is cached as intended, and hence answered from cache to the VPN client, but does not work there. Confusing, indeed. The Android tablet is currently also located in your LAN, just using the VPN (unnecessarily, for testing), right? So that really only enabling the VPN makes the difference? |
I don't have DoH enabled for this network at all. Only Pi-Hole with Unbound DoT. I understand the difference between them. Like how Adguard-DNS / Adguard Home using DNS-over-HTTPS with their SSL cert and privatekey in Encryption Settings.
I have 3 different ways of connecting to the internet.
So I use the Tablet connected to Network number 2 and turn on Wireguard app to connect to PiVPN and it connected to Network number 1 and I open I tried the exact same thing with 4G/LTE network as above. Same result. Which led me to the Title of the issue "Leak DNS query over VPN connection". I know it isn't true. EDIT: changed # to number because Github pinged other issue numbers. |
|
PiVPN means a (regular) WireGuard server in your case. Might be important when you check for answers elsewhere. PiVPN is only the installer, and the CLI to create client configs, otherwise it has no effect on the VPN. With Maybe best to use |
output follow up output with which is expected because of the transparent DNS query. I will do the |
|
Wait. |
|
usually, DNS setting in DietPi itself has no relation to PiHole. And best practice is not to change to a local DNS server like PiHole or AGH. |
|
Background: I brought my Macbook Air to the coffee shop. It took a while to setup the Wireguard because of the newer Wireguard cannot be installed from App Store. I had to deal with brew, Wireguard-tools and, wireguard-go Macbook output Macbook output on Macbook requests after using REMOVED EDIT: I will remove the |
|
Hmm, so pornhub is not accessible on the Mac? Was it every accessible there? So instead of comparing VPN with no VPN, we should probably compare Mac and PC, as the PC is the only device where pornhub was ever accessible? |
True. But if I use Proton VPN, it will work.
Not sure what this comment means.
Ignore this. Test: Devices on Pi-Hole network with Unbound, 3 websites being tested - pornhub, steam, xhamster
Note: Earlier today, Xiaomi could not open I only have a laptop that is a Mac and it uses similar commands to Linux and I can move it around to different locations so that's why I chose it to test out VPN vs no VPN. |
Makes sense, as your ISP then does not see any traffic from your home to pornhub (IP).
I mean whether pornhub was ever accessible on the Mac (when not using ProtonVPN or any other public provider).
Good, so we know the (home) VPN is not the reason for the issue, but it also does not prevent it. And encrypted DNS does not resolve it either, since it is the HTTP(S) connection itself which fails. And for now, based on earlier EDIT: Okay, some more devices are affected, interesting that one Android works, the other one not. Also the DietPi server is affected, getting a 404 response with Can you run the traceroute on Windows like this: tracert www.pornhub.comAnd on the Mac and/or DietPi like this: apt install mtr
mtr www.pornhub.comand paste the results? I want to see whether the routes differ at any point. You can mask IPs/hostnames from your LAN, of course.
|
Never. I hardly ever use this laptop. It's an older Macbook. It has like 4 GB of RAM and Intel duo cores. Super slow. So I never used it to access
Wow. that's a cool application/software. I like it a lot. Thanks for introducing me to output They both seem to work. Another test |
|
After output for Macbook - without VPN on Pi-Hole network output for Macbook - with VPN on Pi-Hole network |
|
Extra tests from Macbook Test 3: mtr number 3 on network number 2 (not Pi-hole) no VPN - default DNS Test 4: mtr number 4 network number 2 with vpn no dns entry in conf file Test 5 : mtr number 5 network number 2 with vpn with dns entry |
|
Ah sorry, right, on Mac of course the Yeah, ISP DNS blocks *.pornhub.com, hence all ends at localhost, makes sense. With any other DNS, the VPN adds the VPN/Pi-hole server as first hop, otherwise has no effect. And also the routes on all devices without VPN are identical. So that is not the difference. Probably it is not even the ISP which blocks this in a way, but the TLS libraries used on those systems somehow have an issue in your particular case. But I am pretty much out of ideas how to further debug. Just one last test, on the DietPi system, which seem to have been affected as well. Can you test again with cp -a /etc/resolv.conf{,.bak}
echo 'nameserver 127.0.0.1' > /etc/resolv.conf
curl -L https://www.pornhub.com
wget -O- https://www.pornhub.com
mv /etc/resolv.conf{.bak,}In case of success, both return the full HTML document, which is huge, so do not wonder, and no need to paste that 😄. If it ends with Not sure whether someone on our forum has further ideas how to debug, or in general how requests can be (made) fail in such a way. Otherwise, probably people on some StackExchange site, like https://superuser.com/, have an idea. In case, it makes sense to leave the (home) VPN out of the game. It was a reasonable idea to test whether it helps, but we showed that it is irrelevant in regards of the issue. But that using an encrypted public DNS (like DoH in browser) is required in any case, as the ISP DNS blocks it, is of course important, and that using a public VPN provider fixes the issue on the affected systems, is relevant as well. So it is somehow these particular systems or clients sending the HTTP(S) request through this particular ISP route, having issues, while other systems/clients, sending the same request through the same route, succeed. An interesting case, so in case you're doing to further debug, I am interested about the outcome. You can ping me on superuser as well (same username). |
Output on DietPi
What now? I certainly could make a superuser account and ask a question there. I don't think anyone else other than you, @MichaIng is willing to debug/troubleshooting this far into an issue. Thank you so much for your help. |
Yup. Mac fails at the Coffee Shop too. Another problem you happened to remind me is my country has 3 major ISP so their DNS and DNS Filter implementation are all different. That could be why my results are varies. What a pain. All I want is to self-host and be in control of my data yet. To have privacy. Apparently it's too much to ask in my country. Super lame. |
|
Okay, so OpenSSL and GnuTLS both run into the same issue. So the TLS library is unrelated as well. EDIT: Fits to the 404/408 response you got with
There is no guarantee, of course, and on StackExchange sites it is a bit RNG and timing, whether someone with interest and knowledge sees the question or not while it is new/highlighted. But there are certainly people around willing to help/answer questions, with much more knowledge/experience than me. So it is worth an attempt, if you do not mind the time to create an account, read a bit into how to ask a question properly and sort/sum up the relevant results we found.
Well, DNS and probably packet filtering by ISP is not directly related to whether you can self-host things or not? You can self-host a VPN and a DNS for DoT or DoH perfectly fine. So you can bypass DNS filtering that way, or use DoH with a public provider instead. Just sadly it does not help with the HTTPS request issue, whatever is causing this. However, yeah, any kind of Internet filtering can be nasty. I was in China for a week, and it was interesting how many websites were blocked, not only |
MichaIng
added
Not a DietPi issue
External bug 🐞
MichaIng
changed the title
MichaIng
changed the title
|
@MichaIng I’m reading up on StackExchanger Superuser rules and tags. Which tags do you think this issue belongs to? Because I know their mods will remove issues or could straight up ban me if I tried to resubmit and commit a “spam” offense against their site. |
|
They closed it really quick with off-topic. The 2nd comment gave me some thoughts. |
I guess you would need to use |
|
@Joulinar |
|
|
|
Yes. I understood that. The pi-hole Docs did mentioned the https upstream that is using Cloudflare DNS. Of course I could change it over to Quad9. I'm just making sure I'm learning all of the other options as well. |
|
Update: It seems that after I switched everything to Quad9, from the DietPi-Config, to the Now I'm not sure what to think now. The issue can be closed I guess. |
|
They are wrong: Cloudflare does not respect country filtering, at least not in your case. We do already know that clients get the correct IP from Cloudflare, and the connection is blocked on HTTP(S) connection level to/from the true host instead. Not sure whether it is really OOT at superuser, but I wouldn't know a better StackExchange site in this case. I left a comment to ask about this, and add the relevant info, let's see ... |
Good news.I went nuts after drinking too many cups of coffee. So I did a factory reset on an Android device (Xiaomi) to reduce all variables. I installed They all work now. No delays. Instant content delivery.I formed a hypothesis it could the Adguard.apk and maybe its the adguardca.crt. Background: To test out the hypothesis, I removed the CA cert from my Samsung tablet and uninstalled the Adguard.apk. Reboot the device. Install Big progress. Now that solves half the issue. Because MacOS and iPhone are still failing to open Either Pornhub SSL/TLS certificate has a problem verify its identity with Adguard CA cert. Not too sure on the traffic and route from the ISP side. Do you have any methods to debug related to SSL/TLS cert instead? I can finally close this issue. Looks like I will have to remove all Adguard app and its CA from the trusted CA. EDIT: I forgot. I need to test this over VPN as well. Before closing this issue. |
|
Okay that is a very good explanation: Note that HTTPS filtering/inspection means that AdGuard decrypts, filters and in case blocks all traffic/packets between you and the website you want to access. This includes porn, online banking (aside of some hardcoded exceptions), etc. So I am quite sure that this AdGuard HTTPS filter/firewall rates some pornhub content as malicious, block it, hence replaces it with these weird 404/408 HTML documents I was wondering about. The only thing I am wondering about, is why the DietPi system was affected by this as well. The way this is implemented, basically compromises HTTPS at its core. It is like a MITM, which HTTPS normally prevents against, but in this case you are explicitly asked to trust the AdGuard certificate which re-encrypts traffic. I mean, AdGuard software is (mostly or completely?) open source, and we have no reason to mistrust them (now). But IMO the principle to break the essential security of HTTPS in the name of security is just wrong. And I am not alone with this opinion. We are using Cloudflare as proxy/CDN/cache/DDoS protection for HTTP(S) traffic to our website, which basically does the same. However, it does so transparently, i.e. resolving |
This might explain it. If this is true, when I reinstall Adguard.apk, install the adguard-ca.crt, I turn off the dns filtering and firewall, it should allow my devices to start opening the HTTPS to the 5 websites I mentioned above and content delivery should work as intended. But the weird part in this is why
I will email their support and create a ticket asking for clarification on their ca crt and dns filtering. Decrypting my HTTPS traffic on their end and see my query is obvious concerning. But I understand why it had to be done. How else Adguard suppose to know if that specific traffic needed to block and not delivery to client device.
Viewing the traffic is just one problem. If they are storing customer traffic and still linking it to me, and my government happens to required them by law to hand it over is another problem. I've looking for ways to encrypt traffic between client (PC) to DNS resolver (Pi-hole or Adguard Home). The internet pointed me to |
|
No need to ask AdGuard about that: It is intended behaviour, to block ads and/or block traffic to malicious sites. But yes, you are right, Filtering will be done locally on an automated level, hence your traffic is, of course, not sent to AdGuard servers where they could read it. Though, not impossible that bits of information are sent to be checked by a cloud block list or heuristic. At least in general this is how some anti virus software do/did it. Though for ads filtering such is surely overkill an not done. So for me, the problem is not concerns or mistrust against AdGuard now, but the method of decrypting HTTPS traffic, and that way breaking end-to-end encryption as well as authentication (installing a generally trusted cert for a website that is not coming from this website), in general. Actors who are trustable now can change, be hacked, or whatever, so having such method installed and generally accepted as "security enhancement" IMO bears a general risk for the future. There may be edge cases, but not as by default enabled feature for common end user software, IMO.
Downstream DoH or DoT with AdGuard Home and/or WireGuard/VPN with Pi-hole/AGH as you were already doing. |
|
I have since reinstall I did a bit more research. In Adguard app Settings > Filtering > Network > HTTPS Filtering > HTTPS-Filtered websites > turn OFF This is the real reason why 3 out of the 5 websites mentioned above broke. Not sure why DietPi itself broke either. I guess I can close this issue now. Thank you for your time and knowledge @MichaIng and @Joulinar . This has been a very productive week and a ton of learning experience for me. |
Creating a bug report/issue
Required Information
cat /boot/dietpi/.versionecho $G_DISTRO_NAME $G_RASPBIANuname -aecho $G_HW_MODEL_NAMEor (EG: RPi3)Additional Information (if applicable)
PiVPNFreshly installed
Not sure.
echo $G_HW_UUIDSteps to reproduce
install Dietpi image on RPi
apt update && apt upgrade -y
dietpi-launcher
install Pi-hole, Unbound, activate DoT with the conf from Dietpi Docs
install pivpn, wireguard, setup DNS support with Pi-Hole ,add user, scan QR code
add conf file to a device (e.g: an Android phone) to connect to pivpn OUTSIDE of the network
check for the websites that is KNOWN to be blocked like 18+ sites, see if it returns query on unbound, load the content on the browser within my Local Network, meaning the DoT is working because my DNS query is encrypted and returned to Unbound with no issue.
Turn on Wireguard VPN on the device in Step 6
check if the known blocked website works and contents are loaded in the browser.
Expected behaviour
It should load all the contents of the website in the browser.
Actual behaviour
The browser will not load any content and return an error like "Unable to connect". Like an unencrypt query was rejected by the ISP DNS.
Extra details
I can use Pi-Hole with Unbound (activated DoT (DNS over TLS)). I can browse normally to any websites I want other than the block list inside Pi-Hole.
My country required ISP to blocks certain websites from their own DNS. So ISP will allow your network to query the website, but it will reject the content of the website.
Help me to troubleshoot if PiVPN is leaking the query because when I access my network over PiVPN, it behaves exactly like when I access the blocked websites WITHOUT Pi-Hole and Unbound DoT which mean my query to unencrypted and intercept by my ISP.
My question is:
How do I check/troubleshoot PiVPN whether it is encrypting my query to Pi-Hole and Unbound on port 5335 or leaving the query unencrypted like port 53?
The text was updated successfully, but these errors were encountered: