iOS 16 compatibility

[agent-purple]

Prerequisites

• I have checked the Wiki and Discussions and found no answer

• I have searched other issues and found no duplicates

• I want to report a bug and not ask a question

Operating system type

Linux, Other (please mention the version in the description)

CPU architecture

AMD64

Installation

GitHub releases or script from README

Setup

On one machine

AdGuard Home version

0.713.13

Description

I am running AdGuard Home in a Proxmox Container (on Debian Bullseye) since a long time without any issues.
So first of all thank you for this great product.

Last week I updated two iPhones (X and 12 Pro) to iOS 16. Since then I am facing the issue that AdGuard Home does not work properly together with the new iOS.
When I wake up a device it takes up to 20 seconds until it gets a response for a DNS request (i.e. via safari or another app).
After a lot of investigation I found, that this issue is likely related to IPv4. (Don't know why iOS prefers IPv4 DNS over IPv6)
This can also be reliably reproduced by turning wifi off and on again.
The issue does not exist for devices running iOS 15.6.1 or iPadOS 15.7!

Here some workarounds that helped me (hopefully temporarily) to come around this issue.
#1: I set up another DNSv4 server and configured it via DHCP to be used for my devices. The upstream for this temp. DNS server is AdGuard Home.
#2: I configured manual DNS settings on iOS only to use my AdGuard Home IPv6 DNS server. (This is not really a workaround.)

Let me know if you need further information like configs.

Edit: No private relay in use.

[ainar-g]

Hello and thanks for the report. I've seen some reports from iOS 16 users on Reddit who also had issues with DNS, and especially encrypted DNS. Here's some info that could shed some light:

1. Which protocol are you using? Plain DNS, DNS-over-TLS, or DNS-over-HTTP? Does changing the protocol fix the issue?

2. If it's DoT or DoH, does your device resolve the certificate server name correctly? If so, does removing A records help?

3. Do I understand correctly that your AdGuard Home listens only on IPv6 addresses? Any particular reason you don't want it to listen on both?

4. Enabling verbose logs could shed some light as well.

[agent-purple]

Hi Ainar,

Thank you for your answer.

AGH accepts connections on IPv4/6, but connections seem to be partially broken at the moment for iOS16.
My workaround is that I have a plain port 53 IPv4 dns on another machine and that seems to solve the issue for now. IPv6 is still served by AGH.
In my config I have encryption activated with an internal cert, which was not a problem in the past.

So far, not good, because I want to know the root cause of the issue.
I created a detailed log and this is what I found so far.

iOS 16
Open ports are published on request

;; QUESTION SECTION:
;_dns.resolver.arpa. IN SVCB

;; ANSWER SECTION:
_dns.resolver.arpa. 10 IN SVCB 1 dns.tld. alpn="h2" port="443" dohpath="/dns-query?dns"
_dns.resolver.arpa. 10 IN SVCB 1 dns.tld. alpn="dot" port="853"
_dns.resolver.arpa. 10 IN SVCB 1 dns.tld. alpn="doq" port="784"

Then some log entries I do not understand
2022/09/19 21:15:12.578650 828#146 [debug] github.com/AdguardTeam/dnsproxy/proxy.(*Proxy).ServeHTTP(): Incoming HTTPS request on /dns-query?dns?dns=AAABAAABAAAAAAABCWt3ZWxsa2>
2022/09/19 21:15:12.578748 828#146 [debug] net/http.StatusText(): Cannot parse DNS request from

Maybe "dns?dns" is the problem here?

Then some TLS requests are flying in
2022/09/19 21:15:19.562349 828#240 [debug] github.com/AdguardTeam/dnsproxy/proxy.(*Proxy).handleTCPConnection(): handling tcp: started handling tls request from 172.20.xx.xx:5>
2022/09/19 21:15:19.580554 828#240 [error] handling tcp: reading msg: reading len: remote error: tls: bad certificate
2022/09/19 21:15:19.585798 828#241 [debug] github.com/AdguardTeam/dnsproxy/proxy.(*Proxy).handleTCPConnection(): handling tcp: started handling tls request from [fd77:xxxx:xxxx:....>
2022/09/19 21:15:19.599970 828#241 [error] handling tcp: reading msg: reading len: remote error: tls: bad certificate

OK, the client complains about a certificate. That's correct because it is a cert signed by internal CA. Nevertheless the client trusts the CA root cert. Bug in iOS?

Seems that I cannot use DNS encryption anymore. Sadly the AGH admin interface is bound to this.
Feature request: Untie DNS encryption (via 443) from admin UI encryption.

Doing something similar on iPadOS 15.7
No HTTPS or TLS requests and therefore no issue

-> iOS 16 is forcing encryted DNS.
-> iOS 16 ignores trusted CA certificates -> Bug (imho)
As the AGH is only used internally it is way to complicated to install a public accepted certificate.
Therefore internal plain DNS is ok (for me), but UI should be accessed via HTTPS anyway. See feature request above.

// Agent Purple

[ainar-g]

Thanks for the investigation!

Maybe "dns?dns" is the problem here?

It could be. We have released a fix for that in v0.108.0-a.299+4fc045de on the Edge channel. Could you please check if that version works at least a bit better?

Feature request: Untie DNS encryption (via 443) from admin UI encryption.

This is already a feature request, and it's planned for v0.108. See #4671. This won't be easy, though, since both the UI and the DoH are typically on the same port.

Judging by the tls: bad certificate errors, the issue is most likely due to the certificate being self-signed. If I recall correctly, Android, too, doesn't like self-signed certificates for DNS servers. Perhaps there is a setting somewhere?

[wgentine]

Same problem here. Request is malformed:

2022/09/21 06:10:24.506276 1#1013 [debug] github.com/AdguardTeam/dnsproxy/proxy.(*Proxy).handleTCPConnection(): handling tcp: started handling tls request from [2a02:a456:da92:0:d820:7842:7264:ef2]:53841
2022/09/21 06:10:24.535422 1#1013 [error] handling tcp: reading msg: reading len: remote error: tls: bad certificate
2022/09/21 06:10:28.350858 1#1014 [debug] github.com/AdguardTeam/dnsproxy/proxy.(*Proxy).ServeHTTP(): Incoming HTTPS request on /dns-query?dns?dns=AAABAAABAAAAAAABBWFwcGxlA2NvbQAAAgABAAApAgAAAAAAAFoADABWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

More importantly, certificate is correct and is not self-signed. Error only is displayed for requests made by IOS16 devices.

Im using v0.108.0-a.299+4fc045de

[ainar-g]

@wgentine, your device has probably cached the previous DDR response. Try either changing the DNS settings back and forth or rebooting it.

More importantly, certificate is correct and is not self-signed. Error only is displayed for requests made by IOS16 devices.

The remote error part of the message indicates that it's most likely a client error. That is, that the iOS device considers AGH's certificate invalid.

Perhaps, it has something to do with the algorithms used to create the certificates? Apple not accepting some weaker key lengths, for example?

[wgentine]

Dear @ainar-g you are right. The DoH requests after some time stopped throwing errors.

However, connections to 853 (DoT) are still throwing remote error. I also thought that certificate could have been the issue as I use elliptical p384 certs but also with rsa-4096 I have got the same issue. Im almost sure that at this point this is an IOS issue. To stop impacting iOS clients, I have disabled DoT for now.

[ainar-g]

@agent-purple, @wgentine, if I recall correctly, the additional requirement that some DDR-capable clients have is that the certificate contain the IP addresses in its SAN extension. See this guide. Which is required as an additional validation that the DDR message sent over the plain protocol wasn't forged.

[wgentine]

Thanks @ainar-g, that's likely the case and letsencrypt doesn't support IPs in SAN (searched and found some feature requests in the past). So, it will be a self-signed or let DoT off.

[ameshkov]

Just in case, ZeroSSL supports IPs

[agent-purple]

I recently installed a letsencrypt certificate. It shows up with green success messages in the AGH config 👍
Sadly they do not support IP and no internal TLDs. But I found a workaround.
Requests from iOS seem to work better now, but sometimes it still stucks.
I will have a look at the detailled logs the next days and will keep you guys posted.

[agent-purple]

Some news from my side:
No chance to get DoT/DoH working in the local network without fancy firewall stuff.
No issue as I consider DNS requests within my network as uncritical (i.e. watch p**n if it makes you happy - no one will blame you ;) ).
But that makes the request #4671 more crucial.

Thank you all for your good support.
From my perspective you can close the ticket.

Conclusion:
iOS 16 now supports encrypted DNS, which is good.
iOS 16 allows the creation of profiles that use encrypted DNS. The downside is that this restricts VPN usage as only one profile can be active at a time.
iOS 16 supports trusted internal CAs, but not all other devices (in SME and home networks) support it.
With reasonable effort encrypted DNS cannot be implemented in SMEs.

[wgentine]

@agent-purple That's curious. I managed to get DoH running with ios16 devices seamlessly without fancy stuff. Using a letsencrypt certificate but as I said before, doesn't support IP addresses in SAN which seems now to be necessary for DoT. As I'm looking for alternatives, I've disabled DoT port which also removes the respective entry from DDR response. IOS16 devices will then default to DoH.

[agent-purple]

Not so curious if you ask me.
What you have discovered is that DoT does not work, because the certificate is being validated correctly according to standards.
on the other hand the DoH certificate seems not to be validated, which contradicts the purpose of using certificates.
I assume that validaten will be activated later (which basically makes Sense) and then the issue will come back.

Are you using an own CA?
What I did not check so far is if this would work if I add local IPs as SAN to my CA (self) signed cert.

[wgentine]

@agent-purple Im using a proper letsencrypt certificate, not a self-signed cert. The only thing out of compliance so far is that I cannot have a IP in SAN with letsencrypt. Im looking for alternatives. This is only enforced for DoT, not for DoH hence DOH works without any issues.

[agent-purple]

Hi!

Some further test with DoT inactive (thank for the hint). iOS behaves strange. Most DNS requests via IPv4 instead of IPv6... and plain port 53,,, :(

Sometimes I see a request on HTTPS port.
Anyway... DoH does not check any certificate. I use now my CA signed cert.

This message is confirmed to be corrected. (I did not not install any testing version, because dunno how to -> was using default installation)

github.com/AdguardTeam/dnsproxy/proxy.(*Proxy).ServeHTTP(): Incoming HTTPS request on /dns-query**?dns?dns**=AAABAAABAAA...
net/http.StatusText(): Cannot parse DNS request from

Not sure if this is causing a little bit laggy behavior sometimes.

[wgentine]

@agent-purple some devices reads DDR config to select best suitable config some others not, hence some will come via 53, others via DoH. In regards to ipv6/ipv4, it also depends on the client, DHCP config, which addresses you are publishing etc...

This parse error is the one we have been discussing in the beginning and has been corrected in the edge channel. Its a bug.

[gokou340]

Was there a resolution to this? I've had issues ever since I moved to iOS 16. My iPhone constantly asks if it should switch to cellular data because wifi is not working. Switching to one of my windows DNS servers resolves all these issues. I've especially noticed issues with my MFA app when I get a push notification to accept, the app says that it is taking to long to get a response from the internet.

I'm using the latest version: v0.107.13, and am using a Let's Encrypt certificate.

EDIT: is the solution to use the version in the Edge channel?

[agent-purple]

Yes you can try the edge channel version.
What definitely works is to disable DoT or encryption as a whole.

[ameshkov]

@ainar-g it seems we should file a bug report to Apple about this since the issue seems to be causing lots of headache.

The short plan would be:

1. Test this on our side. Generate a ZeroSSL cert with IP and check if iOS 16 properly upgrades to DoT (I'll try to do that on monday or tuesday).
2. If this is indeed the issue, we should change the DDR implementation code and not supply DoT record when the certificate does not contain IP sans.
3. Finally, file this bug to Apple (I'll do that, an Apple developer account is required to file bug reports to them).
4. I am not sure if we should roll out a hotfix after that. Should we?

[ainar-g]

@ameshkov, it's up to you. I feel like since DoH is still available, it's not as critical, but we'll see.

[ameshkov]

@ainar-g let's decide after testing it.

[privacy-advo]

Do you have an update? (Postponed the iOS 16 upgrade.)

[ainar-g]

Apologies for the lack of updates. Here's what we have found out or done:

1. The URL path template in the DDR response was invalid. We've fixed that in v0.107.14.

2. Some DoT clients require the IP addresses of the DNS server to be put into subjectAltNames of the server's certificate. At the very least we'd need to warn the user if their certificate doesn't have those on the Encryption settings page. @EugeneOne1 is on it.

3. If the certificate doesn't contain IP-address SANs, we may also want to exclude the dot entry from AdGuard Home's DDR response, so that such devices don't get confused. This is either on me or on @EugeneOne1.

4. There may be some other issues with iOS, but we'll only be able to tell more clearly once we handle the present issues and go through another round of testing.

[wgentine]

@ainar-g, Have been doing some tests and realised that why querying adguard-home with kdig:

kdig -d @192.168.0.1 +tls-ca www.google.com
will fail, since 192.168.0.1 isn't in the letsencrypt certificate, however, if I do:

kdig -d @dns.example.com +tls-ca www.google.com

it will work since its requesting via fqdn.

Having said that, is there any way to announce DDR with FQDN instead? or does it break RFC?

[mxwi]

How can I disable DoT/encrypted DNS?

On iOS or AdGurd Home?

greetz

[ameshkov]

The current warning style is rather problematic IMO, it looks like an error while it is not.

I suggest removing this warning completely as it only confuses people and adds no value. The issue is specific to DDR and already handled by c139287.

[ameshkov]

@wgentine

Having said that, is there any way to announce DDR with FQDN instead? or does it break RFC?

This is exactly how we do it. However, iOS 16 seem to be requiring having IP addresses in the certificate regardless.

[ainar-g]

@ameshkov, apologies for the long wait. I agree that the UI looks bad, but I wouldn't want to remove it completely. Rather, I'd like it to be a more neutral colour, if it's really a warning and not a critical error. I wouldn't want to hide the information regarding the behaviour of AdGuard Home, especially if it depends on the user input.

[Gandulf78]

Hi all.
My iPhone with iOS 16.3 still do not connect automatically to wifi with v0.108.0-b.26. I need to deactivate / reactivate wifi to get it connected.
Where are you with your investigations on the compatibility with iOS 16.3?
Thanks.

[gregtwallace]

I'm not sure where this is dev wise but a workaround that resolves the random delay is to set blocking mode to 'Custom IP' and then set it to local host (127.0.0.1 & ::1).

I have no idea why this works or what Apple did to iOS but my iOS devices haven't had an issue since making this change.