[Proposal] DoH and NextDNS upstream nameserver support

[GustavoKatel]

Summary

Add DNS-over-HTTPS (DoH) and NextDNS as upstream ns_type values alongside the existing UDP nameservers. Admins attach DoH endpoints or NextDNS profiles to a Nameserver Group through the same REST API; the client speaks RFC 8484 to them. For NextDNS, queries carry the local peer's FQDN as the device identifier so devices appear by name in the NextDNS dashboard.

Related: #2232 (DoH/DoT support — this proposal covers the DoH half), #3711 (NextDNS profile compatibility).

I have a working implementation and would like to validate the design before opening the PR. AI assistance was used during development.

Branch: GustavoKatel/netbird:doh-upstream-target-refactor — 8 layered commits, each reviewable independently. Diff vs netbirdio:main.

Proto change

NameServer message gets one optional field:

message NameServer {
  string IP = 1;
  int64  NSType = 2;
  int64  Port = 3;
  string URL = 4;  // new: full https URL for DoH; profile/config ID for NextDNS
}

Older clients ignore an unknown NSType (they already log skipping nameserver … this peer supports only udp and treat the group as empty). Purely additive.

REST change

Nameserver schema:

• ns_type enum gains "doh" and "nextdns"
• new optional url field
• ip and port become optional (still required for udp)

Per-type validation:

• udp — requires ip + port
• doh — requires url (full https://… endpoint)
• nextdns — requires url (just the profile / config ID)

Client implementation

• upstreamRace is reshaped from []netip.AddrPort to a slice of typed targets carrying NSType + AddrPort + URL, so one race can hold mixed UDP/DoH entries without colliding on the zero AddrPort. The per-target health map is rekeyed accordingly.
• A shared dohClient performs the HTTPS exchange. The per-platform UDP impls (upstream_general, upstream_ios, upstream_android) are unchanged.
• For NextDNS, the URL is built as https://dns.nextdns.io/<config_id> and per-request HTTP headers carry X-Device-Name=<FQDN> and X-Device-Id=<short hostname>, matching the convention used by nextdns/nextdns.
• The DoH HTTP transport dials through nbnet.NewDialer() — the same per-platform "bypass the netbird tunnel" dialer netbird already uses for grpc/signal/STUN — so the HTTPS connection to dns.nextdns.io doesn't loop back into the resolver.

Bootstrap resolution

DoH endpoint hostnames can't be resolved through the OS resolver once netbird has taken it over — the lookup would loop. The client bootstraps from hostManager.getOriginalNameservers() (the same pre-takeover snapshot that already backs PriorityFallback), with our own DNS service IP filtered out. No hardcoded fallback resolver; if the bootstrap list is empty the client fails the DoH exchange and the race moves on to the next upstream.

Platform support

DoH coverage matches what nbnet.NewDialer already provides per platform — no new platform abstraction introduced:

Platform	DoH	Notes
Linux desktop	✅	fwmark bypass
macOS	✅	IP_BOUND_IF to underlying interface
Windows	✅	IP_UNICAST_IF
Android	✅	VpnService.protect()
iOS	⚠️	works in default split-tunnel; fragile in full-tunnel or if a routed prefix covers the DoH endpoint IP — same constraint as the existing UDP path for public IPs
FreeBSD	⚠️	no tunnel-bypass mechanism on FreeBSD today; same constraint as UDP
JS / WASM	❌	http.Transport would need to route through netstack; out of scope for v1

Backward compatibility

• Wire format: proto field + REST enum/field additions are additive.
• Older clients receiving a doh/nextdns nameserver in their network map log a single warning and skip that entry, treating the group as if it were empty — same code path as today's "non-UDP" check. Peer keeps working with whatever other nameservers it had.

Verified

• End-to-end against a real NextDNS profile via a 2-client Docker rig — both peers appear in the NextDNS dashboard tagged by FQDN.
• Cross-compile clean for Linux, macOS, Windows, Android, iOS, JS/WASM.
• Unit tests for the model parser, REST validation, race-target reshape, DoH wire format, header injection, and bootstrap callback wiring.

Open design questions

1. Enum values — doh and nextdns as separate types, or generic doh only with NextDNS-vs-other detected from the URL on the client?
2. Proto field shape — URL overloaded as "full URL or config ID depending on NSType", or a separate field per type (e.g. URL + ConfigID)?
3. NextDNS device identification — happy with X-Device-Name = FQDN, X-Device-Id = short hostname (deterministic, no per-peer config), or do you want admins to template the device-name string?
4. Bootstrap escape hatch — hostManager.getOriginalNameservers() is enough for normal operation. Should we also expose an admin-level bootstrap_servers field on the management config as an explicit override, or leave that as a follow-up if anyone hits a corner case?
5. Anything else you want covered before the PR opens?

I'll wait for design feedback before opening the PR.

[linear-code[bot]]

NET-1252

[dasunsrule32]

Needs DoT as well