External integration DNS gotcha!

[SteveDockar]

This isn't a Reitti specific problem but as it has just tripped me up I thought I'd call it out here in case anyone else hits the same issue.

TL;DR - Docker internal DNS does not like CNAME records / chains and won't correctly resolve them from external DNS

The Issue

I have Reitti running on top of TrueNAS Scale and it is now working fine with Authentik OAUTH and Immich integration, both of which run on other hosts on my network. Unfortunately, to get to this point, both of those external connections initially failed even though I had successfully added MQTT connections to my existing broker, and that had worked first time.

The Cause

Adding Authentik, and then Immich did not work first time, but the cause was the same in both cases. As both are behind a Caddy proxy, on my local AdGuardHome instances, their DNS records are CNAMES to the A record of the Caddy instance. Docker integrated DNS does not seem to like CNAME chains, and apparently, this is a longstanding issue. running nslookup immich.<my-domain>.net in the reitti container resulted in an NXDOMAIN and the actual result as you can see here:

/app # nslookup immich.<my-domain>.net
Server:         127.0.0.11
Address:        127.0.0.11:53
** server can't find immich.<my-domain>.net: NXDOMAIN
Non-authoritative answer:
immich.<my-domain>.net        canonical name = caddy.<my-domain>.net
Name:   caddy.<my-domain>.net
Address: 10.113.1.150

The (well "a") Solution

I added "extra_hosts" entries to my docker-compose.yml file inside the reitti service block like this:

reitti:
    image: dedicatedcode/reitti:latest
    ports:
      - 8080:8080
      - 15180:8080
    dns:
      - 10.113.1.53
      - 10.113.1.54
    extra_hosts:
      - "authentik.<my-domain>.net:10.113.1.150"
      - "immich.<my-domain>.net:10.113.1.150" 

N.B.

• The extra_hosts entries are added to the container's /etc/hosts file so the docker DNS isn't needed to resolve them. I have to remember they are in there if I ever change the DNS entries of course, which is not ideal, but I think this is the "least worst" option in my particular circumstances.
• I could have replace the CNAMES with A records pointing directly to the Caddy IP address but I don't like multiple A-records with the same IP addresses. It's valid of course, but that's supposed to be what CNAMES are for I guess...
• Adding the external DNS entries doesn't help with the CNAME problem, but it did allow Reitti to resolve mqtt.<my-domain>.net first time as that's an A-record so they are worth setting, they just can't completely fix the issue.

I hope this helps somebody who might stumble over the same issue.

[dgraf-gh]

Hi @SteveDockar, thank you for the detailed explanation! This will definitely come in handy for anyone working with a similar setup.

I hadn't considered CNAMEs in this context before. Currently, I'm using Cloudflare to route my homelab wildcard subdomain (*.lab.dedicatedcode.com) to my home server (192.168.1.120). A Traefik reverse proxy running on that server then routes incoming traffic to the correct container based on Docker labels.

This approach has been a huge time-saver. I can just deploy a new container with the proper labels and skip manually creating DNS records for every single service.

That said, it's still surprising to me that CNAME resolution doesn't work as expected here. Given how long CNAMEs have been a core part of the DNS standard, you'd think this would be more straightforward. Either way, thanks again for clarifying!

[SteveDockar]

Hi @dgraf-gh, I used Traefik for a time too, but I hit cases where some reverse proxy scenarios were really complex (it might have been websockets or similar) and Caddy "just worked", so I stuck with that. As I have multiple LXCs and VMs and each has a Caddy instance fronting the services on that host, I need to have DNS entries for each, steering towards the correct Caddy instance, so the A-records for the Caddy hosts and CNAMES for the services makes it so much easier if I move a Caddy host, or switch a service from one host/Caddy to another.

I also use Tailscale/Headscale, so pretty much everything is "on-net" or "internal" here, so my internal DNS is more important than when I used external access to services for when I was out and about like your Cloudflare scenario.

Again, absolutely no fault of yours Daniel, I initially thought it was my docker config which was wrong somewhere, but when I ran nslookup in the container and saw the answer and the NXDOMAIN I understood what the issue was.

I should also have mentioned, the v4.0 enhancements are brilliant, thank you as always for your hard work here! 😃

[dgraf-gh]

but I hit cases where some reverse proxy scenarios were really complex (it might have been websockets or similar)

Oh I feel the pain 😆 i activly shy away of those. I remember trying to setup vault or bitwarden and it was a pita because of that.

should also have mentioned, the v4.0 enhancements are brilliant, thank you as always for your hard work here! 😃

Thank you, that means a lot to me. It is so nice to actually get positive feedback after all the labour went into it without knowing how it will be perceived in the end.