Traefik doesn't correctly proxy .well-known on the identity hostname

[throwawayay]

Describe the bug

Have a fairly vanilla install, and migrated over to Traefik using the directions in the changelog.
I noticed that .well-known is probably missing a tag to be correctly reverse-proxied.

To Reproduce

Tested with 6a52be7

Confirmed failure with both self-check and the online federation tester.

TASK [custom/matrix-nginx-proxy : Check .well-known on the identity hostname] *************************************************************
fatal: [matrix.<server>]: FAILED! => changed=false 
  content: ''
  elapsed: 0
  msg: 'Status code was -1 and not [200]: Request failed: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:997)>'
  redirected: false
  status: -1
  url: https://<server>/.well-known/matrix/client
...ignoring

Online federation tester:

{
  "WellKnownResult": {
    "m.server": "",
    "result": "Get \"https://<server>/.well-known/matrix/server\": x509: certificate is valid for <hash>.<hash>.traefik.default, not <server>",
    "CacheExpiresAt": 0
  },
  "DNSResult": {
    "SRVSkipped": false,
    "SRVCName": "",
    "SRVRecords": null,
    "SRVError": {
      "Message": "lookup _matrix._tcp.<server> on 8.8.8.8:53: no such host"
    },
    "Hosts": {
      "<server>": {
        "CName": "<server>.",
        "Addrs": [
          "<public_ip>"
        ],
        "Error": null
      }
    },
    "Addrs": [
      "<public_ip>:8448"
    ]
  },
  "ConnectionReports": {},
  "ConnectionErrors": {
    "<public_ip>:8448": {
      "Message": "Non-200 response 404 from remote server"
    }
  },
  "Version": {
    "error": "Get \"matrix://<server>/_matrix/federation/v1/version\": x509: certificate is valid for <hash>.<hash>.traefik.default, not <server>"
  },
  "FederationOK": false
}

Matrix Server:

• OS: Ubuntu 22.04.1 LTS
• Architecture: amd64

Additional context

It doesn't directly impact me as I don't currently federate, just thought it would be useful to note. Happy to retry/debug as needed.

[gitayam]

same issue though i am using cloudflare dns, was working up unti the last update.

[spantaleev]

This is controlled by:

https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/0b9dc56edffb3f85deb2919a7d4e2f9f69c0fe78/roles/custom/matrix-nginx-proxy/defaults/main.ymll#L52-L55

It's wired to the old variable (matrix_nginx_proxy_base_domain_serving_enabled). Have you removed it, perhaps?

If you have , you'd neither need to bring it back, or you should toggle matrix_nginx_proxy_container_labels_traefik_proxy_base_domain_enabled to true.

Edit: toggling just matrix_nginx_proxy_container_labels_traefik_proxy_base_domain_enabled to true won't be enough, because it will let Traefik proxy, but it's matrix-nginx-proxy which still ultimately serves these files (for now). One really needs to toggle matrix_nginx_proxy_base_domain_serving_enabled to true for this, so that it's enabled on both sides.

---

In my tests, the Traefik configuration for matrix-nginx-proxy (the service which ultimately serves the well-known files and many other things, for now) in /matrix/nginx-proxy/labels looks like this:

traefik.enable=true

traefik.docker.network=traefik

# Base domain
traefik.http.routers.matrix-nginx-proxy-base-domain.rule=Host(`DOMAIN`)
traefik.http.routers.matrix-nginx-proxy-base-domain.service=matrix-nginx-proxy-web
traefik.http.routers.matrix-nginx-proxy-base-domain.tls=true
traefik.http.routers.matrix-nginx-proxy-base-domain.tls.certResolver=default
traefik.http.routers.matrix-nginx-proxy-base-domain.entrypoints=web-secure

# Matrix Client
traefik.http.routers.matrix-nginx-proxy-matrix-client.rule=Host(`matrix.DOMAIN`)
traefik.http.routers.matrix-nginx-proxy-matrix-client.service=matrix-nginx-proxy-web
traefik.http.routers.matrix-nginx-proxy-matrix-client.tls=true
traefik.http.routers.matrix-nginx-proxy-matrix-client.tls.certResolver=default
traefik.http.routers.matrix-nginx-proxy-matrix-client.entrypoints=web-secure

# Matrix Federation
traefik.http.routers.matrix-nginx-proxy-matrix-federation.rule=Host(`matrix.DOMAIN`)
traefik.http.routers.matrix-nginx-proxy-matrix-federation.service=matrix-nginx-proxy-federation
traefik.http.routers.matrix-nginx-proxy-matrix-federation.tls=true
traefik.http.routers.matrix-nginx-proxy-matrix-federation.tls.certResolver=default
traefik.http.routers.matrix-nginx-proxy-matrix-federation.entrypoints=matrix-federation

traefik.http.services.matrix-nginx-proxy-web.loadbalancer.server.port=8080

traefik.http.services.matrix-nginx-proxy-federation.loadbalancer.server.port=8448

I haven't tested if it actually works, but it looks like it should.

[gitayam]

matrix_nginx_proxy_base_domain_serving_enabled

Same issue but not using traefik. matrix_nginx_proxy_base_domain_serving_enabled is enabled, didn't realize this was old, checking documentation to find what replaced this

[spantaleev]

matrix_nginx_proxy_base_domain_serving_enabled is old in terms of "it's a matrix-nginx-proxy thing, which we've reused in the new Traefik setup". We're mostly reusing it, because it's ultimately matrix-nginx-proxy which hosts these /.well-known/matrix files, even if Traefik is used in front. In the future, this will change and we'll be able to rid ourselves of matrix-nginx-proxy.

matrix_nginx_proxy_base_domain_serving_enabled is still the variable that one should use for controlling reverse-proxying of the base domain regardless of whether they're using matrix-nginx-proxy (our default reverse-proxy, for now) or Traefik.

[throwawayay]

Confirmed, re-adding matrix_nginx_proxy_base_domain_serving_enabled resolves the issue. I misunderstood the announcement to not need any of the old variables, that's absolutely my mistake.

Thanks again!

[spantaleev]

Thanks for confirming! Hopefully b648d45 makes things clearer, so that others won't trip up on it too.