Getting 504 error with OIDC through Authentik

[nnagibator228]

I'm trying to connect Authentik OIDC to Synapse Matrix Server.
Now I'm stuck with 504 error while Synapse trying to read authentik url with configuration for OIDC

2023-07-26T14:43:12.072234446Z 2023-07-26 17:43:12,072 - synapse.storage.txn - 842 - DEBUG - _handle_new_device_update_async-0 - [TXN END] {get_uncoverted_outbound_room_pokes-b} 0.001296 sec
2023-07-26T14:43:12.119714015Z 2023-07-26 17:43:12,119 - synapse.http.client - 423 - INFO - sentinel - Error sending request to  GET https://auth.authentikhost.com:443/application/o/matrix/.well-known/openid-configuration: RequestTimedOutError 504: Timeout waiting for response from remote server
2023-07-26T14:43:12.122362364Z 2023-07-26 17:43:12,119 - synapse.app._base - 269 - CRITICAL - sentinel - Error during startup
2023-07-26T14:43:12.122406404Z Traceback (most recent call last):
2023-07-26T14:43:12.122413995Z   File "/usr/local/lib/python3.11/site-packages/synapse/handlers/oidc.py", line 140, in load_metadata
2023-07-26T14:43:12.122418634Z     await p.load_metadata()
2023-07-26T14:43:12.122422323Z   File "/usr/local/lib/python3.11/site-packages/synapse/handlers/oidc.py", line 588, in load_metadata
2023-07-26T14:43:12.122426154Z     return await self._provider_metadata.get()
2023-07-26T14:43:12.122429944Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-07-26T14:43:12.122433654Z   File "/usr/local/lib/python3.11/site-packages/synapse/util/caches/cached_call.py", line 136, in get

My Authentik host uses Let's Encrypt cert. At the first time i faced this problem i've imported the Let's Encrypt R3 root chain (which my authentik cert is signed by) to container's root CA directory via ca-certificates (Here are my Dockefile commands for that:) and also set path to it with SSL_CERT_FILE env like its mentioned here

ADD https://letsencrypt.org/certs/lets-encrypt-r3.pem /usr/share/ca-certificates/mozilla/letsencrypt.crt

RUN chmod 644 /usr/share/ca-certificates/mozilla/letsencrypt.crt && echo "mozilla/letsencrypt.crt" >> /etc/ca-certificates.conf && update-ca-certificates

But the problem didn't gone and the error remained identical.

BTW if I set use_insecure_ssl_client_just_for_testing_do_not_use: true parameter in my homeserver.yaml file, I can get all the configuration work perfectly.

As I assume, its a certificate issue related to Synapse Docker container. I suppose that Let's Encrypt is trusted enough authority to be included in Synapse docker container so there would be no need to import it manually.

As extra info:

i've also tested curl inside the container in multiple conditions:

1. When container is set with use_insecure_ssl_client_just_for_testing_do_not_use: true in ```homeserver.yml`` i can curl uri with configuration with no errors (and the Synapse itself starts & works fine)
2. When container is set only with imported Let's Encrypt cert in root CA i can also run curl fine with no need in -k flag
3. When container is set with imported Let's Encrypt cert & SSL_CERT_FILE env i can also reach the authentik uri with curl
4. And when no certificate and parameter configured i can curl authentik only with -k flag
   (in 2., 3. & 4. container fails after a second of working because the described error, but i've managed to test curl with command like docker stop 28131c0b44de && docker start 28131c0b44de && docker exec -it 28131c0b44de curl https://auth.authentikhost.com/application/o/matrix/.well-known/openid-configuration

I can provide & describe all my configurations used if needed.
I'm trying to figure out the problem for several days now.

[MatMaul]

You should check more carefully on your side because I am really doubtful we have a trouble with LE certs since a good part of the web uses it.

docker run --entrypoint=bash -it ghcr.io/matrix-org/synapse:v1.88.0
curl -v https://letsencrypt.org/

Works fine, and the presented certificate is signed by LE R3.

You should try to do curl -v https://letsencrypt.org/, it it works it probably points to some misconfiguration of the cert in the frontend proxy.

I am closing this for now, please re-open if needed.

[nnagibator228]

@MatMaul , So if i could get the page inside container with curl, but Synapse server couldn't, it's related to Synapse SSL client troubles. I repeat, after using use_insecure_ssl_client_just_for_testing_do_not_use which disables SSL client verification (as i understood) everything works fine, so there is no need for testing through curl and something like that.

Also the 504 error when the host is considered suspicious by SSL client seems like a bug itself.

There are some related issues that also hadn't been handled yet:

#8741

#9662

I would also mention that I'm also using LDAP integration from the same identity provider with same certificate as in OIDC endpoint, and it works totally fine (i guess so, cause I'm using STARTTLS mechanism and Synapse's logs related to LDAP are not enough verbose even after

loggers:
  synapse.storage.SQL:
    # beware: increasing this to DEBUG will make synapse log sensitive
    # information such as access tokens.
    level: INFO
  ldap3:
    level: DEBUG
  ldap_auth_provider:
    level: DEBUG

so i could exactly tell if certificate used in connection)

[nnagibator228]

Ive managed to handle this trouble.

When we started migrating to Rocket.Chat instead of Matrix, the same scenario happened, but with more detailed logging from Rocket.Chat, which helped me to identify the problem.

The problem was in inconsistent certificate chain, that where fixed on the Identity Provider-side

After the fix either Matrix Synapse & RocketChat worked correctly.

Suppose, this answer could help somebody, who's stuck with the same trouble scenario, but couldn't get detailed logging from Synapse 😊