Can't connect to server if Let's Encrypt intermediate are certificates missing

[DuvelCorp]

My Config

JellyFin 10.7.7 running on Debian in my DMZ.
In my main gate (firewall), the port 443 is open and HTTPS traffic is redirected to a reverse proxy in the DMZ (Debian/Apache and not the same server as the Jellyfin instance). Only HTTPS is allowed from WAN.
The reverse proxy handle HTTPS decryption and redirection in HTTP on the proper server/port of the DMZ, depending on URL. The cert (LetsEncrypt renewed 2 days ago) is thus on the reverse proxy.

Say the URL is https://mydomain.name/jellyfin
Config in the reverse proxy is :

ProxyPass /jellyfin http://<the server IP adress>:8096/jellyfin
ProxyPassReverse /jellyfin http://<the server IP adress>:8096/jellyfin

I also have a reverse DNS in the firewall that take care of redirecting my own https://mydomain.name/jellyfin from LAN to the DMZ.

And finally, got several friend users connecting to my Jellyfin (thus from WAN) mostly everyday.

With that said :

• Everything is working perfectly from WAN and LAN from Browser clients (Firefox, Chrome, Safari, Edge...)
• Got a raspberrypi/openelec box running Jellyfin addon in my LAN, configured with the servers DMZ's IP, not URL, and it is working fine too.
• Android app is not working for any of us, whether its in WAN or LAN, unless we apply a stupid trick.

The bug (and the trick)

When we try to connect with android app, (from WAN or LAN), with URL https://mydomain.name/jellyfin, it says it can't reach the server.
I can see the 443 packets flowing through the firewall to the reverse proxy, but I can't see anything landing in the Jellyfin logs (I might have missed something).
I can make it working, but only if I apply the following trick: first establish a connection to https://demo.jellyfin.org/stable/, which always work, then connect to my own server. In this case it works 100%. But once the app got disconnected, the trick should be applied again.

Got the issue with latest Jellyfin Android App from my phone (LineageOS 18.1 / Android 11) , and a friend that has a fresh MI Box has the exact same issue, and the trick works for him as well. When he is finally logged, he says that everything is running sweet and that the performance is great.

Here we are. I searched a lot before discovering the trick on reddit, and only this worked.
So at this stage I am unable to tell if the issue really is in your Android App, or in my pipes. However the reddit answer proves that I am not the only one to experience this issue and that its there for some months.

I can do some tests and provide you with more info and logs if needed.

Cheers and keep up the great work 👍

Logs

No response

Application version

2.4.0

Where did you install the app from?

Google Play

Device information

Moto G7

Android version

Android 11, LineageOS 18.1

Jellyfin server version

10.7.7

Which video player implementations does this bug apply to?

• Web player (default)
• Integrated player (ExoPlayer)
• External player (VLC, mpv, MX Player)

[nielsvanvelzen]

What is the response you get when going to https://mydomain.name/jellyfin/system/info/public?

[DuvelCorp]

I don't want to publish my real server name and folder or I will be spotted instantly with a google search ;)

So let's say I called it "Netflix", and thus my URL is https://mydomain.name/netflix
it gives :

{
"LocalAddress":"http://<edited : no need to be public>:8096/netflix",
"ServerName":"Netflix",
"Version":"10.7.7",
"ProductName":"Jellyfin Server",
"OperatingSystem":"Linux",
"Id":"<edited : no need to be public>",
"StartupWizardCompleted":true
}

[nielsvanvelzen]

The value for ProductName is invalid. It should be Jellyfin Server (with a space). This can cause the app to fail to connect.

[DuvelCorp]

It has the space. I just lost it when editing the json in my post after copy/pasting. I edited my post.

[nielsvanvelzen]

Ok, everything else in the response looks fine so to find the issue we need app logs. They should give us some more information about what's going on behind the "can't reach server" message.

This page explains how to get these logs: https://wiki.lineageos.org/how-to/logcat

[DuvelCorp]

ok her'es the log file edited.
What I did :
Launched the android app (previously disconnected), and tried to connect to my server, in 4G (so from WAN).
Once the connection issue triggered, I downloaded the logcat.

The logcat begins on app startup and I have mass replaced my domain with mydomain.name and my jellyfin instance (folder) name with "/netflix "

I see some bad SSL stuff in there

11-19 20:22:40.290 4702 4752 I cr_X509Util: Failed to validate the certificate chain, error: java.security.cert.CertPathValidatorException: Trust anchor for certification path not found.
11-19 20:22:40.321 4702 4752 I cr_X509Util: Failed to validate the certificate chain, error: Unacceptable certificate: CN=R3, O=Let's Encrypt, C=US
11-19 20:22:40.335 4702 4775 E chromium: [ERROR:ssl_client_socket_impl.cc(980)] handshake failed; returned -1, SSL error code 1, net_error -202
11-19 20:22:40.345 4702 4702 E WebViewFragment$initialize: Received SSL error: primary error: 3 certificate: Issued to: CN=mydomain.name;
11-19 20:22:40.345 4702 4702 E WebViewFragment$initialize: Issued by: CN=R3,O=Let's Encrypt,C=US;
11-19 20:22:40.345 4702 4702 E WebViewFragment$initialize: on URL: https://mydomain.name/netflix
11-19 20:22:40.419 4702 4740 W System.err: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
11-19 20:22:40.419 4702 4740 W System.err: SLF4J: Defaulting to no-operation (NOP) logger implementation

But I swear that my cert is fine, at least I have no issues with htpps in any browser.

If you want to do a test on my install from your devices, we do this in private

Cheers !

.

[nielsvanvelzen]

Your certificate is, actually, invalid. Let's Encrypt X3 is not trusted anymore. See https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ and the a discussion about this issue in our server repository jellyfin/jellyfin#6650

[DuvelCorp]

Wait... do not burry the problem that fast....
As I told you, my cert is valid !

Chrome

Firefox

So that is a first point.

The second point is : this doesn't explain at all "the trick".
Why connecting to demo.jellyfin.org and then to my own instance, would magically make the app handshake with my certificate working??? This make no sense at all if the issue is on my cert as you say !

You should definitely come here and test. I can send you the URL in private

[DuvelCorp]

Note that I made a test with https://www.ssllabs.com/ssltest

So I admit its not totally fine. Looking on my side to solve this but I still would like some feedback from you about my previous thoughts.

[Maxr1998]

Yes, it seems like you incorrectly configured your server and picked the wrong certificate file.
Let's Encrypt should provide a fullchain.pem, that's what you need on your server.

As this serverfault answer explains, browsers typically have intermediate certificates cached as well, whereas Android only has the root certs available, thus requiring a full certificate chain to verify a certificate's validity. Your "trick" works because the demo server also uses a (correctly configured) Let's Encrypt certificate with the intermediate certificate(s) included, and the WebView will apparently remember those for the rest of the session.

Closing this since it's a configuration issue, but I hope you'll figure it out.

[DuvelCorp]

OK that makes perfect sense now. You can indeed close it because this was the explanation and my issue is solved.
I added the LetsEncrypt intermediate cert to my reverse proxy config, and android app now works perfectly

thanks for the help :)

Steps if someone need to do this:

First download the correct certificate from https://letsencrypt.org/certs
In my case it was the Intermediate certificate "Let’s Encrypt R3"

$ sudo mkdir letsencrypt_intermediate_certs
$ cd letsencrypt_intermediate_certs/
$ sudo wget https://letsencrypt.org/certs/lets-encrypt-r3.pem

Then in your Apache's <VirtualHost *:443> config, add the line
SSLCertificateChainFile /etc/apache2/certs/letsencrypt_intermediate_certs/lets-encrypt-r3.pem

restart apache
$ sudo service apache2 restart

[Maxr1998]

Sure, glad it works now. Also good to know in case someone has issues with their certificate in the future. 👍

Edit: Changed the title a bit to make it easier to find for others having the same issue (and for us to link).