Unable to Connect [webOS 3.0, 3.5, more?]

[anthonylavado]

Issue
During certification testing, devices running LG webOS 3.0 or 3.5 were unable to connect to the demo server.

Notes from Testing

When selecting the Connect UI Button after entering the received URL, the message ""Unknown error occurred. Are you connecting to a Jellyfin sever?"" is displayed.

Next Steps
It is possible that this was an intermittent issue, but because of the failed result, LG stopped testing on webOS 3/3.5 devices at this point.

For internal (Jellyfin) testing:

• Install more webOS emulators and test connectivity with app to internal demo server.
• Check equipment list for team members with webOS 3/3.5 equipment and ask them to test against internal demo server.

For community help:

If you have a webOS 3/3.5 device, please let us know if you are able to test this. You can use the public demo server, https://demo.jellyfin.org/stable.
The username is demo, with no password. If you need a compiled IPK to install, please let me know.
Note that the public server is reset on the hour, every hour. It is recommended to avoid testing at that moment, as it will likely result in a connection error. This does not apply to our internal testing server.

[dab2020]

I have multiple webOS devices and for me The Jellyfin app has been working on the devices that are running webOS 3.0 and 3.5. but I haven't updated the app in a while so I may be a bit behind the current release. I'll try to recompile it today and test it out.

[anthonylavado]

Connecting to the public demo server from the app may not work just yet - there's a change I have to make to the reverse proxy config.

This change was already made for the internal demo server, so that won't be part of the issue affecting the client.

[oddstr13]

The error message originates from here

jellyfin-webos/org.jellyfin.webos/js/index.js

Lines 320 to 336 in 25b7744

	function handleFailure(data) {
	console.log("Failure:", data)
	console.log("Could not connect to server...")
	if (data.error == 'timeout') {
	displayError("The request timed out.")
	} else if (data.error == 'abort') {
	displayError("The request was aborted.")
	} else if (isNaN(data.error)) {
	displayError("Got HTTP error " + data.error.toString() + " from server, are you connecting to a Jellyfin Server?")
	} else {
	displayError("Unknown error occured, are you connecting to a Jellyfin Server?")
	}
	hideConnecting();
	storage.remove('connected_server');
	curr_req = false;
	}

Looks like the isNaN check is inverted, so this would most likely have been caused by a 4xx or 5xx status code returned by the server.

[anthonylavado]

A testing release is available here: #50

[anthonylavado]

In #50, a user mentions that they are also unable to connect to the public demo server when using the 3.0 emulator, but it works fine under 5.0 (which is mainly what I tested with). We have a variety of CSP and other headers in place, so I'll have to review these to see what's stopping it.

[anthonylavado]

In #50, @agrisci has figured out the issue. It looks like we need to enable older protocols for SSL for the internal testing server.

#50 (comment)

[dovecode]

As mentioned in #46, I also cannot connect to the public demo server from my WebOS 6.0.5 TV. Not sure if there's a debug log available for a physical TV I can inspect to see if it is a TLS issue as well,...

[anthonylavado]

So there's been a bunch of work behind the scenes. We've made sure that the internal and public demo servers are using TLS 1.2, as that is supported across all webOS versions.

From some more cursory searching on the internet, it seems that the app interface doesn't like Let's Encrypt certificates for some reason, even though the regular browser is okay with it. Since we use LE for all our sites, this is presenting an issue for us.

We've currently made alternate arrangements to provide LG with an HTTP URL for testing, but obviously we'll still want to figure out how we can make this work for everyone.

[caeci11ius]

From some more cursory searching on the internet, it seems that the app interface doesn't like Let's Encrypt certificates for some reason, even though the regular browser is okay with it. Since we use LE for all our sites, this is presenting an issue for us.

Have run into this specific issue before when using Emby's app with letsencrypt in the past. You're right to note that the tv browser works fine with letsencrypt, but there is a certificate problem that seems specific to apps on webos (this is on an LG C8 running what would've been late v4 or early v5 firmware at the time). I don't imagine it's fixable without cooperation from LG, but would be great if you could draw it to their attention. Maybe they'll do a firmware update!

[mono-of-pg]

I've dealt a lot with SSL/TLS related problems in the past.
This specific one could simply be a TLS handshake mismatch.

Your demo server supports the following cipher suites for TLS 1.2:

TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384

https://webostv.developer.lge.com/discover/specifications/web-engine/ states that WebOS 3.x uses a Chromium 38 engine which just might not be capable of using those ciphers.

I'd suggest adding something like this to the TLS 1.2 configuration on your demo server:

TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

[WilcoVertegaal]

Are we sure that this is related at all to the Jellyfin client? I tried connecting to the demo server through the LG built-in web browser, and it didn't connect. Nothing happens. When I pointed it to http://demo.jellyfin.org/stable it was routed correctly to https://demo.jellyfin.org/stable - so it's not a routing issue of any kind. Seems to line up with @mono-of-pg 's findings.

[anthonylavado]

There is a separate private server that we provide to companies like LG and Roku in order to test clients. We have selectively enabled HTTP without a redirect to HTTPS on that server.

[WilcoVertegaal]

Sorry, that was not the point I was trying to make. I was trying to say that LG televisions seem to have trouble connecting to the demo server via https in general, either through the Jellyfin client or through the built-in web browser. Supporting the observation by others that it is probably a Let's Encrypt and/or TLS version specific issue that can't be solved in the Jellyfin weOS client software, or even in the Jellyfin server software at all.

[anthonylavado]

@WilcoVertegaal Ah yes, I agree. All good then :-)

We'll try @mono-of-pg's findings soon to be sure. If anything, it helps inform our documentation.

[mono-of-pg]

Assuming there's a reverse proxy in front of your demo server this could easily be changed in its configuration.

It even might work with an off-site proxy, so someone with an LG TV could simply set up a local reverse proxy pointing to the demo server without having to change anything on the server at all.

[anthonylavado]

@mono-of-pg There is a reverse proxy, but the item is a bit lower on the list since we're forcing the HTTP option for passing certification at least. I'll pass this along to my fellow Core Team member who set that up.

[mono-of-pg]

I fully understand we're looking for a quick fix here and that there are more important things to spend time on.
That's why I'm suggesting to test this because if it really turns out to be that way then it's no jellyfin issue at all.

@WilcoVertegaal can you do some tests with your LG TV if I provide a reverse proxy?

[WilcoVertegaal]

@mono-of-pg I have a personal Jellyfin server behind a reverse proxy myself, Apache, with a Let's Encrypt certificate. Not sure if I know how to reconfigure it properly though :-) If you could either give me some directions, or a correctly configured reverse proxy, I would be happy to test it.

[mono-of-pg]

Very nice @WilcoVertegaal.
I assume the connection through your proxy fails right now.

1. Test your ssl configuration with SSLLabs or testssl.sh if you want to run that from your computer.
2. Look for "Cipher Suites" and save that for reference.
3. Edit your Apache reverse proxy configuration to resemble this:

SSLProtocol -all +TLSv1.2 +TLSv1.3
SSLCipherSuite EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA:EECDH:EDH+AESGCM:EDH:+3DES:ECDH+AESGCM:ECDH+AES:ECDH:AES:HIGH:MEDIUM:!RC4:!CAMELLIA:!SEED:!aNULL:!MD5:!eNULL:!LOW:!EXP:!DSS:!PSK:!SRP
SSLHonorCipherOrder on

This should give you a rating of "A" on SSLLabs and still provide compatibility for older clients/browsers.

4. Restart Apache and give it a try...

[WilcoVertegaal]

Erm .. As it turns out, connecting to my own server is not failing at all ... SSLLabs cipher suites gives me a whole sh*tload of suites available, then the handshake simulation succeeded for all clients except "Yahoo Slurp Jan 2015".

Then I did the same for https://demo.jellyfin.org - cipher suites are much more limited, handshakes fail for like 40% of the clients.

What I learn from this: it's not Let's Encrypt, but it might very well be a missing SSL cipher suite.

Would you agree @mono-of-pg ?

[mono-of-pg]

Yes, that's exactly what I was talking about...

Now try to point your proxy to the jellyfin demo server and let it terminate SSL for your TV.
LG-TV -> Proxy -> Jellyfin-Demo-Server

If that actually works it might be a good indication that this is where the problem really sits.

[WilcoVertegaal]

@mono-of-pg nailed it 🥇

With this configuration, my LG TV connects to the demo server and plays content without any problem. Connecting directly to https://demo.jellyfin.org/stable/ says something like "website not found".

[mono-of-pg]

@mono-of-pg nailed it 🥇

With this configuration, my LG TV connects to the demo server and plays content without any problem. Connecting directly to https://demo.jellyfin.org/stable/ says something like "website not found".

That's awesome 👍

Now @WilcoVertegaal try this in your config:

SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:!RC4:!CAMELLIA:!SEED:!aNULL:!MD5:!eNULL:!LOW:!EXP:!DSS:!PSK:!SRP:!AESCCM

It should make your proxy look more like the demo server and offer only the ciphers mentioned above.

If that fails we can be pretty sure we found the root cause of this issue.

[WilcoVertegaal]

That's great - I now have A+ score on SSLLabs :-S

Cipher suites as reported by SSLLabs:

TLS_AES_256_GCM_SHA384 (0x1302)                                                                                    ECDH x25519 (eq. 3072 bits RSA)   FS | 256
-- | --
TLS_CHACHA20_POLY1305_SHA256 (0x1303)                                                                                    ECDH x25519 (eq. 3072 bits RSA)   FS | 256
TLS_AES_128_GCM_SHA256 (0x1301)                                                                                    ECDH x25519 (eq. 3072 bits RSA)   FS | 128
# TLS 1.2                                                 (suites in server-preferred order)
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)                                                                                    ECDH x25519 (eq. 3072 bits RSA)   FS | 256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)                                                                                    ECDH x25519 (eq. 3072 bits RSA)   FS | 128
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)                                                                                    ECDH x25519 (eq. 3072 bits RSA)   FS | 256
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x9e)                                                                                                                              DH 2048 bits   FS | 128
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)                                                               ECDH x25519 (eq. 3072 bits RSA)   FS                                                       WEAK | 128
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)                                                               ECDH x25519 (eq. 3072 bits RSA)   FS                                                       WEAK | 256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)                                                               ECDH x25519 (eq. 3072 bits RSA)   FS                                                       WEAK | 128
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)                                                               ECDH x25519 (eq. 3072 bits RSA)   FS                                                       WEAK | 256
TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x33)                                                                                                         DH 2048 bits   FS                                                                   WEAK | 128
TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x39)                                                                                                         DH 2048 bits   FS                                                                   WEAK | 256
TLS_RSA_WITH_AES_128_CBC_SHA (0x2f)                                                                                       WEAK | 128
TLS_RSA_WITH_AES_256_CBC_SHA (0x35)                                                                                       WEAK | 256

All handshake simulations passed.

Also the demo server still connects via LG TV.

[mono-of-pg]

While an A+ rating looks nice it's not what we're looking for in this case :-D

Your mod_ssl version seems to be a little different from mine so we need to fiddle a bit with the settings to just allow the same ciphers as the demo server does.

[WilcoVertegaal]

I'm on Ubuntu 20.04 with Apache 2.4.41, mod_ssl the same version info and OpenSSL 1.1.1f.

[mono-of-pg]

Apache 2.4.37 and OpenSSL 1.1.1g here.
Anyway, we just need to get that cipher string right so we can be sure to compare the right things.
openssl should return only ECDHE-RSA-AES256-GCM-SHA384 and DHE-RSA-AES256-GCM-SHA384 like it does on my machine:

[root@pg-dmzweb01 conf.d]# openssl ciphers -v 'ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384' | grep TLSv1.2
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(256) Mac=AEAD
[root@pg-dmzweb01 conf.d]#

edit: OpenSSL 1.1.1f on Ubuntu 20.04.2 LTS gives me the same result - just checked

[WilcoVertegaal]

OpenSSL 1.1.1f on Ubuntu 20.04.2 LTS gives me the same result - just checked

Correct, mine is the same.

[mono-of-pg]

OpenSSL 1.1.1f on Ubuntu 20.04.2 LTS gives me the same result - just checked

Correct, mine is the same.

So why does your mod_ssl do something different here?

Try with:

SSLProtocol -all +TLSv1.2
SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384

Is this a VirtualHost or your serverwide ssl config?

[mono-of-pg]

Anyway, @anthonylavado as you can read above this issue very likely seems to be connected with the ciphers (not) offered by your reverse proxy.
Thanks to @WilcoVertegaal for testing!

[WilcoVertegaal]

Is this a VirtualHost or your serverwide ssl config?

Yes.

Try this:

Still no dice. Sorry @mono-of-pg - I have zero expertise in this area whatsoever.

Thanks to @WilcoVertegaal for testing!

No problem, very happy to be able to contribute! (Next on the list: a VIDAA/Hisense client ... )

[kupje]

The root certificate "DST Root CA X3", which was used for cross-signing the Let's Encrypt R3 CA, expires on September 30 this year: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
From that point, the ISRG Root X1 is the only available trust path for Let's Encrypt certificates. It wasn't fully supported by major platforms until 2018: https://letsencrypt.org/2018/08/06/trusted-by-all-major-root-programs.html

Is this going to be a potential issue with older versions of webOS? I don't know LG's testing requirements, but maybe they dictate that the app must work on a device which has received zero updates (including the CA bundle) in X years? It would be a shame to see a certification attempt go to waste because they happen to test on some ancient device after that date and see a trust issue...
Unfortunately I don't have a working dev environment for testing right now, but maybe it's possible to set up an emulator on a firewalled network which can only access the public demo server, install the oldest supported version of webOS, set its system date beyond Sep 30 and attempt to connect?

There's also Buypass, a Norwegian CA whose root certificate has been trusted by major browsers since 2010. They fully support the ACME protocol and can be used with clients such as Certbot: https://www.buypass.com/ssl/products/acme
It might be a solution if the new LE trust path turns out to be an issue with legacy devices.

[mono-of-pg]

While this might be an issue after september I think it's pretty obvious that the findings of me and @WilcoVertegaal are the root cause of the problem.
Has this been looked at by a dev in the last two months?

[anthonylavado]

As I mentioned before, while I appreciate all of the testing, this in particular isn't the highest for the release issues because we're just using a workaround for the time being.

Basically, for all of our vendor accounts (including Google Play, Apple App Store, etc) we have a private demo server that we provide for them to test with. It's got mostly similar content to the public demo server (just a few items removed for rating reasons). On this private server, we've given the LG Testing IP range access to connect without HTTPS, solely for testing purposes.

We will eventually want to apply these cipher option fixes here and on the publicly accessible server. We'll also use this info to make sure our Reverse Proxy guides have a notice and are up to date once the app releases, so that public users know what to expect. I'll keep this issue open for tracking purposes, but I would deem it no longer "release critical".

[oddstr13]

The exact error reported is net::ERR_SSL_VERSION_OR_CIPHER_MISMATCH

ares-device-info

modelName : 49UH603V-ZE
sdkVersion : 3.3.0
firmwareVersion : 05.31.00
boardType : K2L_DVB_EU

Console output

Navigating 1...
http://demo.jellyfin.org false
GET https://demo.jellyfin.org/System/Info/Public net::ERR_SSL_VERSION_OR_CIPHER_MISMATCH
Failure: Object {error: true}
Could not connect to server...
Failure: Object {error: 0}
Could not connect to server... 

[mono-of-pg]

Thanks for your update @anthonylavado.
As long as it's passing the certification your workaround will do just fine.
Very few will be facing this issue anyway, so having it in the documentation will probably be good enough.

[kxzaon]

Guys, the 30th of september is here !
The Jellyfin app on my rooted LG B8 (WebOS : 4.3.0-9604) refuse to connect to my "Letsencrypted" jellyfin server. Same behaviour with https://demo.jellyfin.org/stable
Do you have a workaround beside obtain a buypass certificate ???

[oddstr13]

Guys, the 30th of september is here ! The Jellyfin app on my rooted LG B8 (WebOS : 4.3.0-9604) refuse to connect to my "Letsencrypted" jellyfin server. Same behaviour with https://demo.jellyfin.org/stable Do you have a workaround beside obtain a buypass certificate ???

Use http over LAN?

[dmitrylyzo]

Some investigation about #43 (comment)
From our current AJAX implementation:

jellyfin-webos/frontend/js/ajax.js

Line 77 in 0e3c02b

settings.error({error: true});

So, that isNaN bug protects us from getting true as a string.
We probably need to fix/update/replace AJAX first.