Livekit does not use configured TURN server

[TomM-cos]

Hello, we need some guidance on how to activate TURN server and TCP/TLS communication:
I installed Livekit in our AWS EKS Kubernetes cluster and followed the configuration in the helm templates. The application is configured to be Internet-facing including loadbalancer and the K8S resources seem to be configured properly.
When trying to connect via Livekit Meet I can see that the signalling connection on port 7880 is working and the TURN server data are visible in the first websocket response.
BUT: In the following WebRTC handshake Livekit only sends ICE candidates using the internal host IP addresses on port 7880, which can not be reached from the client. The TURN server is not addressed. How can we direct Livekit to use the TURN server including TLS? Any config values missing or wrongly set?

This is the resulting config map extract:
port: 7880
rtc:
port_range_end: 60000
port_range_start: 50000
tcp_port: 7881
use_external_ip: false
turn:
domain: turn.myhost.com
enabled: true
external_tls: false #no effect when setting to true
loadBalancerAnnotations:...
secretName: livekit-turn-tls-secret
tls_port: 5349
The data are also in the log during startup:
Starting TURN server {"turn.relay_range_start": 30000, "turn.relay_range_end": 40000, "turn.portTLS": 5349, "turn.externalTLS": false}
INFO livekit service/server.go:265 starting LiveKit server {"portHttp": 7880, "nodeID": "ND_TnWZiwvBkho7", "nodeIP": "10.202.41.230", "version": "1.9.0", "rtc.portTCP": 7881, "rtc.portICERange": [50000, 60000]})

Thanks in adavance for your support!
Regards,
Thomas

[biglittlebigben]

Could you please provide a full set of server logs when the ICE attempt occurs?

[TomM-cos]

Hello, find below the log extract and the screenshot of the developemnt tools.
Looks like Livekit at first tries ICE via TCP(unfortunately with the not Internet exposed internal IPs), but then does not progress to TURN/TLS as expected, though the turn server seems up and running and the hostname is available in the client. Please also take a look at the /settings request with Http 404 response - this one also does not work locally. May this is causing TURN/TLS not to be tried?
Thanks for your support!

livekit-livekit-server-7868b65c5b-nk5rv-livekit-server.log

[boks1971]

/settings is not a supported end point in OSS.

Have you set up the TURN load balancer to redirect 443 to the internal TURN/TLS port which is 5349 in the config you shared? 443 is always given to the clients. The load balancer should listen on 443 and connect to 5349 on the backend.

[TomM-cos]

Hello,
yes, on the load balancer side we configured the Host and 443 port mapping to 5349.
The problem seems to be, that the TURN endpoint https:/turn... is not requested / tried on client side (see DevTools screenshot). After the 404 errors the client stops all activities and the session is disconnected. I expected, that after the ICE/TCP tries the TURN/TLS is used.
Regards, Thomas

[boks1971]

That is a browser behaviour and not something managed/controlled by LiveKit client SDK. Browser is supposed to try those candidates. Which browser (version, os) are you using? Are you seeing the issue across different browsers?

[TomM-cos]

Hello,
OK, that may be an interesting path to follow. I am using Firefox 140 on Win11 in our enterprise environment. In Edge and Chrome the connections fail in an earlier phase - but this seems to be another connectivity issue which I try to fix in parallel. Is there any firefox setting necessary to pick the TURN/TLS candidate? Thanks!
Thomas

[boks1971]

It should be automatic. @lukasIO are you aware of any Firefox 140 issues.

I have seen Firefox issues when there is no TURN server configured. It fails to connect with an error like "no TURN server configured, please check about:webrtc" in the console, but not when a TURN server is there.

[TomM-cos]

Hello, in the attachment you find the Firefox WebRTC protocol with this morning's failed connection . In the config anythin looks right, including credentials being present. Can I test the turn server access somehow without the complete protocol (such as livekit endpoint via Https)? Thanks!
Regards, Thomas

Firefox_webrtc_report.txt

[boks1971]

That looks fine to me.

And the server from your logs is reachable via this test too - https://www.metered.ca/turn-server-testing.

I think it would be better to get Chrome issue solved first and have it connect as that is the most widely used, hence most tested browser. Is there something we can help with to get Chrome working? And then we can come back to the Firefox issue.

[boks1971]

oh, I was running that test from Chrome.

Just tried Firefox 143 (the above testing page) with your TURN server and it says not reachable.

[TomM-cos]

Hello,
this may be a problem with load balancer or TLS settings? Maybe the browser does not show a connection since TLS handshake does not work? Can I test easily, whether I can reach the turn server?

I used the Helm charts provided in the livekit-helm repository. My installation is based on an existing AWS Application Load Balancer controller, which also routes other traffic into the cluster.
Livekit server got connected to an ALB, but for the Turn-Server an AWS NLB network load balancer was set up. Is this correct for an AWS EKS environment? Is there any special load balancer annotation I have to provide or other special ingress configuration? Right now I can see the DNS points to this load balancer, that port 443 is listened to and forwarded to port 5349 of the container.

Concerning TLS: Unlike ALB I can not attach a TLS certificate directly to NLB, so that the TLS termination needs to be done in TURN itself using the provided TLS secrets. I checked, that the correct TLS Cert and private Key are provided in the mount path of the container, that the mounting succeeded and the values properly filled - is there any way to see whether these are properly accepted by TURN server?
Hopefully this gives a good hint of what may be wrong. Thanks again for your great support!
Thomas

[TomM-cos]

Concerning reachability of our test infrastructures: out of IT security reasons we restricted access and also DNS entries to our intranet, so you will not be able to access.. Is there any local test possible using my browser?

[boks1971]

I do not know fully know the ins and outs of AWS EKS and the gotchas there. Seems like you have everything set up.

Some things to try

• maybe run tcpdump or some other packet tracing tool on the server to see if packets to 5349 are coming in at all
• deploy something like this (https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/) internally and test

[boks1971]

And BTW, the test using metered.ca TURN test fails in Chrome also. I have no idea what I was seeing before. I was also trying a different TURN server. Maybe, I tested the wrong thing. Makes sense that it fails given that you mentioned it is locked down.

[boks1971]

And we also have this connection tester if you want to give it a connection URL and token to try - https://livekit.io/connection-test