Always install a local TURN server and share port 443 #579
Conversation
This patch changes the current setup in the following way: * install coturn on all machines * install haproxy and let it serve port 443 * configure nginx to use port 127.0.0.1:81 without encryption * configure BBB to use the local coturn using the `turns` protocol haproxy will terminate the TLS connection and use the first byte to decide if the traffic will go to nginx or coturn.
HTTP2 is a binary protocol. So switching on the first byte does not work. If the client wants to speak HTTP2, it will negotiate this by ALPN in the TLS handshake. haproxy can use this to send traffic in the right direction. As of section 3.1 in RFC 7540 using APLN to negotiate on HTTP2 is mandatory for encrypted connections.
|
Nice! Will test and let you know how it works. |
|
And a last comment (for now) to get this to work is the redirect of greenlight: bbb-install/bbb-install-2.6.sh Lines 699 to 703 in 738e806 This needs to be changed/added to Reasoning:
|
thanks @EmmyGraugans for feedback
schrd
force-pushed
the
always-local-turn
branch
from
8e4702d
to
d1e4572
Compare
TURN connections should be offered on two transports to the client: * port 443 TCP (thats what haproxy redirects to coturn) * port 3478 UDP The UDP port likely offers better quality for clients than the TCP connections as TCP connections stall when packets are lost. So UDP connections should be always preferred over TCP connections for WebRTC connections as long as udp is not blocked by firewalls. Firefox needs to relay all its traffic that goes to mediasoup through a turn server as a workaround for their broken ICE implementation. With fullaudio which lets mediasoup proxy audio connections to freeswitch this is becoming even more important for a good out of the box experience.
This is part of the operator config and it belongs to /etc according to FHS.
| mode tcp | ||
| server localhost 127.0.0.1:82 | ||
| END | ||
| chown root:haproxy "$HAPROXY_CFG" |
There was a problem hiding this comment.
In testing we need to add
cat /etc/letsencrypt/live/$HOST/{fullchain,privkey}.pem > /etc/haproxy/certbundle.pem
to create /etc/haproxy/certbundle.pem
haproxy could not use the certificate because it wasn't there yet
|
At some point we need to fix Currently I'm getting output like this: It's just a parsing issue, and we have larger nuts to crack, but I just wanted to note it. |
|
I'm testing using Firefox on a GNS3 test network. This is what I'm seeing in The flurry of SSL handshake errors comes after the create (using API Mate), but before the join, with about a five second pause between the successful API call and the log messages. There's no HTML5 client connected at that point. I'm just learning haproxy, but the way I read these logs files is that none of the connections is being relayed to the turn server, and that is consistent with All I have to do to get it working, though, is to switch I'm packing it in for today. @schrd, in particular, I welcome any suggestions on how to debug this further. |
I have the same error 1007 ICE when try to connect with audio (webcam and sharescreen work perfect) my server in amazon ec2 |
...
I'd probably start using |
|
@schrd, I'm pretty sure it's using I'm looking at this line in Is the end of this advising the client that it should select either 'h2' or 'http/1.1' as its ALPN choices? If so, wouldn't a compliant client never select TURN? I'm asking, since I'm just learning In any event, I can confirm that I'm seeing failures with both Firefox and Chromium, and am continuing to investigate. |
yes it's works now i was missing something thank you it's works now very good and i send the output of the WebRTC i hope you see it to let me if something wrong
|
The DNS names in that image start with "turn". That suggests to me the older setup with a separate TURN server. Any idea what you did to get this working? |
ALPN is a mechanism to negotiate protocols during the TLS handshake before any inner payload data is exchanged. This statement means that haproxy should signal to the client that it supports h2 and http/1.1 protocols. ALPN is not required to establish a TLS connection. The logic in this haproxy configuration works like this:
You said you have these failed requests the create and join API calls. I doubt that browsers will try to establish either a turn connection before the html5 client is loaded. What frontend do you use to create/join sessions? Is there any monitoring tool or load balancer etc polling the API in between? Does the request come from the client IP? I am curious what could be causing this. I'd really like to debug this interactively with you and see what is going on. Drop me a mail if you like. |
|
For reference, you should know that STUN/TURN also has an ALPN label: When I was testing before, I think I ended not requiring any traffic inspection (decryption in haproxy) at all, the ALPN was sufficient. |
any luck to solve this error? I'm installed the script in a fresh server and have the same errors |
|
An update on my end. I'm testing the local TURN server on Amazon EC2. EC2 instances have the public/private IP address. I needed to change the configuration of the TURN server to define and external IP address. I also commented out So I could test use the TURN server stand-alone with another BigBlueButton server and make sure it was working. I tested with FireFox where I can force it to use a TURN server by setting When accessing BigBlueButton + TURN server configuration, sharing of video with FireFox works, but not FreeSWITCH (I'm not using fullaudio). I check the logs and I can see that FreeSWITCH FreeSWITCH receives the call, but at some point it switches to use the internal IP address After that the FreeSWITCH log pauses, and then a few seconds later the BigBlueButton client gives a 1010 error. Interested if anyone else could get further with running a local TURN server on an EC2 instance. |
With proxy_protocol support the original IP from the client will appear in nginx log instead of 127.0.0.1 for all requests
|
Added some fixes suggested by @BrentBaccala and Christoph Martin. These include:
I did not yet test on dual stack setups, these may need more tweaks. |
|
Hello, i'm figured out 4 way to fix problem error 1010 or 1007 in ec2 or aws lightsail i'm tested the 4 way and it's works well 1- For fresh install (before install script just check opened port and just open port 80 and port 22(ssh) ) after install script just open port 443/tcp, 3478/tcp , 16384-65535/udp 2- if you already installed script change the value of server localhost in the haproxy.cfg :
and check turnserver.conf and make sure to change valuse like example below: `listening-port=3478 listening-ip=private-ip external-ip=public-ip/private-ip min-port=32769 fingerprint keep-address-family no-cli allowed-peer-ip=private-ip ` then: restart haproxy and turnserve then it will work 3- check the value for haproxy.cfg: and turnserver config listening-port=3478 listening-ip=public-ip/private-ip external-ip=public-ip/private-ip min-port=32769 fingerprint keep-address-family no-cli no-loopback-peers denied-peer-ip=0.0.0.0-255.255.255.255 then: restart haproxy and turnserve 4- i'm not sure it's right or not but we will make freeswitch and kurento to listen only to private-ip in (local, external) I hope this will help |
…s media.peerconnection.ice.relay_only = true
since `systemctl restart bigbluebutton` doesn't restart nginx
…lso work over TURN
Add iptables rule to bbb-install-2.6.sh
|
Seems to be working fine now. I'm no longer seeing the SSL handshake errors and it seems to work fine either with or without NAT. I haven't tested Greenlight. There was some discussion of trying to do this with a newer |
|
The TURN server doesn't get connected if the IP address is proxied through Cloudflare. Is it possible to proxy the IP address using Cloudflare and still use the TURN server? |
This patch changes the current BBB 2.6 setup in the following way:
turnsprotocolhaproxy will terminate the TLS connection and use the first byte to decide if the traffic will go to nginx or coturn.