Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Worker crash when closing WebRtcServer if there are TCP connections active #1388

Closed
ibc opened this issue May 2, 2024 · 2 comments · Fixed by #1390
Closed

Worker crash when closing WebRtcServer if there are TCP connections active #1388

ibc opened this issue May 2, 2024 · 2 comments · Fixed by #1390
Labels
Milestone

Comments

@ibc
Copy link
Member

ibc commented May 2, 2024

Steps to reproduce in the mediasoup-demo

  • Connect a browser with forceTcp=true&consume=false to ensure a single WebRtcTransport is created with a single RTC::TcpConnection.
  • Browser is connected at ICE and DTLS levels.
  • In server interactive console enter terminal mode by typing "t" + ENTER.
  • Then type webRtcServer.close().

Here the logs. Notice that I've added dump logs in constructor and destructors and close() methods of TcpConnectionHandle and RTC::Transport and a few others:

During the initial connection browser creates 2 TCP connections then it closes one of them (expected):

cmd> TcpServerHandle::TcpServerHandle() | ++++++++++ TcpServerHandle constructor
RTC::TcpServer::TcpServer() | ++++++++++ RTC::TcpServer constructor
  mediasoup-demo-server:INFO creating Express app... +377ms
  mediasoup-demo-server:INFO running an HTTPS server... +10ms
  mediasoup-demo-server:INFO running protoo WebSocketServer... +70ms
  mediasoup-demo-server:INFO protoo connection request [roomId:ky6y7cwd, peerId:qbw9p7nb, address:127.0.0.1, origin:https://local.aliax.net:3000] +715ms
  mediasoup-demo-server:INFO creating a new Room [roomId:ky6y7cwd] +0ms
  mediasoup-demo-server:INFO:Room create() [roomId:ky6y7cwd] +0ms
TcpConnectionHandle::TcpConnectionHandle() | ++++++++++ TcpConnectionHandle constructor
RTC::TcpConnection::TcpConnection() | ++++++++++ RTC::TcpConnection constructor
TcpConnectionHandle::TcpConnectionHandle() | ++++++++++ TcpConnectionHandle constructor
RTC::TcpConnection::TcpConnection() | ++++++++++ RTC::TcpConnection constructor
TcpConnectionHandle::Close() | ----------- TcpConnectionHandle Close()
RTC::TcpConnection::~TcpConnection() | ---------- RTC::TcpConnection DESTRUCTOR
TcpConnectionHandle::~TcpConnectionHandle() | ----------- TcpConnectionHandle DESTRUCTOR

Then we call webRtcServer.close():

terminal> webRtcServer.close()
undefined
terminal> RTC::TcpServer::~TcpServer() | ----------- RTC::TcpServer DESTRUCTOR
TcpServerHandle::~TcpServerHandle() | --------- TcpServerHandle DESTRUCTOR
TcpServerHandle::Close() | --------- TcpServerHandle Close()
RTC::TcpConnection::~TcpConnection() | ---------- RTC::TcpConnection DESTRUCTOR
TcpConnectionHandle::~TcpConnectionHandle() | ----------- TcpConnectionHandle DESTRUCTOR
TcpConnectionHandle::Close() | ----------- TcpConnectionHandle Close()
  mediasoup:ERROR:Worker worker process died unexpectedly [pid:8247, code:null, signal:SIGSEGV] +0ms
  mediasoup-demo-server:ERROR mediasoup Worker died, exiting  in 2 seconds... [pid:8247] +0ms

Core dump:

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000078
Exception Codes:       0x0000000000000001, 0x0000000000000078

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [8247]

VM Region Info: 0x78 is not in any region.  Bytes before following region: 4334239624
      REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      UNUSED SPACE AT START
--->  
      __TEXT                      102574000-102a13000    [ 4732K] r-x/r-x SM=COW  /Volumes/VOLUME/*/mediasoup-worker

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   mediasoup-worker              	       0x1028e6f39 uv_try_write + 9
1   mediasoup-worker              	       0x102589de5 TcpConnectionHandle::Write(unsigned char const*, unsigned long, unsigned char const*, unsigned long, std::__1::function<void (bool)> const*) + 213
2   mediasoup-worker              	       0x10260210a RTC::TcpConnection::Send(unsigned char const*, unsigned long, std::__1::function<void (bool)> const*) + 42
3   mediasoup-worker              	       0x1026238f6 RTC::WebRtcTransport::OnDtlsTransportSendData(RTC::DtlsTransport const*, unsigned char const*, unsigned long) + 182
4   mediasoup-worker              	       0x1025a774c RTC::DtlsTransport::SendDtlsData(unsigned char const*, unsigned long) + 28
5   mediasoup-worker              	       0x1025a64c9 onSslBioOut(bio_st*, int, char const*, unsigned long, int, long, int, unsigned long*) + 57
6   mediasoup-worker              	       0x10267ca15 bio_write_intern + 229
7   mediasoup-worker              	       0x10267c917 BIO_write + 23
8   mediasoup-worker              	       0x1028ae7a9 ssl3_write_pending + 265
9   mediasoup-worker              	       0x1028ad51e do_dtls1_write + 1198
10  mediasoup-worker              	       0x102882386 dtls1_dispatch_alert + 70
11  mediasoup-worker              	       0x10288652d ssl3_shutdown + 77
12  mediasoup-worker              	       0x1025a6558 RTC::DtlsTransport::~DtlsTransport() + 40
13  mediasoup-worker              	       0x1025a65ee RTC::DtlsTransport::~DtlsTransport() + 14
14  mediasoup-worker              	       0x10261f76e RTC::WebRtcTransport::~WebRtcTransport() + 238
15  mediasoup-worker              	       0x10261fdbe RTC::WebRtcTransport::~WebRtcTransport() + 14
16  mediasoup-worker              	       0x102617236 RTC::WebRtcServer::~WebRtcServer() + 470
17  mediasoup-worker              	       0x1026173be RTC::WebRtcServer::~WebRtcServer() + 14
18  mediasoup-worker              	       0x102581dc1 Worker::HandleRequest(Channel::ChannelRequest*) + 1537
19  mediasoup-worker              	       0x10259045e Channel::ChannelSocket::OnConsumerSocketMessage(Channel::ConsumerSocket*, char*, unsigned long) + 366
20  mediasoup-worker              	       0x102590729 Channel::ConsumerSocket::UserOnUnixStreamRead() + 105
21  mediasoup-worker              	       0x1028e5773 uv__stream_io + 915
22  mediasoup-worker              	       0x1028eb190 uv__io_poll + 1200
23  mediasoup-worker              	       0x1028db2c1 uv_run + 369
24  mediasoup-worker              	       0x10257694c DepLibUV::RunLoop() + 28
25  mediasoup-worker              	       0x10257feb9 Worker::Worker(Channel::ChannelSocket*) + 393
26  mediasoup-worker              	       0x1025751d1 mediasoup_worker_run + 465
27  mediasoup-worker              	       0x10265e3d1 main + 289
28  dyld                          	    0x7ff819e57366 start + 1942

Your environment

  • Operating system: macOS Intel
  • gcc/clang version: Apple clang version 15.0.0 (clang-1500.3.9.4)
  • mediasoup version: 3.14.4
@ibc ibc added the bug label May 2, 2024
@ibc ibc added this to the v3 updates milestone May 2, 2024
@ibc
Copy link
Member Author

ibc commented May 2, 2024

It looks like we are trying to send DTLS data (probably a DTLS Close Alert) on a WebRtcTransport that was already freed.

@ibc
Copy link
Member Author

ibc commented May 3, 2024

Issue also happens with UDP. So probably it's what said in previous comment.

ibc added a commit that referenced this issue May 3, 2024
Fixes #1388

### Details

- `WebRtcServer` destructor must first close its `WebRtcTransports` and then the UDP sockets and TCP servers (so also its TCP connections). Otherwise those `WebRtcTransports` will try to send DTLS Close Alert on an already closed/freed `UdpSocketHandle` or `TcpConnectionHandle`.

### Steps to verify

1. Have `WebRtcServer` running and a client connected.
2. In server side call `webRtcServer.close()`.
3. It should not crash anymore.
jmillan pushed a commit that referenced this issue May 3, 2024
…ts (#1390)

* Fix Worker crash when closing WebRtcServer with active WebRtcTransports

Fixes #1388

### Details

- `WebRtcServer` destructor must first close its `WebRtcTransports` and then the UDP sockets and TCP servers (so also its TCP connections). Otherwise those `WebRtcTransports` will try to send DTLS Close Alert on an already closed/freed `UdpSocketHandle` or `TcpConnectionHandle`.

### Steps to verify

1. Have `WebRtcServer` running and a client connected.
2. In server side call `webRtcServer.close()`.
3. It should not crash anymore.

* Update CHANGELOG
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

Successfully merging a pull request may close this issue.

1 participant