Crash: new-delete-type-mismatch

[ibc]

Bug Report

Crash trace with libasan enabled:

2023-07-27T13:25:04.436Z mediasoup:ERROR:Worker (stderr) ==17==ERROR: AddressSanitizer: new-delete-type-mismatch on 0x61100000cfc0 in thread T0:
2023-07-27T13:25:04.436Z mediasoup:ERROR:Worker (stderr)   object passed to delete has wrong type:
2023-07-27T13:25:04.436Z mediasoup:ERROR:Worker (stderr)   size of the allocated type:   248 bytes;
2023-07-27T13:25:04.436Z mediasoup:ERROR:Worker (stderr)   size of the deallocated type: 96 bytes.
2023-07-27T13:25:04.534Z mediasoup:ERROR:Worker (stderr)     #0 0x7f4ba2393467 in operator delete(void*, unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:172
2023-07-27T13:25:04.534Z mediasoup:ERROR:Worker (stderr)     #1 0x55e39ee5daa4 in uv_run (/usr/src/app/my-app/node_modules/mediasoup/worker/out/Release/mediasoup-worker+0xf5baa4)
2023-07-27T13:25:04.534Z mediasoup:ERROR:Worker (stderr)     #2 0x55e39e1f0f6b in DepLibUV::RunLoop() (/usr/src/app/my-app/node_modules/mediasoup/worker/out/Release/mediasoup-worker+0x2eef6b)
2023-07-27T13:25:04.534Z mediasoup:ERROR:Worker (stderr)     #3 0x55e39e219d1d in Worker::Worker(Channel::ChannelSocket*, PayloadChannel::PayloadChannelSocket*) (/usr/src/app/my-app/node_modules/mediasoup/worker/out/Release/mediasoup-worker+0x317d1d)
2023-07-27T13:25:04.534Z mediasoup:ERROR:Worker (stderr)     #4 0x55e39e1ebd37 in mediasoup_worker_run (/usr/src/app/my-app/node_modules/mediasoup/worker/out/Release/mediasoup-worker+0x2e9d37)
2023-07-27T13:25:04.534Z mediasoup:ERROR:Worker (stderr)     #5 0x55e39e1e20f4 in main (/usr/src/app/my-app/node_modules/mediasoup/worker/out/Release/mediasoup-worker+0x2e00f4)
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr)     #6 0x7f4ba1de1d09 in __libc_start_main ../csu/libc-start.c:308
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr)     #7 0x55e39e1e8d99 in _start (/usr/src/app/my-app/node_modules/mediasoup/worker/out/Release/mediasoup-worker+0x2e6d99)
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr) 0x61100000cfc0 is located 0 bytes inside of 248-byte region [0x61100000cfc0,0x61100000d0b8)
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr) allocated by thread T0 here:
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr)     #0 0x7f4ba2392647 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:99
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr)     #1 0x55e39e273474 in TcpConnectionHandler::TcpConnectionHandler(unsigned long) (/usr/src/app/my-app/node_modules/mediasoup/worker/out/Release/mediasoup-worker+0x371474)
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr)     #2 0x55e39e6b39f4 in RTC::TcpConnection::TcpConnection(RTC::TcpConnection::Listener*, unsigned long) (/usr/src/app/my-app/node_modules/mediasoup/worker/out/Release/mediasoup-worker+0x7b19f4)
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr)     #3 0x55e39e6b3d24 in RTC::TcpServer::UserOnTcpConnectionAlloc() (/usr/src/app/my-app/node_modules/mediasoup/worker/out/Release/mediasoup-worker+0x7b1d24)
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr)     #4 0x55e39ee710dc in uv__server_io (/usr/src/app/my-app/node_modules/mediasoup/worker/out/Release/mediasoup-worker+0xf6f0dc)
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr)     #5 0x55e39ee8596f in uv__io_poll (/usr/src/app/my-app/node_modules/mediasoup/worker/out/Release/mediasoup-worker+0xf8396f)
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr)     #6 0x55e39ee5d6ad in uv_run (/usr/src/app/my-app/node_modules/mediasoup/worker/out/Release/mediasoup-worker+0xf5b6ad)
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr)     #7 0x55e39e1f0f6b in DepLibUV::RunLoop() (/usr/src/app/my-app/node_modules/mediasoup/worker/out/Release/mediasoup-worker+0x2eef6b)
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr)     #8 0x55e39e219d1d in Worker::Worker(Channel::ChannelSocket*, PayloadChannel::PayloadChannelSocket*) (/usr/src/app/my-app/node_modules/mediasoup/worker/out/Release/mediasoup-worker+0x317d1d)
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr)     #9 0x55e39e1ebd37 in mediasoup_worker_run (/usr/src/app/my-app/node_modules/mediasoup/worker/out/Release/mediasoup-worker+0x2e9d37)
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr)     #10 0x55e39e1e20f4 in main (/usr/src/app/my-app/node_modules/mediasoup/worker/out/Release/mediasoup-worker+0x2e00f4)
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr)     #11 0x7f4ba1de1d09 in __libc_start_main ../csu/libc-start.c:308
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr) SUMMARY: AddressSanitizer: new-delete-type-mismatch ../../../../src/libsanitizer/asan/asan_new_delete.cpp:172 in operator delete(void*, unsigned long)
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr) ==17==HINT: if you don't care about these errors you may set ASAN_OPTIONS=new_delete_type_mismatch=0
2023-07-27T13:25:04.535Z mediasoup:ERROR:Worker (stderr) ==17==ABORTING
2023-07-27T13:25:04.539Z mediasoup:ERROR:Channel Producer Channel error: Error: read ECONNRESET
2023-07-27T13:25:04.539Z mediasoup:ERROR:Worker worker process died unexpectedly [pid:17, code:1, signal:null]

Your environment

• Operating system: Linux
• mediasoup version: 3.12.8

Info

It "could" be related to this change introduced in version 3.12.6: #1114 (not sure).

[ibc]

@satoren just in case: you think it could be related to #1114?

[ibc]

BTW the above ASAN trace could be "normal". We don't really know if it has any relationship with the crash we are observing. The core dump of the crash is always useless since it's about memory corruption so trace is useless. But still I suspect that the crash culprit maybe that mediasoup change indicated in the description.

[nazar-pc]

If it is heap that is corrupted, dump may not be that useless, right?

[ibc]

If it is heap that is corrupted, dump may not be that useless, right?

With libasan enabled maybe not, without it they are terribly useless. And with libasan enabled there is no crash since it overrides malloc system ¯_(ツ)_/¯

[penguinol]

I'm not using DataChannel. But it seems sctp has it's own thread, and the OnSctpAssociationSendData is calling from the sctp thread, am i right?
Consider such a situation:
SCTP Thread: OnSctpAssociationSendData
SCTP Thread: if (this->destroying)（destroying = false）
Main Thread: delete Transport
Main Thread: this->destroying = true
SCTP Thread: if (this->sctpAssociation)....

So #1114 does not work.

[ibc]

But it seems sctp has it's own thread

Nope. We use usrsctp lib in single thread mode. That cannot be the problem. And we don't even know if the problem is related to this Destroying() thing/change. We have no idea yet plus we are using (in our app testing server) and old version of jemalloc which replaced malloc, so absolutely no idea yet.

[satoren]

I guess that this is simply because the type that performs new and the type that delete are completely different.

mediasoup/worker/src/handles/TcpConnectionHandler.cpp

Line 45 in dc4f8f3

delete handle;

mediasoup/worker/src/handles/TcpConnectionHandler.cpp

Line 65 in dc4f8f3

this->uvHandle = new uv_tcp_t;

Since it is a C language type, there is no real harm. To turn it off use malloc/free or cast to the uv_tcp_t* and delete.

https://learn.microsoft.com/en-us/cpp/sanitizers/error-new-delete-type-mismatch?view=msvc-170

[ibc]

Thanks @satoren, I'll write a PR doing those casts.

[ibc]

@satoren PR done here: #1129