New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OpenSSL error 140E0197 with Qt >= 5.12.2 #3679
Comments
|
Confirmed this exact issue happens to me and another user using the rc1 client. |
|
Same thing on FreeBSD 11.3-RELEASE with Clients connect and get disconnected instantly with this log entry: Same thing with |
|
I failed to reproduce the issue on FreeBSD 12.0 (Qt 5.12.2) and Fedora 30 (Qt 5.12.4). |
|
Just confirmed it's happening with Qt 5.13.0 and murmur 1.3.0-rc2 as well. Steps to reproduce:
|
|
After some testing it seems like the database used is somehow related. I got no problems after switching the same config from the QMYSQL adapter to sqlite. Same settings otherwise. Just to confirm, @metsuke0 & @reelsense, what database backend are you using? EDIT: I'm gonna try and get a SQL log from mariadb for this later today. |
|
I'll try to look into this again some time in the next week. I have a feeling it has to do with my large(900MB) sqlite database and murmur 1.3. I moved my murmur server to Ubuntu 18.04 and it runs 1.2.19 PPA. |
|
I didn't have time to figure out what dependency was having a problem. So I moved all my mumble servers to Ubuntu 18.04. And since I upgraded them to the new murmur 1.3 release I haven't had a problem. |
|
This is a critical issue, we received a private report this week explaining exactly how to trigger it. We will probably disclose it through a CVE as soon as it's fixed and 1.3.1 is released. |
|
We've also noticed this trying to build with a new OpenSSL (using the Dockerfile, with the unreleased Ubuntu focal instead of disco). Interestingly, only some users randomly get kicked out every few minutes. Example log: |
|
@davidebeatrici Seeing this issue frequently as well, also with an arch server. Is there any mitigation for this aside from an update, or if not, an ETA on 1.3.1? I haven't seen it on plumble (third-party android app), but have had pymumble bots with the same issue. It also seems to occur with more users more frequently. The server sees clients as ghosts and disconnects them, but client reports the remote host closed the connection. Also mentions being unable to send UDP packets and switching to TCP mode, and as @AnonFuncsAreAwesome mentioned, disconnects usually every five minutes or so (though has disconnected twice in a row within ~15 seconds before). Update: tested on an alpine server and had the same result. |
- don't use the github images for now, because we don't want to authenticate - use phlak/mumble instead of our own build, until mumble-voip/mumble#3679 is resolved
|
@bin as mentioned above, downgrading Qt on the server should help. Alternatively, you can use a static build, such as the one used in this docker image: https://github.com/PHLAK/docker-mumble |
|
@cfstras Do this for client, server, or both? |
|
@bin on the server. I wouldn't expect any user of my mumble server to use specific versions of anything... |
|
Thanks, so static linking against QT 15.2.1 should be a safe work-around? |
|
@bin you will have to try that for yourself ;) |
|
I stumbled upon an astonishingly easy way to trigger this reliably as a denial of service. @davidebeatrici, do you need help tracking down exactly where this issue is? Reason I ask is because I don't know if you are working on this privately and I don't want to waste time debugging this if you already know how to fix it. |
|
Thank you for asking! We're currently finishing up the new build environment and system, so that all developers can easily build Mumble. Help is of course appreciated to find the root cause of this issue. The following function is called right after the warning by OpenSSL: Lines 1424 to 1475 in b54cf63
Returning early from the function prevents the client from being disconnected. It's definitely an issue related to Qt, but I'm not sure whether it's due to a bug or wrong API usage and we're only now seeing the effects due to internal Qt changes. |
|
Found it. In qt/qtbase@93a803a Qt incorrectly calls SSL_shutdown() in OpenSSL mid-handshake. I compiled qt5-base on Arch Linux with that commit reverted and Murmur no longer disconnects clients or shows the error message. |
|
@TredwellGit What condition has caused this to trigger for you on the murmur server? |
|
I don't feel that comfortable sharing how to trigger it because it is a very effective denial of service attack and since @davidebeatrici already stated in #3679 (comment) that he wants to wait for it to be fixed. |
@TredwellGit Good job! I actually had found that commit too, but didn't investigate further because I mistakenly thought From https://www.openssl.org/docs/manmaster/man3/SSL_shutdown.html:
@bin Please send me an email, I will reply with details. |
Does this mean this is ultimately a Qt bug? If so has it been reported already? If it has not, we should probably report it in order to get it fixed... |
|
More interestingly is why only mumble seems to be affected. Is mumble doing something unique with Qt's SSL implementation? |
Even then, I wouldn't expect other SSL sessions to be affected.
OpenSSL's behavior seems to be fine: when returning early from
Line 1288 in b54cf63
Line 39 in b54cf63
Lines 140 to 142 in b54cf63
It seems like Qt is emitting |
It appears to be a Qt bug. Calling SSL_shutdown() mid-handshake is incorrect and not handling the resulting error is really incorrect. See how NGINX handles this: https://github.com/nginx/nginx/blob/65ae8b315211988a821bdc32050768f41571ddae/src/event/ngx_event_openssl.c#L2732-L2824
What other server programs are you aware of that use Qt's TLS implementation?
This is what is confusing me. |
|
Workaround in #4032. @TredwellGit Could you open an issue in Qt Bug Tracker, please? I can do it too, but you deserve the credits since you're the one who pinpointed the exact line causing the bug. |
|
Do you know why Qt emits error() from unrelated QSslSocket(s)? Is this an additional bug? I will open an issue report, but I am really more concerned about this getting fixed than I am getting credit. |
|
I believe it's an additional bug that was already present and didn't cause any issues until qt/qtbase@93a803a. |
|
@TredwellGit Did you open an issue in Qt's tracker yet? |
|
Excellent, the description clearly explains the issue in detail. |
|
Fixed: qt/qtbase@36a8bdb |
|
Any indication in which version this will make it? |
|
No, but the fix has been merged in the |
|
So I guess we can surround our workaround by an |
|
Yes, we can. However, I would also like to disable the workaround for the next 5.12 and 5.14 versions. |
I guess as long as we're checking for the minor version number as well, this should be fine :) |
|
Qt 5.12.9: qt/qtbase@36a8bdb |
|
Glad that this was resolved |
|
We requested a CVE ID for Mumble, but MITRE created it for Qt because it's effectively an upstream issue: CVE-2020-13962 |
After upgrading to Qt 5.12.2 all clients got a disconnect after about 5 minutes of connection time. Reproducible with Qt 5.12.2 and Qt 5.12.3, downgrading to 5.12.1 solves this issue.
Not sure if this is a Qt bug or not, so I'm reporting it here first, however it looks to me like a bug in Qt.
murmur.log
Mumble/Murmur 1.3.0_rc1
Archlinux
Edit: Only the Qt version of murmur is relevant, client runs latest Qt just fine with murmur on Qt 5.12.1.
The text was updated successfully, but these errors were encountered: