[BUG] segfaults since upgrading to 3.8.0 #4787
Labels
Comments
|
Could you please do the same with the asan build? |
|
Unfortunately the ASAN build crashes when starting the daemon. It does not generate a coredump. systemd just tries to restart it until it reaches the startup limits. The ASAN log files created contain the following: |
|
Set |
|
I'll do that later today. Just wanted to state that this is not mentioned in the ASAN section of the FAQ. |
|
Got it running (and segfaulting later) with The generated ASAN reports are kind of hard to correlate to the coredumps, as the PIDs in the ASAN file names don't match the PIDs from It's also quite possible that the ASAN routines don't have time to write a report anymore due to the segfaults terminating the processes? Here's the journal from one such segfault with the ASAN binaries: Here's the corresponding I don't have an ASAN log for that PID, only for others. But like I said, they only contain leak information. Here's an example ASAN log from a different PID: |
|
In the core file inspection, could you please type |
|
Sure. Unfortunately the core dump I used above has already been removed (auto-updates installed the 3.8.0 last night, meaning there were TONS of new segfaults & core dumps over during the night…). Here's a new full backtrace and the requested peek into tokens from a newly observed crash from a couple of minutes ago: Still crashing in the same function; still from the ASAN build. |
|
Unfortunately, I have not a single idea now. It is not reproduced anywhere in our CI/test environments. |
|
Thanks for all your feedback so far. I'll try to build a recreatable test case by tcpdump-ing the mail server traffic. Won't get around to it until next week, though, as I'll be out of town for business for a couple of days. |
|
I think I have one idea now, hold on. |
|
Do you have any per user script for the statistics? |
|
I… don't think so. Don't even know what those are or how to check if I do, to be honest. |
|
We're seeing the same segfaults on a Debian 12 bookworm system since version 3.8.0 (also rolled back to 3.7.5 as a stopgap). If you need more things to troubleshoot with, ask away. |
1 task
|
I'm currently testing again with rspamd 3.8.4, let's see |
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Prerequisites
Describe the bug
We've been running rpsamd 3.7.x on Ubuntu 22.04 successfully for quite a while. We're using packages from rspamd.com/apt-stable/
On Friday we updated to the latest available package,
3.8.0-1~28391190c~jammy. Shortly afterwards rspamd starts to segfault quite a bit. Not for each and every email, but for enough of them that keeping it running is unfeasible; this is a production system for our whole company.We've circumvented the problem by downgrading to
3.7.5-2~8c86c1676~jammyagain. No other changes were made.All other packages on this system are up to date.
The journal contains the following information:
Steps to Reproduce
Versions
See above, and:
Ubuntu 22.04 on amd64. Packages from rspamd.com/apt-stable/
Additional Information
I have not been able to run the ASAN builds for 3.8.0. Those crash right when starting the daemon, unfortunately.
I have installed the non-ASAN debugging symbols & run
coredumpctl debug <PID>withbt fullfor one of the crashes. The output was:The text was updated successfully, but these errors were encountered: