3.4.1 imapd segfaults

[Stephan76]

Hi,
I recently upgraded to 3.4.1 and ran into the issue of imapd segfaulting under very specific circumstances:
Client: horde - it doesn't happen with thunderbird
search_engine:squat - it doesn't happen, if search_engine is undefined
Distros: debian 9 and debian 10 on x64
Compiled with
./configure --prefix=/usr/local/cyrus-imapd-3.4.1 --with-cyrus-prefix=/usr/local/cyrus-imapd-3.4.1 --with-syslogfacility=MAIL --enable-replication --with-perl=/usr/bin/perl --with-openssl --enable-idled --mandir=/usr/share/man --docdir=/usr/share/doc --with-zlib --with-bdb=no

The core dump looks like this:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  convert_cat (rock=rock@entry=0x5627e47c8c20, s=s@entry=0x1 <error: Cannot access memory at address 0x1>) at lib/charset.c:349
349         while (*s) {
(gdb) bt
#0  convert_cat (rock=rock@entry=0x5627e47c8c20, s=s@entry=0x1 <error: Cannot access memory at address 0x1>) at lib/charset.c:349
#1  0x00007f0e11d8b508 in charset_convert (s=s@entry=0x1 <error: Cannot access memory at address 0x1>, charset=<optimized out>, flags=5) at lib/charset.c:2086
#2  0x00007f0e122faaa3 in match (bx=0x5627e47c1d00, part=<optimized out>, str=0x1 <error: Cannot access memory at address 0x1>) at imap/search_squat.c:320
#3  0x00007f0e122f0c06 in subquery_run_indexed (key=<optimized out>, data=<optimized out>, rock=0x5627e47c0fd0) at imap/search_query.c:717
#4  0x00007f0e11aa79d1 in hash_enumerate (table=table@entry=0x5627e47c1018, func=func@entry=0x7f0e122f0a70 <subquery_run_indexed>, rock=rock@entry=0x5627e47c0fd0) at lib/hash.c:311
#5  0x00007f0e122f0e93 in search_query_run (query=query@entry=0x5627e47c0fd0) at imap/search_query.c:996
#6  0x00007f0e122b2844 in index_search (state=0x5627e477f6b0, searchargs=searchargs@entry=0x5627e477f540, usinguid=usinguid@entry=0) at imap/index.c:1931
#7  0x00005627e276dc60 in cmd_search (tag=0x5627e477b2c0 "6", usinguid=0) at imap/imapd.c:6080
#8  0x00005627e2785380 in cmdloop () at imap/imapd.c:1969
#9  0x00005627e278b8b8 in service_main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at imap/imapd.c:999
#10 0x00005627e2779c7d in main (argc=<optimized out>, argv=<optimized out>, envp=0x7fffd14456a8) at master/service.c:647 

This seems to go away when rerunning squatter

[elliefm]

(Just a note that I have edited the OP to format the backtrace as a code block, so that all the <...> sections are correctly escaped and don't get swallowed by the browser.)

[elliefm]

Does this happen for a specific user or set of users, or is there no apparent pattern? Are you able to reproduce it on demand?

I can see from the backtrace that the user has sent some kind of search command (thus we end up in cmd_search at frame 7).

If the crash reproduces reliably for a particular user, it would be helpful to enable telemetry on that user to see what search query is being handled when the crash occurs. There's instructions for enabling telemetry here: http://www.cyrusimap.org/imap/reference/faqs/o-telemetry.html

Please be mindful that the telemetry may contain private data, and truncate it as much as possible. I don't need the whole telemetry log, only a line or two before the crash!

It's strange that it's passing around 0x1 as a string pointer. Though maybe that argument is doing double duty as a flag, I don't know this API.

[Stephan76]

It's pretty reproducible, most of the times just logging in will trigger it. Since I am not using this version on a live system I can't tell how many users this affects, but my two test users are affected.

Ah yes, the telemetry log:

<1626841886<4 EXAMINE INBOX (QRESYNC (1451399466 1402))
>1626841886>* 1141 EXISTS
* 75 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS ()] Ok
* OK [UNSEEN 2] Ok
* OK [UIDVALIDITY 1451399466] Ok
* OK [UIDNEXT 1236] Ok
* OK [HIGHESTMODSEQ 1402] Ok
* OK [URLMECH INTERNAL] Ok
* OK [ANNOTATIONS 65536] Ok
4 OK [READ-ONLY] Completed
<1626841886<5 SEARCH RETURN (MIN COUNT) UNSEEN

And that's it.

[elliefm]

Great, thanks, I'll see if I can reproduce it locally, and go from there :)

[elliefm]

My work in progress is here, but I haven't managed to reproduce the problem yet: cyrusimap/cassandane@master...elliefm:v34/3562-esearch-crash

The messages the test is creating are being uploaded by the same session, and I wonder if that's why it's not reproducing the crash. The next thing I think I want to try is delivering a message in over SMTP/LMTP rather than uploading it from within the same IMAP session, but I haven't yet done so, so I'm writing this intention down here before I forget it.

[Devstellar]

Seems related to my reported issue #3588. Can you merge?

Workaround: disable specific optimization in compile time: -fno-toplevel-reorder

I suppose problem is related to accesing memory out-of-bounds for any reason.

[tacerus]

Edit: Moved my report to a separate issue / upon inspecting my backtrace, I found it to be a different problem.