Redis crashes under load IndexCriteriaTester segmentation error

[danialfarid]

We have redisearch module running on single redis instance and the app uses mix of redis and rediseach commands, once a while when the load is high redis crashes. It seems to crashes quiet randomly and it works fine if the load is low only when there are multiple users are using the app this happens. Any idea what this could be related too?

This is the output from the gdb redis debugger:

Continuing.
Detaching after fork from child process 16809.
Detaching after fork from child process 16810.

Program received signal SIGSEGV, Segmentation fault.
0x00007f6a99d19848 in NI_GetCriteriaTester (ctx=0x7f6aa35cc740) at /u/python/external/RediSearch/src/index.c:964
964	  IndexCriteriaTester *ct = nc->base.GetCriteriaTester(nc->base.ctx);
#0  0x00007f6a99d19848 in NI_GetCriteriaTester (ctx=0x7f6aa35cc740) at /u/python/external/RediSearch/src/index.c:964
#1  0x00007f6a99d1984b in NI_GetCriteriaTester (ctx=<optimized out>) at /u/python/external/RediSearch/src/index.c:964
#2  0x00007f6a99d1984b in NI_GetCriteriaTester (ctx=<optimized out>) at /u/python/external/RediSearch/src/index.c:964
#3  0x00007f6a99d1984b in NI_GetCriteriaTester (ctx=<optimized out>) at /u/python/external/RediSearch/src/index.c:964
#4  0x00007f6a99d1984b in NI_GetCriteriaTester (ctx=<optimized out>) at /u/python/external/RediSearch/src/index.c:964
#5  0x00007f6a99d1984b in NI_GetCriteriaTester (ctx=<optimized out>) at /u/python/external/RediSearch/src/index.c:964
#6  0x00007f6a99d1984b in NI_GetCriteriaTester (ctx=<optimized out>) at /u/python/external/RediSearch/src/index.c:964
#7  0x00007f6a99d1984b in NI_GetCriteriaTester (ctx=<optimized out>) at /u/python/external/RediSearch/src/index.c:964
#8  0x00007f6a99d1984b in NI_GetCriteriaTester (ctx=<optimized out>) at /u/python/external/RediSearch/src/index.c:964

(gdb) info registers
rax            0x7f8895bcc680	140224604456576
rbx            0x0	0
rcx            0x2	2
rdx            0x7f888c319840	140224444340288
rsi            0x7f887fa34bc0	140224233688000
rdi            0x7f8895bcc680	140224604456576
rbp            0x7f888735e900	0x7f888735e900
rsp            0x7ffd08af9000	0x7ffd08af9000
r8             0x558982555100	94049085509888
r9             0x558982555300	94049085510400
r10            0x7f8896d46768	140224622782312
r11            0x7f8895fccf40	140224608653120
r12            0x2	2
r13            0x0	0
r14            0x447	1095
r15            0x1	1
rip            0x7f888c319848	0x7f888c319848 <NI_GetCriteriaTester+8>
eflags         0x10202	[ IF RF ]
cs             0x33	51
ss             0x2b	43
ds             0x0	0
es             0x0	0
fs             0x0	0
gs             0x0	0

[MeirShpilraien]

Can you specify RediSearch version?

[danialfarid]

26964:C 02 Apr 2020 02:19:41.157 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=26964, just started

26780:M 02 Apr 2020 02:18:10.233 # Server initialized
26780:M 02 Apr 2020 02:18:10.234 * <ft> RediSearch version 99.99.99 (Git=v1.6.6-45-gc13f342)

Looks like running this query: FT.SEARCH rli "(-@exs:{CA})|(@au:{cflqa1})" will make redis crash all the time. I have an index rli with a bunch of fields about 50 of them, mostly tag fields and the exc and au are 2 of the tags. If I create another test index and run the same query it works so it has to do with the data that is in redisearch right now which is 20 documents and one of them has the field au value set to cflqa1 and empty string for exs. If I change cflqa1 to anything else the query works without crashing also if I change the order of union to be (@au:{cflqa1})|(-@exs:{CA}) it also works.

I tried creating only one or 2 documents with simplified index with those two fields only but couldn't get it to crash, seems like only for those specific documents and having the index with more fields makes it crash.

[MeirShpilraien]

Can you share an rdb so we can reproduce locally?

[danialfarid]

dump.txt
Here is the output from FT.SEARCH and FT.INFO.

[danialfarid]

Any updates on this?

[MeirShpilraien]

@danialfarid sorry I missed your message.
Do you think it is possible to get an rdb with all the data so it will be possible to reproduce, another option is if you can give us a reproduction steps, just by creating the same index and populate with some data is not enough to reproduce.

[danialfarid]

@MeirShpilraien Here is the rdb file that makes redis crash on this query FT.SEARCH rli "(-@exs:{CA})|(@au:{cflqa1})". https://drive.google.com/open?id=1a0cQdDHKzKE5F-akn4n3LbrDjn24ITXN
(remove .txt)

[MeirShpilraien]

Thanks, will check it out.

[MeirShpilraien]

I could reproduce it using the RDB you gave us, this PR should fix it: #1177

Thanks for reporting.