Manticore crashes unexpected and cant to normal restart

[pbabkin]

Describe the bug
Manticore crashes unexpected and cant to normal restart (via systemctl) and process requests

To Reproduce
Cant to reproduce manually

Describe the environment:
4.2.1 611d55d@220319 release (columnar 1.14.1 fce43f7@220203) and 4.2.0 installed via official repos
Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-96-generic x86_64)

Messages from log files:
searchd log fragments and coredumps with gdb debug info from several cases uploaded on FTP (dev.manticoresearch.com). Crashes happens once or twice per day. I can provide any info by request.

Additional context
As a rule, crashes happen immediately or almost immediately after creating a new RT-indexes (based on almost same data) and deleting old ones. The crash may happen during process any SQL query: select, insert, truncate...

[tomatolog]

could you test recent dev version package (https://repo.manticoresearch.com/repository/manticoresearch_focal_dev/dists/focal/main/binary-amd64/manticore_4.2.1-220404-e85558b1a_amd64.deb) as from log files and GDB coredumps I see these crashes were from memory corruptions and since 611d55d we already fixed some memory leaks and memory related issues.

[pbabkin]

sounds great. already updated Manticore to a latest dev version. I'll observe few days and give feedback.
Thanks! 🤞

[pbabkin]

unfortunately it crashed again.
fresh coredumps, debug info from GDB and searchd fragments uploaded on ftp (..\issue-743\case 12\ and newer)

the correct version manticore-dbgsym also installed
(https://repo.manticoresearch.com/repository/manticoresearch_focal_dev/dists/focal/main/binary-amd64/manticore-dbgsym_4.2.1-220404-e85558b1a_amd64.ddeb)

[tomatolog]

from core file you dumped at case 12 I see that crash ended in re2::DFA::InlinedSearchLoop at parse query stage however at core dump I do not see what regexp_filter got checked - could you provide you index idxoff_test_minutashop_ru_defd2782e48f427d8c2cb1a40859c7c9 along with config to check reproduce this crash here locally?

[tomatolog]

Other crashes you reported related to data daemon try to process.
Could you check your crash log for record local index:index_name then upload that index into our FTP to preproduce this crash locally?

[pbabkin]

could you provide you index along with config to check reproduce this crash here locally?

already uploaded whole index folder + show create table sql into "case 12" subfolder. However, it is not exactly the same index. The structure of the index remains the same, but the data is slightly different.

Could you check your crash log for record local index:index_name then upload that index into our FTP to preproduce this crash locally?

as soon as the crash happens again, I will upload fresh index data with debug info and give feedback here.

P.S.: I will update the manticore today to the latest version 4.2.1 4a07772@220419 release (columnar 1.14.1 fce43f7@220203).

[pbabkin]

hmm... there were two crashes today. both happened in the process of creating new indexes and then deleting the old ones
Also, both crashes, judging by the searchd.log happened during the execution of the DROP TABLE IF EXISTS some_index_name command 🤔
(note: FYI. i dont need use replace into syntax or transactions for refill same index due some reasons - I need to create exactly a new index to replace the old one. when new index will be created, my app switches queries on that new index, awaites a bit, and then drop old index).

GDBs, coredumps, index exmpl and searchd.log uploaded into "case 16" subfolder

[pbabkin]

Could you check your crash log for record local index:index_name then upload that index into our FTP to preproduce this crash locally?

case 17 uploaded

[pbabkin]

Case 18 uploaded to FTP. Managed to catch the crash after just a few minutes. the index on which the crash happened exactly and all the debug info is attached.

[tomatolog]

I've reproduced the memory corruption at disk chunk on search query with ranker=expr and multiple terms these got transformed by lemmatizer with data you provide and going to fix that crash and push the fix into master after upcoming release.

However I can not reproduce or trace any other crashes you also reported and provided core files. These could be from that memory corruption or not it is not clear.

I will inform you on the fix .

[tomatolog]

I've fixed crash on data you provided but the fix is in the branch until release test will finish. After that fix will be merged into master version and you could get package with the fix from the dev repo.

However I uploaded package from that branch after CI passed here
https://dev.manticoresearch.com/packages/4b368a9d_focal/manticore_4.2.1-220503-4b368a9d2_amd64.deb
https://dev.manticoresearch.com/packages/4b368a9d_focal/manticore-dbgsym_4.2.1-220503-4b368a9d2_amd64.ddeb

that you could install and test is these crashes fixed for you or not.

[pbabkin]

However I uploaded package from that branch

thank you!
Right now I'm testing one hypothesis that might be causing the crashes. I'm talking about special characters (logic operators), such as "(", ")", "|", which are listed in my indexes as blend_chars and exists in indexes data. I excluded it from blend_chars and awaiting for new crashes. If new crashes happen i will test new versions that you provided at once .

[pbabkin]

I'm talking about special characters (logic operators), such as "(", ")", "|", which are listed in my indexes as blend_chars and exists in indexes data. I excluded it from blend_chars and awaiting for new crashes

Manticore crashes stopped after making changes to blend_chars. Not a single crases happened in almost a week. while before, crases happened several times a day

PS: #783

[sanikolaev]

Manticore crashes stopped after making changes to blend_chars

In what version?

[pbabkin]

In what version?

4.2.1 4a07772@220419 release (columnar 1.14.1 fce43f7@220203) git branch HEAD (no branch)

[tomatolog]

I just merged this branch into master and you could grab package from the dev repo when CI passed