Manticore crash random time

[vaslatyshev]

Describe the bug
Несколько раз в день крешится searchd.
Из подозрительного за несколько секунд/минут в логах searchd видны записи
[Wed Aug 16 05:14:04.221 2023] [515053] rt: table products: diskchunk 10370(86), segments 29 saved in 1.661892 (1.667161) sec, RAM saved/new 128104536/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)

To Reproduce
Падает совершенно не предзсказуемо, повторить не получается. При этом падения регулярные, на разных запросах в ходе эксплуатации в продуктовой среде.

Describe the environment:

• Manticore Search version: 6.2.0 45680f95d@230804 (columnar 2.2.0 dc33868@230804) (secondary 2.2.0 dc33868@230804)
• OS version: Ubuntu 22.04
• Hardware:

Процессор 2 × Intel Silver 4214R (12x2.4 ГГц HT)
Память 64 ГБ — 8 × 8 ГБ DDR4 ECC Reg
Диск 1000 ГБ SSD NVMe M.2; 2 × 240 ГБ SSD SATA Enterprise
Сетевые карты
2 × 10 GE + port to Private network 10 Гбит/s
Материнская плата
X11DPi-NT

Messages from log files:

[Wed Aug 16 05:14:04.221 2023] [515053] rt: table products: diskchunk 10370(86), segments 29  saved in 1.661892 (1.667161) sec, RAM saved/new 128104536/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
------- FATAL: CRASH DUMP -------
[Wed Aug 16 05:18:31.207 2023] [515034]

--- crashed SphinxQL request dump ---
SELECT
    
    id,
    title,
    1000000000000 * publishing_year + sales_started_at 
as newness,
    weight() as weight
FROM products
WHERE
    MATCH('@(title,author_full_name_list,
translator_full_name_list,category_tree_list,brand,item_title_synonyms,umk_title,school_subject_title_list,
material_type_title) (леска для)')
    AND (available = 5)
    
ORDER BY
 
   
    
    weight() DESC
LIMIT 3
OFFSET 0
OPTION
    max_matches = 3,
    ranker 
= expr('sum((lcs+4*(min_hit_pos==1)+exact_hit*2)*user_weight)*1000+bm25 + (3000*is_book) 
+ (15000*available) + 60*purchase_month_count*available + (300 * is_new)'),
    field_weights 
= (author_full_name_list=9, title=6, series_title=1)
--- request dump end ---
--- local index:products
Manticore 6.2.0 45680f95d@230804 (columnar 2.2.0 dc33868@230804) (secondary 2.2.0 dc33868@230804)
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with Clang 15.0.7
Configured with flags: Configured with these definitions: -DDISTR_BUILD=jammy -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.21 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux x86_64 (jammy) (cross-compiled)
Stack bottom = 0x7f67508ce9a0, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x1)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x1, stack=0x7f67508d0000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x22a)[0x55d73653256a]
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi+0x355)[0x55d7363b1065]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f67f414b520]
/usr/bin/searchd(_ZNK14Expr_GetBits_c7IntEvalERK9CSphMatch+0x35)[0x55d7370f0f35]
/usr/bin/searchd(_ZNK11Expr_Madd_c7IntEvalERK9CSphMatch+0x27)[0x55d7370ed1e7]
/usr/bin/searchd(_ZN19RankerState_Expr_fnILb0ELb1EE8FinalizeERK9CSphMatch+0x34)[0x55d7371366a4]
/usr/bin/searchd(_ZN17ExtRanker_State_TI19RankerState_Expr_fnILb0ELb1EELb1EE10GetMatchesEv+0x4c6)[0x55d7371351d6]
/usr/bin/searchd(_ZNK13CSphIndex_VLN13MatchExtendedILb1ELb0ELb0EEEvR16CSphQueryContextRK9CSphQueryRK11VecTraits_TIP15ISphMatchSorterEP10ISphRankerii+0x86)[0x55d7364fb236]
/usr/bin/searchd(_ZNK13CSphIndex_VLN16ParsedMultiQueryERK9CSphQueryR15CSphQueryResultRK11VecTraits_TIP15ISphMatchSorterERK9XQQuery_t17CSphRefcountedPtrI8CSphDictERK18CSphMultiQueryArgsP18CSphQueryNodeCachel+0xd26)[0x55d736465946]
/usr/bin/searchd(+0xf8fefc)[0x55d7364e6efc]
/usr/bin/searchd(+0xf0d11b)[0x55d73646411b]
/usr/bin/searchd(_ZNK13CSphIndex_VLN10MultiQueryER15CSphQueryResultRK9CSphQueryRK11VecTraits_TIP15ISphMatchSorterERK18CSphMultiQueryArgs+0x909)[0x55d736463eb9]
/usr/bin/searchd(+0x1c1c124)[0x55d737173124]
/usr/bin/searchd(+0x1d21f22)[0x55d737278f22]
/usr/bin/searchd(_ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEESt4pairIN5boost7context13stack_contextENS_14StackFlavour_EEEENUlNS6_6detail10transfer_tEE_8__invokeESB_+0x1c)[0x55d73768187c]
/usr/bin/searchd(make_fcontext+0x37)[0x55d7376a1b57]
Trying boost backtrace:
 0# sphBacktrace(int, bool) in /usr/bin/searchd
 1# CrashLogger::HandleCrash(int) in /usr/bin/searchd
 2# 0x00007F67F414B520 in /lib/x86_64-linux-gnu/libc.so.6
 3# Expr_GetBits_c::IntEval(CSphMatch const&) const in /usr/bin/searchd
 4# Expr_Madd_c::IntEval(CSphMatch const&) const in /usr/bin/searchd
 5# RankerState_Expr_fn<false, true>::Finalize(CSphMatch const&) in /usr/bin/searchd
 6# ExtRanker_State_T<RankerState_Expr_fn<false, true>, true>::GetMatches() in /usr/bin/searchd
 7# void CSphIndex_VLN::MatchExtended<true, false, false>(CSphQueryContext&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, ISphRanker*, int, int) const in /usr/bin/searchd
 8# CSphIndex_VLN::ParsedMultiQuery(CSphQuery const&, CSphQueryResult&, VecTraits_T<ISphMatchSorter*> const&, XQQuery_t const&, CSphRefcountedPtr<CSphDict>, CSphMultiQueryArgs const&, CSphQueryNodeCache*, long) const in /usr/bin/searchd
 9# 0x000055D7364E6EFC in /usr/bin/searchd
10# 0x000055D73646411B in /usr/bin/searchd
11# CSphIndex_VLN::MultiQuery(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&) const in /usr/bin/searchd
12# 0x000055D737173124 in /usr/bin/searchd
13# 0x000055D737278F22 in /usr/bin/searchd
14# Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in /usr/bin/searchd
15# make_fcontext in /usr/bin/searchd

-------------- backtrace ends here ---------------

Индексы можно скачать по ссылке https://drive.google.com/file/d/1XaPsSXW95AnezEDOBAfzMEku8px0G_5n/view?usp=share_link
searchd.log - по ссылке https://drive.google.com/file/d/1u8zr-UsVXlr-Myes99BgIqfz881dLeJy/view?usp=sharing

[sanikolaev]

Should be fixed in 6.2.12.

[Mokto]

Are we talking 6.2.12.0 ? It is still happening in manticoresearch/helm-worker:6.2.12.0

[sanikolaev]

Are we talking 6.2.12.0

Yes. 6.2.12

It is still happening in manticoresearch/helm-worker:6.2.12.0

Exactly same backtrace?

[Mokto]

It was but I can't reproduce anymore.

I'll come back to you if it happens again.