Ипользование JSON аргументов в UDF функциях вызывает крэш #967

Romchegzap · 2022-12-18T08:28:13Z

Describe the bug
При использовании в UDF функции JSON поля в аргументе происходит крэш мантикоры. Интересно что, как я заметил, крэш зависит от кол-ва продуктов через который прогоняется данная функция.

To Reproduce
Steps to reproduce the behavior:

Сделал обычную строчную функцию, которая просто выводит захардкоженное значение '[]'.

#include "sphinxudf.h"    
    
int str_test_ver()    
{    
    return SPH_UDF_VERSION;    
}    
    
int str_test_init ( SPH_UDF_INIT *init, SPH_UDF_ARGS *args, char *error_message )    
{    
    // return a success code    
    return 0;    
}    
    
    
char* str_test ( SPH_UDF_INIT *init, SPH_UDF_ARGS * args, char *error_flag )    
{    
char * out;    
    
out = args->fn_malloc(2   1);    
out[0] = '[';    
out[1] = ']';    
out[2] = '```    
SELECT id, str_test(3, '123', matchings_data) as a FROM products_rt;    
```';    
    
return out;    
}    
    
void str_test_deinit(SPH_UDF_INIT * init)     
{    
    
}

В индексе products_rt - 336000 продуктов.
Вызываю её так

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".    
Core was generated by `searchd --config /etc/manticoresearch/manticore.conf --coredump'.    
Program terminated with signal SIGSEGV, Segmentation fault.    
#0  __GI___libc_free (mem=0x29) at malloc.c:3113    
3113malloc.c: No such file or directory.    
[Current thread is 1 (Thread 0x7f5a0c7d3700 (LWP 6178))]    
(gdb) bt    
#0  __GI___libc_free (mem=0x29) at malloc.c:3113    
#1  0x0000000000927f13 in Expr_Udf_c::FreeArgs (this=0x7f59fc046e10) at /builds/manticoresearch/dev/src/sphinxexpr.cpp:5338    
#2  Expr_UdfStringptr_c::StringEval (this=0x7f59fc046e10, tMatch=..., ppStr=<optimized out>) at /builds/manticoresearch/dev/src/sphinxexpr.cpp:5529    
#3  0x000000000090fdb0 in ISphExpr::StringEvalPacked (this=0x7f59fc046e10, tMatch=...) at /builds/manticoresearch/dev/src/sphinxexpr.cpp:133    
#4  0x00000000006b0a5b in CalcContextItem (tMatch=..., tCalc=...) at /builds/manticoresearch/dev/src/sphinx.cpp:7282    
#5  0x00000000006babbb in CalcContextItems (tMatch=..., dItems=...) at /builds/manticoresearch/dev/src/sphinx.cpp:7309    
#6  CSphQueryContext::CalcFinal (this=<optimized out>, tMatch=...) at /builds/manticoresearch/dev/src/sphinx.cpp:7327    
#7  SphFinalMatchCalc_t::Process (this=0x7f5a0682bee0, pMatch=0x7f59fc073ad8) at /builds/manticoresearch/dev/src/sphinx.cpp:7592    
#8  0x00000000008cf5f8 in CSphMatchQueue<MatchGeneric1_fn, false>::Finalize (this=0x7f59fc0736e0, tProcessor=...,     
    bCallProcessInResultSetOrder=<optimized out>, bFinalizeMatches=<optimized out>) at /builds/manticoresearch/dev/src/sphinxsort.cpp:750    
#9  0x00000000006658e0 in CSphIndex_VLN::MultiScan(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&, long) const::$_21::operator()(ISphMatchSorter*) const (this=0x7f5a0682bf78, p=0x29, p@entry=0x7f5a0682c520)    
    at /builds/manticoresearch/dev/src/sphinx.cpp:8236    
#10 for_each<VecTraits_T<ISphMatchSorter*> const, CSphIndex_VLN::MultiScan(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&, long) const::$_21>(VecTraits_T<ISphMatchSorter*> const&, CSphIndex_VLN::MultiScan(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&, long) const::$_21&&) (dData=..., fnAction=...)    
    at /builds/manticoresearch/dev/src/std/iterations_impl.h:55    
#11 VecTraits_T<ISphMatchSorter*>::Apply<CSphIndex_VLN::MultiScan(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&, long) const::$_21>(CSphIndex_VLN::MultiScan(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&, long) const::$_21&&) const (this=this@entry=0x7f5a0682c520, Verb=...) at /builds/manticoresearch/dev/src/std/vectraits_impl.h:287    
#12 0x000000000066491c in CSphIndex_VLN::MultiScan (this=this@entry=0x7f5a0400ae20, tResult=..., tQuery=..., dSorters=..., tArgs=..., tmMaxTimer=0)    
    at /builds/manticoresearch/dev/src/sphinx.cpp:8236    
#13 0x000000000066ee0c in CSphIndex_VLN::MultiQuery (this=0x7f5a0400ae20, tResult=..., tQuery=..., dAllSorters=..., tArgs=...)    
    at /builds/manticoresearch/dev/src/sphinx.cpp:10657    
#14 0x0000000000676fa2 in RunSplitQuery(CSphIndex const*, CSphQuery const&, CSphQueryResultMeta&, VecTraits_T<ISphMatchSorter*>&, CSphMultiQueryArgs const&, QueryProfile_c*, CSphOrderedHash<long, CSphString, CSphStrHashFunc, 256> const*, long, char const*, int, long)::$_33::operator()() const (    
    this=0x7f5a0003f590) at /builds/manticoresearch/dev/src/sphinx.cpp:10560    
#15 std::_Function_handler<void (), RunSplitQuery(CSphIndex const*, CSphQuery const&, CSphQueryResultMeta&, VecTraits_T<ISphMatchSorter*>&, CSphMultiQueryArgs const&, QueryProfile_c*, CSphOrderedHash<long, CSphString, CSphStrHashFunc, 256> const*, long, char const*, int, long)::$_33>::_M_invoke(std::_Any_data const&) (__functor=...) at /sysroot/root/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c  /7.5.0/bits/std_function.h:316    
#16 0x0000000000aa9462 in std::function<void ()>::operator()() const (this=0x7f5a00040088)    
    at /sysroot/root/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c  /7.5.0/bits/std_function.h:706    
#17 myinfo::OwnMiniNoCount(std::function<void ()>)::$_3::operator()() const (this=0x7f5a00040080) at /builds/manticoresearch/dev/src/task_info.cpp:161    
#18 std::_Function_handler<void (), myinfo::OwnMiniNoCount(std::function<void ()>)::$_3>::_M_invoke(std::_Any_data const&) (__functor=...)    
    at /sysroot/root/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c  /7.5.0/bits/std_function.h:316    
#19 0x0000000000ecbdcc in std::function<void ()>::operator()() const (this=0x7f5a0682d400)    
    at /sysroot/root/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c  /7.5.0/bits/std_function.h:706    
#20 Threads::Coro::ExecuteN(int, std::function<void ()>&&) (iConcurrency=<optimized out>, iConcurrency@entry=2, fnWorker=...)    
    at /builds/manticoresearch/dev/src/coroutine.cpp:793    
#21 0x000000000066e99d in RunSplitQuery (pIndex=0x7f5a0400ae20, tQuery=..., tResult=..., dSorters=..., tArgs=..., pProfiler=0x0,     
---Type <return> to continue, or q <return> to quit---    
    pLocalDocs=<optimized out>, iTotalDocs=<optimized out>, iSplit=2, tmMaxTimer=<optimized out>, szIndexName=<optimized out>)    
    at /builds/manticoresearch/dev/src/sphinx.cpp:10532    
#22 CSphIndex_VLN::SplitQuery (this=this@entry=0x7f5a0400ae20, tResult=..., tQuery=..., dAllSorters=..., tArgs=..., tmMaxTimer=<optimized out>)    
    at /builds/manticoresearch/dev/src/sphinx.cpp:10613    
#23 0x000000000066ec03 in CSphIndex_VLN::MultiQuery (this=0x7f5a0400ae20, tResult=..., tQuery=..., dAllSorters=..., tArgs=...)    
    at /builds/manticoresearch/dev/src/sphinx.cpp:10639    
#24 0x00000000009ac18c in QueryDiskChunks(CSphQuery const&, CSphQueryResultMeta&, CSphMultiQueryArgs const&, RtGuard_t const&, VecTraits_T<ISphMatchSorter*>&, QueryProfile_c*, bool, CSphOrderedHash<long, CSphString, CSphStrHashFunc, 256> const*, long, char const*, SorterSchemaTransform_c&, long)::$_57::operator()() const (this=0x7f5a041a7920) at /builds/manticoresearch/dev/src/sphinxrt.cpp:7242    
#25 std::_Function_handler<void (), QueryDiskChunks(CSphQuery const&, CSphQueryResultMeta&, CSphMultiQueryArgs const&, RtGuard_t const&, VecTraits_T<ISphMatchSorter*>&, QueryProfile_c*, bool, CSphOrderedHash<long, CSphString, CSphStrHashFunc, 256> const*, long, char const*, SorterSchemaTransform_c&, long)::$_57>::_M_invoke(std::_Any_data const&) (__functor=...)    
    at /sysroot/root/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c  /7.5.0/bits/std_function.h:316    
#26 0x0000000000aa9252 in std::function<void ()>::operator()() const (this=0x7f5a041a79e8)    
    at /sysroot/root/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c  /7.5.0/bits/std_function.h:706    
#27 myinfo::OwnMini(std::function<void ()>)::$_2::operator()() const (this=0x7f5a041a79e0) at /builds/manticoresearch/dev/src/task_info.cpp:151    
#28 std::_Function_handler<void (), myinfo::OwnMini(std::function<void ()>)::$_2>::_M_invoke(std::_Any_data const&) (__functor=...)    
    at /sysroot/root/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c  /7.5.0/bits/std_function.h:316    
#29 0x0000000000ece96c in std::function<void ()>::operator()() const (this=0x7f59f0001d60)    
    at /sysroot/root/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c  /7.5.0/bits/std_function.h:706    
#30 Threads::CoRoutine_c::WorkerLowest (this=0x7f59f0001d58, tCtx=<optimized out>) at /builds/manticoresearch/dev/src/coroutine.cpp:85    
#31 Threads::CoRoutine_c::CreateContext(std::function<void ()>, VecTraits_T<unsigned char>)::{lambda(boost::context::detail::transfer_t)#1}::operator()(boost::context::detail::transfer_t) const (pT=..., this=<optimized out>) at /builds/manticoresearch/dev/src/coroutine.cpp:118    
#32 Threads::CoRoutine_c::CreateContext(std::function<void ()>, VecTraits_T<unsigned char>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) (pT=...) at /builds/manticoresearch/dev/src/coroutine.cpp:117    
#33 0x0000000000eec32f in make_fcontext ()    
#34 0x0000000000000000 in ?? ()

Поле matchings_data - json.
Если использовать другие аргументы, например, string, int то всё работает отлично. Но если подставить JSON поле хоть первым аргументом, хоть последним - то происходит крэш
Вот вывод бектрейса из GDB кордампа

SELECT id, str_test(3, '123', matchings_data) as a, LENGTH(a) as l FROM products_rt WHERE l > 2;    
Ошибка -     
[Thread debugging using libthread_db enabled]    
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".    
Core was generated by `searchd --config /etc/manticoresearch/manticore.conf --coredump'.    
Program terminated with signal SIGABRT, Aborted.    
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51    
51../sysdeps/unix/sysv/linux/raise.c: No such file or directory.    
[Current thread is 1 (Thread 0x7f5a0c7f4700 (LWP 18734))]    
(gdb) bt    
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51    
#1  0x00007f5a0a9827f1 in __GI_abort () at abort.c:79    
#2  0x00007f5a0a9cb837 in __libc_message (action=action@entry=do_abort,     
    fmt=fmt@entry=0x7f5a0aaf8a7b "%s\n") at ../sysdeps/posix/libc_fatal.c:181    
#3  0x00007f5a0a9d28ba in malloc_printerr (str=str@entry=0x7f5a0aaf6c8e "free(): invalid size")    
    at malloc.c:5342    
#4  0x00007f5a0a9d9dfc in _int_free (have_lock=0, p=0x7f59fc010ab0, av=0x7f5a0ad2dc40 <main_arena>)    
    at malloc.c:4171    
#5  __GI___libc_free (mem=0x7f59fc010ac0) at malloc.c:3134    
#6  0x0000000000927f13 in Expr_Udf_c::FreeArgs (this=0x7f59fc0103b0)    
    at /builds/manticoresearch/dev/src/sphinxexpr.cpp:5338    
#7  Expr_UdfStringptr_c::StringEval (this=0x7f59fc0103b0, tMatch=..., ppStr=<optimized out>)    
    at /builds/manticoresearch/dev/src/sphinxexpr.cpp:5529    
#8  0x000000000090fdb0 in ISphExpr::StringEvalPacked (this=0x7f59fc0103b0, tMatch=...)    
    at /builds/manticoresearch/dev/src/sphinxexpr.cpp:133    
#9  0x00000000006b0a5b in CalcContextItem (tMatch=..., tCalc=...)    
    at /builds/manticoresearch/dev/src/sphinx.cpp:7282    
#10 0x000000000067d8cb in CalcContextItems (tMatch=..., dItems=...)    
    at /builds/manticoresearch/dev/src/sphinx.cpp:7309    
#11 CSphQueryContext::CalcFilter (this=<optimized out>, tMatch=...)    
    at /builds/manticoresearch/dev/src/sphinx.cpp:7315    
#12 Fullscan<true, false, true, false, false, true, RowIterator_T<true>, CSphIndex_VLN::RunFullscanOnAttrs(RowIdBoundaries_t const&, CSphQueryContext const&, CSphQueryResultMeta&, VecTraits_T<ISphMatchSorter*> const&, CSphMatch&, int, bool, int, long) const::$_16&>(RowIterator_T<true>&, CSphIndex_VLN::RunFullscanOnAttrs(RowIdBoundaries_t const&, CSphQueryContext const&, CSphQueryResultMeta&, VecTraits_T<ISphMatchSorter*> const&, CSphMatch&, int, bool, int, long) const::$_16&, CSphQueryContext const&, CSphQueryResultMeta&, VecTraits_T<ISphMatchSorter*> const&, CSphMatch&, int, int, long) (tIterator=...,     
---Type <return> to continue, or q <return> to quit---    
    ic=..., tCtx=..., tMeta=..., dSorters=..., tMatch=..., iCutoff=20, iIndexWeight=<optimized out>,     
    tmMaxTimer=<optimized out>) at /builds/manticoresearch/dev/src/sphinx.cpp:7804    
#13 0x0000000000660e1d in RunFullscan<RowIterator_T<true>, CSphIndex_VLN::RunFullscanOnAttrs(RowIdBoundaries_t const&, CSphQueryContext const&, CSphQueryResultMeta&, VecTraits_T<ISphMatchSorter*> const&, CSphMatch&, int, bool, int, long) const::$_16&>(RowIterator_T<true>&, CSphIndex_VLN::RunFullscanOnAttrs(RowIdBoundaries_t const&, CSphQueryContext const&, CSphQueryResultMeta&, VecTraits_T<ISphMatchSorter*> const&, CSphMatch&, int, bool, int, long) const::$_16&, CSphQueryContext const&, CSphQueryResultMeta&, VecTraits_T<ISphMatchSorter*> const&, CSphMatch&, int, bool, int, long) (tIterator=..., fnToStatic=..., tCtx=..., tMeta=..., dSorters=...,     
    tMatch=..., iCutoff=8, tmMaxTimer=0, bRandomize=<optimized out>, iIndexWeight=<optimized out>)    
    at /builds/manticoresearch/dev/src/sphinx.cpp:7867    
#14 CSphIndex_VLN::RunFullscanOnAttrs (this=this@entry=0x7f5a068dac70, tBoundaries=..., tCtx=..., tMeta=..., dSorters=...,     
    tMatch=..., iCutoff=<optimized out>, bRandomize=<optimized out>, iIndexWeight=<optimized out>, tmMaxTimer=<optimized out>)    
    at /builds/manticoresearch/dev/src/sphinx.cpp:7887    
#15 0x00000000006b0eee in CSphIndex_VLN::ScanByBlocks<false> (this=this@entry=0x7f5a0400ae20, tCtx=..., tMeta=..., dSorters=...,     
    tMatch=..., iCutoff=<optimized out>, iCutoff@entry=20, bRandomize=<optimized out>, iIndexWeight=<optimized out>,     
    tmMaxTimer=<optimized out>, pBoundaries=<optimized out>) at /builds/manticoresearch/dev/src/sphinx.cpp:7950    
#16 0x00000000006647ff in CSphIndex_VLN::MultiScan (this=this@entry=0x7f5a0400ae20, tResult=..., tQuery=..., dSorters=...,     
    tArgs=..., tmMaxTimer=<optimized out>) at /builds/manticoresearch/dev/src/sphinx.cpp:8205    
#17 0x000000000066ee0c in CSphIndex_VLN::MultiQuery (this=0x7f5a0400ae20, tResult=..., tQuery=..., dAllSorters=..., tArgs=...)    
    at /builds/manticoresearch/dev/src/sphinx.cpp:10657    
#18 0x00000000009ac18c in QueryDiskChunks(CSphQuery const&, CSphQueryResultMeta&, CSphMultiQueryArgs const&, RtGuard_t const&, VecTraits_T<ISphMatchSorter*>&, QueryProfile_c*, bool, CSphOrderedHash<long, CSphString, CSphStrHashFunc, 256> const*, long, char const*, SorterSchemaTransform_c&, long)::$_57::operator()() const (this=0x7f5a068bb3a0) at /builds/manticoresearch/dev/src/sphinxrt.cpp:7242    
#19 std::_Function_handler<void (), QueryDiskChunks(CSphQuery const&, CSphQueryResultMeta&, CSphMultiQueryArgs const&, RtGuard_t const&, VecTraits_T<ISphMatchSorter*>&, QueryProfile_c*, bool, CSphOrderedHash<long, CSphString, CSphStrHashFunc, 256> const*, long, char const*, SorterSchemaTransform_c&, long)::$_57>::_M_invoke(std::_Any_data const&) (__functor=...)    
    at /sysroot/root/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c  /7.5.0/bits/std_function.h:316    
#20 0x0000000000aa9252 in std::function<void ()>::operator()() const (this=0x7f5a0688fd98)    
    at /sysroot/root/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c  /7.5.0/bits/std_function.h:706    
#21 myinfo::OwnMini(std::function<void ()>)::$_2::operator()() const (this=0x7f5a0688fd90)    
    at /builds/manticoresearch/dev/src/task_info.cpp:151    
#22 std::_Function_handler<void (), myinfo::OwnMini(std::function<void ()>)::$_2>::_M_invoke(std::_Any_data const&) (__functor=...)    
    at /sysroot/root/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c  /7.5.0/bits/std_function.h:316    
#23 0x0000000000ece96c in std::function<void ()>::operator()() const (this=0x7f59f8002810)    
    at /sysroot/root/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c  /7.5.0/bits/std_function.h:706    
#24 Threads::CoRoutine_c::WorkerLowest (this=0x7f59f8002808, tCtx=<optimized out>)    
    at /builds/manticoresearch/dev/src/coroutine.cpp:85    
#25 Threads::CoRoutine_c::CreateContext(std::function<void ()>, VecTraits_T<unsigned char>)::{lambda(boost::context::detail::transfer---Type <return> to continue, or q <return> to quit---q

И еще вот второй кейс

%CODE_BLOCK%

Еще ловил ошибку double free(), но повторить её не смог

Expected behavior
Крэша мантикоры не должно быть при использовании json полей как аргументов в udf фукнции.

Describe the environment:

Manticore 5.0.3 a477a69@220907 dev
Linux dev.priceshape.dk 4.15.0-196-generic #207-Ubuntu SMP Thu Oct 27 21:24:58 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Messages from log files:

Additional context

The text was updated successfully, but these errors were encountered:

sanikolaev · 2022-12-19T08:28:44Z

Manticore 5.0.3

Can you reproduce the crash in the latest dev version?

Romchegzap · 2022-12-21T12:51:24Z

Установил
Server version: 5.0.3 260dee9@221214 dev
Так же повторяется крэш и зависания даже

UPD: бывает что отрабатывает 1-2 раза и потом падает

tomatolog · 2023-03-28T07:48:01Z

fixed daemon crash on processing search with pseudo-sharding enabled and UDF with JSON argument at the 02977de

You need to install package from the dev repository to get this crash fixed

sanikolaev added the waiting Waiting for the original poster (in most cases) or something else label Dec 19, 2022

sanikolaev added bug and removed waiting Waiting for the original poster (in most cases) or something else labels Jan 2, 2023

tomatolog closed this as completed Mar 28, 2023

chenrui333 mentioned this issue Aug 4, 2023

manticoresearch 6.2.0 Homebrew/homebrew-core#138555

Closed

sanikolaev added the rel::6.2.0 label Sep 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ипользование JSON аргументов в UDF функциях вызывает крэш #967

Ипользование JSON аргументов в UDF функциях вызывает крэш #967

Romchegzap commented Dec 18, 2022 •

edited by githubmanticore

sanikolaev commented Dec 19, 2022

Romchegzap commented Dec 21, 2022 •

edited

tomatolog commented Mar 28, 2023

Ипользование JSON аргументов в UDF функциях вызывает крэш #967

Ипользование JSON аргументов в UDF функциях вызывает крэш #967

Comments

Romchegzap commented Dec 18, 2022 • edited by githubmanticore

sanikolaev commented Dec 19, 2022

Romchegzap commented Dec 21, 2022 • edited

tomatolog commented Mar 28, 2023

Romchegzap commented Dec 18, 2022 •

edited by githubmanticore

Romchegzap commented Dec 21, 2022 •

edited