6.3.0 - Query crashes cluster nodes when sent via master

[kashak88]

Bug Description:

Hi,
It seems like a query that exceeds sort-by attributes limit of 5 is allowed to be sent to cluster nodes from master and doing so crashes those nodes, for example:
When I run the query directly on the cluster node:

[root@grs-uat-manticore1 ~]# mysql -h10.48.1.119 -P9312
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 40
Server version: 6.3.0 1811a9efb@24052209 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206) git branch HEAD (no branch)

Copyright (c) 2000, 2024, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> SELECT id   FROM RewardTranslationIndexEn WHERE  MATCH('@(model,brand,name,group_name) (=airpods)|(airpods*) @gpc_keyword _2172_')  AND ANY(gallery_partner_catalog_id)=2172  AND is_searchable=1  GROUP BY reward_group_id WITHIN GROUP ORDER BY total_cost ASC  ORDER BY featured_priority DESC, popularity DESC, is_new DESC, release_date DESC, id DESC, WEIGHT() DESC  LIMIT 0, 50  OPTION max_matches=150000, field_weights=(model=2, brand=10, name=600, group_name=600)  FACET category_facets.catalog_1452._t2level1 AS category_facets__t2level1 LIMIT 2000 FACET category_facets.catalog_1452._t2level2 AS category_facets__t2level2 LIMIT 2000 FACET category_facets.catalog_1452._t2level3 AS category_facets__t2level3 LIMIT 2000 FACET category_facets.catalog_1452._t2level10 AS category_facets__t2level10 LIMIT 2000 ;
ERROR 1064 (42000): table RewardTranslationIndexEn: too many sort-by attributes; maximum count is 5

When I run it on master and it's forwarded to the node above:

[root@grs-uat-manticore-master ~]# mysql -h10.48.1.121 -P9312
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2132
Server version: 6.3.0 1811a9efb@24052209 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206) git branch HEAD (no branch)

Copyright (c) 2000, 2024, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> SELECT id   FROM RewardTranslationIndexEn WHERE  MATCH('@(model,brand,name,group_name) (=airpods)|(airpods*) @gpc_keyword _2172_')  AND ANY(gallery_partner_catalog_id)=2172  AND is_searchable=1  GROUP BY reward_group_id WITHIN GROUP ORDER BY total_cost ASC  ORDER BY featured_priority DESC, popularity DESC, is_new DESC, release_date DESC, id DESC, WEIGHT() DESC  LIMIT 0, 50  OPTION max_matches=150000, field_weights=(model=2, brand=10, name=600, group_name=600)  FACET category_facets.catalog_1452._t2level1 AS category_facets__t2level1 LIMIT 2000 FACET category_facets.catalog_1452._t2level2 AS category_facets__t2level2 LIMIT 2000 FACET category_facets.catalog_1452._t2level3 AS category_facets__t2level3 LIMIT 2000 FACET category_facets.catalog_1452._t2level10 AS category_facets__t2level10 LIMIT 2000 ;
ERROR 1064 (42000): table RewardTranslationIndexEn: agent 10.48.1.119:9315: agent closed connection;
table RewardTranslationIndexEn: agent 10.48.1.120:9315: agent closed connection

Crash log from grs-uat-manticore1:

------- FATAL: CRASH DUMP -------
[Wed May 29 13:22:38.843 2024] [2262459]

--- crashed SphinxAPI request dump ---
AAABJgAAEz8AAAAVAAAABQAABEAAAAAAAAJJ8AAAAAYAAAAAAAAABAAAAA50b3RhbF9jb3N0IEFTQwAAAEhAKG1v
ZGVsLGJyYW5kLG5hbWUsZ3JvdXBfbmFtZSkgKD1haXJwb2RzKXwoYWlycG9kcyopIEBncGNfa2V5d29y
ZCBfMjE3Ml8AAAAAAAAAGFJld2FyZFRyYW5zbGF0aW9uSW5kZXhFbgAAAAEAAAAAAAAAAP//////////
AAAAAgAAABpnYWxsZXJ5X3BhcnRuZXJfY2F0YWxvZ19pZAAAAAAAAAABAAAAAAAACHwAAAAAAAAAAQAA
AAEAAAAAAAAAAAAAAAEAAAANaXNfc2VhcmNoYWJsZQAAAAAAAAABAAAAAAAAAAEAAAAAAAAAAQAA
AAEAAAAAAAAAAAAAAAAAAAAEAAAAD3Jld2FyZF9ncm91cF9pZAACSfAAAABfZmVhdHVyZWRfcHJpb3Jp
dHkgREVTQywgcG9wdWxhcml0eSBERVNDLCBpc19uZXcgREVTQywgcmVsZWFzZV9kYXRlIERFU0MsIGlk
IERFU0MsIFdFSUdIVCgpIERFU0P/////AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAVtb2Rl
bAAAAAIAAAAFYnJhbmQAAAAKAAAABG5hbWUAAAJYAAAACmdyb3VwX25hbWUAAAJYAAAAAAAAAAAA
AAACaWQAAAAAAAAAAAAAAAAAAAAA/////wAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAYAAAACaWQAAAACaWQAAAAAAAAAGWNhdGVnb3J5X2ZhY2V0c19fdDJsZXZlbDEAAAAmY2F0ZWdv
cnlfZmFjZXRzLmNhdGFsb2dfMTQ1Mi5fdDJsZXZlbDEAAAAAAAAACGNvdW50KCopAAAACGNvdW50KCop
AAAAAAAAABljYXRlZ29yeV9mYWNldHNfX3QybGV2ZWwyAAAAJmNhdGVnb3J5X2ZhY2V0cy5jYXRh
bG9nXzE0NTIuX3QybGV2ZWwyAAAAAAAAABljYXRlZ29yeV9mYWNldHNfX3QybGV2ZWwzAAAAJmNhdGVn
b3J5X2ZhY2V0cy5jYXRhbG9nXzE0NTIuX3QybGV2ZWwzAAAAAAAAABpjYXRlZ29yeV9mYWNldHNfX3Qy
bGV2ZWwxMAAAACdjYXRlZ29yeV9mYWNldHMuY2F0YWxvZ18xNDUyLl90MmxldmVsMTAAAAAAAAAAAQAA
AAJpZAAAAAJpZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJAAAAAAAACSfAAAAAGAAAA
AAAAAAQAAAAMQHdlaWdodCBkZXNjAAAASEAobW9kZWwsYnJhbmQsbmFtZSxncm91cF9uYW1lKSAoPWFp
cnBvZHMpfChhaXJwb2RzKikgQGdwY19rZXl3b3JkIF8yMTcyXwAAAAAAAAAYUmV3YXJkVHJhbnNsYXRp
b25JbmRleEVuAAAAAQAAAAAAAAAA//////////8AAAACAAAAGmdhbGxlcnlfcGFydG5lcl9jYXRhbG9n
X2lkAAAAAAAAAAEAAAAAAAAIfAAAAAAAAAABAAAAAQAAAAAAAAAAAAAAAQAAAA1pc19zZWFyY2hh
YmxlAAAAAAAAAAEAAAAAAAAAAQAAAAAAAAABAAAAAQAAAAAAAAAAAAAAAAAAAAQAAAAmY2F0ZWdvcnlf
ZmFjZXRzLmNhdGFsb2dfMTQ1Mi5fdDJsZXZlbDEAAknwAAAADEB3ZWlnaHQgZGVzY/////8AAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAmY2F0ZWdvcnlfZmFjZXRzLmNhdGFsb2dfMTQ1
Mi5fdDJsZXZlbDEAAAAAAAAAAAAAAAAAAAAA/////wAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAYAAAACaWQAAAACaWQAAAAAAAAAGWNhdGVnb3J5X2ZhY2V0c19fdDJsZXZlbDEA
AAAmY2F0ZWdvcnlfZmFjZXRzLmNhdGFsb2dfMTQ1Mi5fdDJsZXZlbDEAAAAAAAAACGNvdW50KCopAAAA
CGNvdW50KCopAAAAAAAAABljYXRlZ29yeV9mYWNldHNfX3QybGV2ZWwyAAAAJmNhdGVnb3J5X2ZhY2V0
cy5jYXRhbG9nXzE0NTIuX3QybGV2ZWwyAAAAAAAAABljYXRlZ29yeV9mYWNldHNfX3QybGV2ZWwz
AAAAJmNhdGVnb3J5X2ZhY2V0cy5jYXRhbG9nXzE0NTIuX3QybGV2ZWwzAAAAAAAAABpjYXRlZ29yeV9m
YWNldHNfX3QybGV2ZWwxMAAAACdjYXRlZ29yeV9mYWNldHMuY2F0YWxvZ18xNDUyLl90MmxldmVsMTAA
AAAAAAAAAgAAABljYXRlZ29yeV9mYWNldHNfX3QybGV2ZWwxAAAAJmNhdGVnb3J5X2ZhY2V0cy5jYXRh
bG9nXzE0NTIuX3QybGV2ZWwxAAAAAAAAAAhjb3VudCgqKQAAAAhjb3VudCgqKQAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAJAAAAAAAACSfAAAAAGAAAAAAAAAAQAAAAMQHdlaWdodCBkZXNjAAAA
SEAobW9kZWwsYnJhbmQsbmFtZSxncm91cF9uYW1lKSAoPWFpcnBvZHMpfChhaXJwb2RzKikgQGdwY19r
ZXl3b3JkIF8yMTcyXwAAAAAAAAAYUmV3YXJkVHJhbnNsYXRpb25JbmRleEVuAAAAAQAAAAAAAAAA////
//////8AAAACAAAAGmdhbGxlcnlfcGFydG5lcl9jYXRhbG9nX2lkAAAAAAAAAAEAAAAAAAAIfAAA
AAAAAAABAAAAAQAAAAAAAAAAAAAAAQAAAA1pc19zZWFyY2hhYmxlAAAAAAAAAAEAAAAAAAAAAQAAAAAA
AAABAAAAAQAAAAAAAAAAAAAAAAAAAAQAAAAmY2F0ZWdvcnlfZmFjZXRzLmNhdGFsb2dfMTQ1Mi5fdDJs
ZXZlbDIAAknwAAAADEB3ZWlnaHQgZGVzY/////8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAmY2F0ZWdvcnlfZmFjZXRzLmNhdGFsb2dfMTQ1Mi5fdDJsZXZlbDIAAAAAAAAAAAAA
AAAAAAAA/////wAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAYAAAACaWQAAAAC
aWQAAAAAAAAAGWNhdGVnb3J5X2ZhY2V0c19fdDJsZXZlbDEAAAAmY2F0ZWdvcnlfZmFjZXRzLmNhdGFs
b2dfMTQ1Mi5fdDJsZXZlbDEAAAAAAAAACGNvdW50KCopAAAACGNvdW50KCopAAAAAAAAABljYXRlZ29y
eV9mYWNldHNfX3QybGV2ZWwyAAAAJmNhdGVnb3J5X2ZhY2V0cy5jYXRhbG9nXzE0NTIuX3QybGV2
ZWwyAAAAAAAAABljYXRlZ29yeV9mYWNldHNfX3QybGV2ZWwzAAAAJmNhdGVnb3J5X2ZhY2V0cy5jYXRh
bG9nXzE0NTIuX3QybGV2ZWwzAAAAAAAAABpjYXRlZ29yeV9mYWNldHNfX3QybGV2ZWwxMAAAACdjYXRl
Z29yeV9mYWNldHMuY2F0YWxvZ18xNDUyLl90MmxldmVsMTAAAAAAAAAAAgAAABljYXRlZ29yeV9mYWNl
dHNfX3QybGV2ZWwyAAAAJmNhdGVnb3J5X2ZhY2V0cy5jYXRhbG9nXzE0NTIuX3QybGV2ZWwyAAAA
AAAAAAhjb3VudCgqKQAAAAhjb3VudCgqKQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJAAAAA
AAACSfAAAAAGAAAAAAAAAAQAAAAMQHdlaWdodCBkZXNjAAAASEAobW9kZWwsYnJhbmQsbmFtZSxncm91
cF9uYW1lKSAoPWFpcnBvZHMpfChhaXJwb2RzKikgQGdwY19rZXl3b3JkIF8yMTcyXwAAAAAAAAAYUmV3
YXJkVHJhbnNsYXRpb25JbmRleEVuAAAAAQAAAAAAAAAA//////////8AAAACAAAAGmdhbGxlcnlfcGFy
dG5lcl9jYXRhbG9nX2lkAAAAAAAAAAEAAAAAAAAIfAAAAAAAAAABAAAAAQAAAAAAAAAAAAAAAQAAAA1p
c19zZWFyY2hhYmxlAAAAAAAAAAEAAAAAAAAAAQAAAAAAAAABAAAAAQAAAAAAAAAAAAAAAAAAAAQAAAAm
Y2F0ZWdvcnlfZmFjZXRzLmNhdGFsb2dfMTQ1Mi5fdDJsZXZlbDMAAknwAAAADEB3ZWlnaHQgZGVz
Y/////8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAmY2F0ZWdvcnlfZmFjZXRz
LmNhdGFsb2dfMTQ1Mi5fdDJsZXZlbDMAAAAAAAAAAAAAAAAAAAAA/////wAAAAAAAAAAAAAAAQAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAYAAAACaWQAAAACaWQAAAAAAAAAGWNhdGVnb3J5X2ZhY2V0c19f
dDJsZXZlbDEAAAAmY2F0ZWdvcnlfZmFjZXRzLmNhdGFsb2dfMTQ1Mi5fdDJsZXZlbDEAAAAAAAAA
CGNvdW50KCopAAAACGNvdW50KCopAAAAAAAAABljYXRlZ29yeV9mYWNldHNfX3QybGV2ZWwyAAAAJmNh
dGVnb3J5X2ZhY2V0cy5jYXRhbG9nXzE0NTIuX3QybGV2ZWwyAAAAAAAAABljYXRlZ29yeV9mYWNldHNf
X3QybGV2ZWwzAAAAJmNhdGVnb3J5X2ZhY2V0cy5jYXRhbG9nXzE0NTIuX3QybGV2ZWwzAAAAAAAAABpj
YXRlZ29yeV9mYWNldHNfX3QybGV2ZWwxMAAAACdjYXRlZ29yeV9mYWNldHMuY2F0YWxvZ18xNDUy
Ll90MmxldmVsMTAAAAAAAAAAAgAAABljYXRlZ29yeV9mYWNldHNfX3QybGV2ZWwzAAAAJmNhdGVnb3J5
X2ZhY2V0cy5jYXRhbG9nXzE0NTIuX3QybGV2ZWwzAAAAAAAAAAhjb3VudCgqKQAAAAhjb3VudCgqKQAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJAAAAAAAACSfAAAAAGAAAAAAAAAAQAAAAMQHdlaWdo
dCBkZXNjAAAASEAobW9kZWwsYnJhbmQsbmFtZSxncm91cF9uYW1lKSAoPWFpcnBvZHMpfChhaXJw
b2RzKikgQGdwY19rZXl3b3JkIF8yMTcyXwAAAAAAAAAYUmV3YXJkVHJhbnNsYXRpb25JbmRleEVuAAAA
AQAAAAAAAAAA//////////8AAAACAAAAGmdhbGxlcnlfcGFydG5lcl9jYXRhbG9nX2lkAAAAAAAAAAEA
AAAAAAAIfAAAAAAAAAABAAAAAQAAAAAAAAAAAAAAAQAAAA1pc19zZWFyY2hhYmxlAAAAAAAAAAEAAAAA
AAAAAQAAAAAAAAABAAAAAQAAAAAAAAAAAAAAAAAAAAQAAAAnY2F0ZWdvcnlfZmFjZXRzLmNhdGFs
b2dfMTQ1Mi5fdDJsZXZlbDEwAAJJ8AAAAAxAd2VpZ2h0IGRlc2P/////AAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAJ2NhdGVnb3J5X2ZhY2V0cy5jYXRhbG9nXzE0NTIuX3QybGV2ZWwx
MAAAAAAAAAAAAAAAAAAAAAD/////AAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
BgAAAAJpZAAAAAJpZAAAAAAAAAAZY2F0ZWdvcnlfZmFjZXRzX190MmxldmVsMQAAACZjYXRlZ29y
eV9mYWNldHMuY2F0YWxvZ18xNDUyLl90MmxldmVsMQAAAAAAAAAIY291bnQoKikAAAAIY291bnQoKikA
AAAAAAAAGWNhdGVnb3J5X2ZhY2V0c19fdDJsZXZlbDIAAAAmY2F0ZWdvcnlfZmFjZXRzLmNhdGFsb2df
MTQ1Mi5fdDJsZXZlbDIAAAAAAAAAGWNhdGVnb3J5X2ZhY2V0c19fdDJsZXZlbDMAAAAmY2F0ZWdvcnlf
ZmFjZXRzLmNhdGFsb2dfMTQ1Mi5fdDJsZXZlbDMAAAAAAAAAGmNhdGVnb3J5X2ZhY2V0c19fdDJs
ZXZlbDEwAAAAJ2NhdGVnb3J5X2ZhY2V0cy5jYXRhbG9nXzE0NTIuX3QybGV2ZWwxMAAAAAAAAAACAAAA
GmNhdGVnb3J5X2ZhY2V0c19fdDJsZXZlbDEwAAAAJ2NhdGVnb3J5X2ZhY2V0cy5jYXRhbG9nXzE0NTIu
X3QybGV2ZWwxMAAAAAAAAAAIY291bnQoKikAAAAIY291bnQoKikAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAA
--- request dump end ---
--- local index:RewardTranslationIndexEn
Manticore 6.3.0 1811a9efb@24052209 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with Clang 16.0.6
Configured with flags: Configured with these definitions: -DDISTR_BUILD=focal -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.21 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux x86_64 (focal) (cross-compiled)
Stack bottom = 0x7f3ecc063fb0, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x20000)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x20000, stack=0x7f3ecc070000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x227)[0x55e955760d87]
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi+0x364)[0x55e9555d67c4]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f3f18036420]
/usr/bin/searchd(_Z10SendResultiR16ISphOutputBufferRK12AggrResult_tbRK9CSphQueryt+0xaaf)[0x55e9555e1e1f]
/usr/bin/searchd(_Z19HandleCommandSearchR16ISphOutputBuffertR13InputBuffer_c+0x387)[0x55e9555f57d7]
/usr/bin/searchd(_Z17ExecuteApiCommand16SearchdCommand_etiR13InputBuffer_cR21GenericOutputBuffer_c+0xcd)[0x55e9555feadd]
/usr/bin/searchd(_Z8ApiServeSt10unique_ptrI16AsyncNetBuffer_cSt14default_deleteIS0_EE+0x7b5)[0x55e95553b785]
/usr/bin/searchd(_Z10MultiServeSt10unique_ptrI16AsyncNetBuffer_cSt14default_deleteIS0_EESt4pairIitE7Proto_e+0x108)[0x55e9555397c8]
/usr/bin/searchd(+0xdf825f)[0x55e95553a25f]
/usr/bin/searchd(_ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEESt4pairIN5boost7context13stack_contextENS_14StackFlavour_EEEENUlNS6_6detail10transfer_tEE_8__invokeESB_+0x1c)[0x55e95677c19c]
/usr/bin/searchd(make_fcontext+0x37)[0x55e9567be607]
Trying boost backtrace:
 0# sphBacktrace(int, bool) in /usr/bin/searchd
 1# CrashLogger::HandleCrash(int) in /usr/bin/searchd
 2# 0x00007F3F18036420 in /lib/x86_64-linux-gnu/libpthread.so.0
 3# SendResult(int, ISphOutputBuffer&, AggrResult_t const&, bool, CSphQuery const&, unsigned short) in /usr/bin/searchd
 4# HandleCommandSearch(ISphOutputBuffer&, unsigned short, InputBuffer_c&) in /usr/bin/searchd
 5# ExecuteApiCommand(SearchdCommand_e, unsigned short, int, InputBuffer_c&, GenericOutputBuffer_c&) in /usr/bin/searchd
 6# ApiServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >) in /usr/bin/searchd
 7# MultiServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >, std::pair<int, unsigned short>, Proto_e) in /usr/bin/searchd
 8# 0x000055E95553A25F in /usr/bin/searchd
 9# Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in /usr/bin/searchd
10# make_fcontext in /usr/bin/searchd

-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the manual
(https://manual.manticoresearch.com/Reporting_bugs)
Dump with GDB via watchdog
--- active threads ---
thd 0 (work_3), proto sphinx, state query, command search
--- Totally 2 threads, and 1 client-working threads ---
------- CRASH DUMP END -------

I image that this can be replicated on your side without our indices.
Master config is pretty simple:

[root@grs-uat-manticore-master manticoresearch]# cat manticore.conf
#
index RewardTranslationIndexEn
{
    type    = distributed
    agent   = 10.48.1.119:9315:RewardTranslationIndexEn
    agent   = 10.48.1.120:9315:RewardTranslationIndexEn
}

index CouponTranslationIndexEn
{
    type     = distributed
    agent    = 10.48.1.119:9315:CouponTranslationIndexEn
    agent    = 10.48.1.120:9315:CouponTranslationIndexEn
}

index RewardCost
{
    type    = distributed
    agent   = 10.48.1.119:9315:RewardCost
    agent   = 10.48.1.120:9315:RewardCost
}

index CategoryIndexEn
{
    type    = distributed
    agent   = 10.48.1.119:9315:CategoryIndexEn
    agent   = 10.48.1.120:9315:CategoryIndexEn
}

index BrandIndex
{
    type    = distributed
    agent   = 10.48.1.119:9315:BrandIndex
    agent   = 10.48.1.120:9315:BrandIndex
}


common
{
    plugin_dir          = /var/lib/manticore/data/plugins
}

searchd
{
    # ------------------------------------------------
    # Address, Port, Protocol
    # ------------------------------------------------
    listen              = 10.48.1.121:9312:mysql
    listen_tfo		= 1
    net_workers		= 2
    ha_ping_interval	= 0
    pseudo_sharding     = 1
    listen_backlog      = 4096
    net_wait_tm		= 4
    qcache_max_bytes = 0
    agent_connect_timeout	= 10s
    read_buffer_hits = 128K
    read_buffer_docs = 128K
    buddy_path = #
    # ------------------------------------------------
    # Agent Configurations
    # ------------------------------------------------
    max_packet_size     = 128M
    agent_query_timeout = 10000

    # ------------------------------------------------
    # Logging
    # ------------------------------------------------
    log                 = /var/log/manticore/searchd.log
    query_log           = /var/log/manticore/searchd_query.log
    query_log_format    = sphinxql
    binlog_path         =

    # ------------------------------------------------
    # Timeouts
    # ------------------------------------------------
    network_timeout     = 60s
    client_timeout      = 3600

    # ------------------------------------------------
    # Config
    # ------------------------------------------------
#    max_children        = 0
    pid_file            = /var/run/manticore/searchd.pid
    seamless_rotate     = 1
    preopen_indexes     = 1
    unlink_old          = 1

    # ------------------------------------------------
    # UDF - User Defined Function Info
    # ------------------------------------------------
#    workers             = threads
    sphinxql_state      = /var/lib/manticore/data/state.sql
}

#searchd {
#    listen = 127.0.0.1:9312
#    listen = 127.0.0.1:9306:mysql
#    listen = 127.0.0.1:9308:http
#    log = /var/log/manticore/searchd.log
#    query_log = /var/log/manticore/query.log
#    pid_file = /var/run/manticore/searchd.pid
#    data_dir = /var/lib/manticore
#    query_log_format = sphinxql
#}

# --eof--

Manticore Search Version:

6.3.0

Operating System Version:

ubuntu 20.04

Have you tried the latest development version?

• Yes

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

• Task estimated
• Specification created, reviewed, and approved
• Implementation completed
• Tests developed
• Documentation updated
• Documentation proofread
• Changelog updated

[sanikolaev]

Thanks @kashak88

MRE

mysql> drop table if exists t; create table t(a int, b int, c int, d int, e int, f int); insert into t(id) values(1); drop table if exists d; create table d type='distributed' agent='127.0.0.1:9312:t'; select id from d group by a order by a desc, b desc, c desc, d desc, e desc, f desc facet a;
--------------
drop table if exists t
--------------

Query OK, 0 rows affected (0.01 sec)

--------------
create table t(a int, b int, c int, d int, e int, f int)
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
insert into t(id) values(1)
--------------

Query OK, 1 row affected (0.00 sec)

--------------
drop table if exists d
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
create table d type='distributed' agent='127.0.0.1:9312:t'
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
select id from d group by a order by a desc, b desc, c desc, d desc, e desc, f desc facet a
--------------

ERROR 2013 (HY000): Lost connection to MySQL server during query
No connection. Trying to reconnect...
ERROR 2003 (HY000): Can't connect to MySQL server on '0:9306' (61)
ERROR:
Can't connect to the server

Notes:

• FACET is important
• group by is important
• the excessive ordering is important