Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash after upgrading from 6.0.4 to 6.2.0 #1364

Closed
sanikolaev opened this issue Aug 17, 2023 · 5 comments
Closed

Crash after upgrading from 6.0.4 to 6.2.0 #1364

sanikolaev opened this issue Aug 17, 2023 · 5 comments
Assignees
@tomatolog
Labels
bug rel::6.2.12

Comments

@sanikolaev
Copy link
Collaborator

If you do this (i.e. upgrade an empty Manticore instance from 6.0.4 to 6.2.0):

apt -y update
apt -y install libcurl4
wget https://repo.manticoresearch.com/repository/manticoresearch_bookworm/dists/bookworm/main/binary-amd64/manticore-server-core_6.0.4-230314-1a3a4ea82_amd64.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_bookworm/dists/bookworm/main/binary-amd64/manticore-backup_0.5.2-23020607-4a37932_all.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_bookworm/dists/bookworm/main/binary-amd64/manticore-buddy_0.4.2-23031500-36757ee_all.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_bookworm/dists/bookworm/main/binary-amd64/manticore-columnar-lib_2.0.4-230306-5a49bd7_amd64.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_bookworm/dists/bookworm/main/binary-amd64/manticore-common_6.0.4-230314-1a3a4ea82_all.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_bookworm/dists/bookworm/main/binary-amd64/manticore-dev_6.0.4-230314-1a3a4ea82_all.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_bookworm/dists/bookworm/main/binary-amd64/manticore-extra_0.6.2-23012604-d95e43e_all.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_bookworm/dists/bookworm/main/binary-amd64/manticore-server-core-dbgsym_6.0.4-230314-1a3a4ea82_amd64.ddeb
wget https://repo.manticoresearch.com/repository/manticoresearch_bookworm/dists/bookworm/main/binary-amd64/manticore-server-core_6.0.4-230314-1a3a4ea82_amd64.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_bookworm/dists/bookworm/main/binary-amd64/manticore-server_6.0.4-230314-1a3a4ea82_amd64.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_bookworm/dists/bookworm/main/binary-amd64/manticore-tools_6.0.4-230314-1a3a4ea82_amd64.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_bookworm/dists/bookworm/main/binary-amd64/manticore_6.0.4-230314-1a3a4ea82_amd64.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_bookworm/dists/bookworm/main/binary-amd64/manticore-executor_0.6.2-23012605-d95e43e_amd64.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_bookworm/dists/bookworm/main/binary-amd64/manticore-icudata-65l.deb
dpkg -i *.deb
systemctl start manticore
mysql -P9306 -h0 -e "show status like 'version'"
wget https://repo.manticoresearch.com/manticore-repo.noarch.deb
sudo dpkg -i manticore-repo.noarch.deb
sudo apt update
sudo apt upgrade
mysql -P9306 -h0 -e "show status like 'version'"
systemctl restart manticore

in Debian Bookworm you'll get the following in the searchd log:

[Thu Aug 17 01:10:33.936 2023] [509] starting daemon version '6.0.4 1a3a4ea82@230314 (columnar 2.0.4 5a49bd7@230306) (secondary 2.0.4 5a49bd7@230306)' ...
[Thu Aug 17 01:10:33.936 2023] [509] listening on 127.0.0.1:9312 for sphinx and http(s)
[Thu Aug 17 01:10:33.936 2023] [509] listening on 127.0.0.1:9306 for mysql
[Thu Aug 17 01:10:33.936 2023] [509] listening on 127.0.0.1:9308 for sphinx and http(s)
[Thu Aug 17 01:10:33.943 2023] [513] prereading 0 tables
[Thu Aug 17 01:10:33.943 2023] [513] preread 0 tables in 0.000 sec
[Thu Aug 17 01:10:33.948 2023] [509] accepting connections
[Thu Aug 17 01:10:34.188 2023] [533] [BUDDY] started '/usr/share/manticore/modules/manticore-buddy --listen=http://127.0.0.1:9312  --threads=32' at http://127.0.0.1:34591
[Thu Aug 17 01:12:56.366 2023] [509] caught SIGTERM, shutting down
[Thu Aug 17 01:12:56.441 2023] [509] shutdown daemon version '6.0.4 1a3a4ea82@230314 (columnar 2.0.4 5a49bd7@230306) (secondary 2.0.4 5a49bd7@230306)' ...
[Thu Aug 17 01:12:56.441 2023] [509] shutdown complete
[Thu Aug 17 01:12:56.443 2023] [508] watchdog: main process 509 exited cleanly (exit code 0), shutting down
[Thu Aug 17 01:12:56.458 2023] [2134] watchdog: main process 2135 forked ok
[Thu Aug 17 01:12:56.459 2023] [2135] starting daemon version '6.2.0 45680f95d@230804 (columnar 2.2.0 dc33868@230804) (secondary 2.2.0 dc33868@230804)' ...
[Thu Aug 17 01:12:56.459 2023] [2135] listening on 127.0.0.1:9312 for sphinx and http(s)
[Thu Aug 17 01:12:56.459 2023] [2135] listening on 127.0.0.1:9306 for mysql
[Thu Aug 17 01:12:56.459 2023] [2135] listening on 127.0.0.1:9308 for sphinx and http(s)
[Thu Aug 17 01:12:56.473 2023] [2135] FATAL: binlog meta file /var/lib/manticore/binlog/binlog.meta is v.13, binary is v.14; recovery requires previous binary version
[Thu Aug 17 01:12:56.474 2023] [2135] WARNING: GlobalCrashQueryGetRef: thread-local info is not set! Use ad-hoc

--- crashed invalid query ---

--- request dump end ---
--- local index:
Manticore 6.2.0 45680f95d@230804 (columnar 2.2.0 dc33868@230804) (secondary 2.2.0 dc33868@230804)
Handling signal 6
-------------- backtrace begins here ---------------
Program compiled with Clang 15.0.7
Configured with flags: Configured with these definitions: -DDISTR_BUILD=bookworm -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_
NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1
 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmariadb.so.3 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOS
TGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux x86_64 (bookworm) (cross-compiled)
Stack bottom = 0x7ffe12584417, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x1)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x1, stack=0x55c102940000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x22a)[0x55c1004ff96a]
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi+0x355)[0x55c10037e465]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bfd0)[0x7fed99586fd0]
/lib/x86_64-linux-gnu/libc.so.6(+0x8ad3c)[0x7fed995d5d3c]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12)[0x7fed99586f32]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7fed99571472]
/lib/x86_64-linux-gnu/libc.so.6(+0x7f340)[0x7fed995ca340]
/lib/x86_64-linux-gnu/libc.so.6(+0x946ba)[0x7fed995df6ba]
/lib/x86_64-linux-gnu/libc.so.6(+0x94f64)[0x7fed995dff64]
/lib/x86_64-linux-gnu/libc.so.6(+0x965eb)[0x7fed995e15eb]
/lib/x86_64-linux-gnu/libc.so.6(__libc_free+0x6f)[0x7fed995e3d9f]
/usr/bin/searchd(_ZN3sph8Vector_TI14CharsetAlias_tNS_13DefaultCopy_TIS1_EENS_14DefaultRelimitENS_16DefaultStorage_TIS1_EEED2Ev+0x45)[0x55c101560ce5]
/lib/x86_64-linux-gnu/libc.so.6(+0x3e4dd)[0x7fed995894dd]
/lib/x86_64-linux-gnu/libc.so.6(+0x3e61a)[0x7fed9958961a]
/usr/bin/searchd(+0xedd772)[0x55c100401772]
/usr/bin/searchd(_ZN8Binlog_c8LoadMetaEv+0x200)[0x55c101531700]
/usr/bin/searchd(_ZN8Binlog_c9ConfigureERK17CSphConfigSectionbjb+0xfa)[0x55c1015314da]
/usr/bin/searchd(_Z11ServiceMainiPPc+0x2221)[0x55c1003db871]
/usr/bin/searchd(main+0x4b)[0x55c1003dd27b]
/lib/x86_64-linux-gnu/libc.so.6(+0x271ca)[0x7fed995721ca]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7fed99572285]
/usr/bin/searchd(_start+0x21)[0x55c1002a4981]
Trying boost backtrace:

Same with systemctl stop / start instead of restart.

The task is to fix the crash. There's another issue about why the binlog files are left after a clean shutdown.
@sanikolaev sanikolaev added the bug label Aug 17, 2023
@sanikolaev sanikolaev mentioned this issue Aug 17, 2023
Closed
@sanikolaev
Copy link
Collaborator Author

As discussed on today's dev call, it might make sense to not generate a crash log in the case of "FATAL: binlog meta file /var/lib/manticore/binlog/binlog.meta is v.13, binary is v.14; recovery requires previous binary version" and perhaps some other cases when the FATAL error message is sufficient to understand the context.

@tomatolog
Copy link
Contributor

the FATAL message on reading empty binlog file with the old format fixed at 828d139

And daemon now could start with the binlog of old format if the binlog empty or has only empty ADD_CACHE operation.

@tomatolog
Copy link
Contributor

However there are still crash or deadlocks after FATAL message of the wrong binlog. These are not related to the binlog itself but could happen at the any stage of daemon startup from any Die or FATAL message and should be fixed by investigate and cleanup startup \ shutdown sequence of daemon

@tomatolog
Copy link
Contributor

created manticoresoftware/dev/issues/557 for general investigation and refactor of Die or FATAL daemon termination

@sanikolaev sanikolaev assigned tomatolog and unassigned klirichek Aug 18, 2023
@githubmanticore githubmanticore mentioned this issue Aug 18, 2023
Open
@sanikolaev
Copy link
Collaborator Author

Test of the latest dev package on Debian Bookworm - all OK.

root@4ec9ef08b9de:~# mysql -P9306 -h0 -e "show status like 'version'"
+---------+-----------------------------------------------------------------------------------------+
| Counter | Value                                                                                   |
+---------+-----------------------------------------------------------------------------------------+
| version | 6.0.4 1a3a4ea82@230314 (columnar 2.0.4 5a49bd7@230306) (secondary 2.0.4 5a49bd7@230306) |
+---------+-----------------------------------------------------------------------------------------+

root@4ec9ef08b9de:~# wget https://repo.manticoresearch.com/manticore-dev-repo.noarch.deb && \
> sudo dpkg -i manticore-dev-repo.noarch.deb && \
> sudo apt -y update
--2023-08-21 07:27:37--  https://repo.manticoresearch.com/manticore-dev-repo.noarch.deb
Resolving repo.manticoresearch.com (repo.manticoresearch.com)... 49.12.119.254
Connecting to repo.manticoresearch.com (repo.manticoresearch.com)|49.12.119.254|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2620 (2.6K) [application/octet-stream]
Saving to: 'manticore-dev-repo.noarch.deb'

manticore-dev-repo.noarch.deb                       100%[=================================================================================================================>]   2.56K  --.-KB/s    in 0s

2023-08-21 07:27:37 (100 MB/s) - 'manticore-dev-repo.noarch.deb' saved [2620/2620]

Selecting previously unselected package manticore-repo.
(Reading database ... 13102 files and directories currently installed.)
Preparing to unpack manticore-dev-repo.noarch.deb ...
Unpacking manticore-repo (0.0.4) ...
Setting up manticore-repo (0.0.4) ...
gpg: key 14D612DB3D2730E2: "Manticore Search <support@manticoresearch.com>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Get:1 http://repo.manticoresearch.com/repository/manticoresearch_bookworm_dev bookworm InRelease [2880 kB]
Hit:2 http://deb.debian.org/debian bookworm InRelease
Hit:3 http://deb.debian.org/debian bookworm-updates InRelease
Hit:4 http://deb.debian.org/debian-security bookworm-security InRelease
Get:5 http://repo.manticoresearch.com/repository/manticoresearch_bookworm_dev bookworm/main amd64 Packages [5387 kB]
Fetched 8267 kB in 1s (13.7 MB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
44 packages can be upgraded. Run 'apt list --upgradable' to see them.

root@4ec9ef08b9de:~# apt upgrade
...

root@4ec9ef08b9de:~# searchd -v
Manticore 6.2.1 b220b6bcc@230820 dev (columnar 2.2.1 c6dbbcb@230820) (secondary 2.2.1 c6dbbcb@230820)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)

root@4ec9ef08b9de:~# mysql -P9306 -h0 -e "show status like 'version'"
+---------+-----------------------------------------------------------------------------------------+
| Counter | Value                                                                                   |
+---------+-----------------------------------------------------------------------------------------+
| version | 6.0.4 1a3a4ea82@230314 (columnar 2.0.4 5a49bd7@230306) (secondary 2.0.4 5a49bd7@230306) |
+---------+-----------------------------------------------------------------------------------------+

root@4ec9ef08b9de:~# systemctl restart manticore

root@4ec9ef08b9de:~# mysql -P9306 -h0 -e "show status like 'version'"
+---------+---------------------------------------------------------------------------------------------+
| Counter | Value                                                                                       |
+---------+---------------------------------------------------------------------------------------------+
| version | 6.2.1 b220b6bcc@230820 dev (columnar 2.2.1 c6dbbcb@230820) (secondary 2.2.1 c6dbbcb@230820) |
+---------+---------------------------------------------------------------------------------------------+

@chenrui333 chenrui333 mentioned this issue Aug 29, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tomatolog tomatolog

Labels
bug rel::6.2.12
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tomatolog @klirichek @sanikolaev