Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash on OPTIMIZE INDEX #170

Closed
drewblin opened this Issue Feb 24, 2019 · 17 comments

Comments

Projects
None yet
5 participants
@drewblin
Copy link

drewblin commented Feb 24, 2019

Describe the environment

Manticore Search version: 2.7.5

OS version: FreeBSD 11.2-RELEASE-p4

Describe the problem

Description of the issue:

Crash on OPTIMIZE INDEX. Config file is attached. Index file is too large (61Mb) for github. Tell me where can i put it.

Steps to reproduce:

  1. Run manticore with --logdebug
  2. Call
    optimize index prod_addr_v2

Messsages from log files:

[Fri Feb 22 18:02:37.288 2019] [100290] DEBUG: progressive merge - merging 1 (179801 kb) with 2 (2 kb)
------- FATAL: CRASH DUMP -------
[Fri Feb 22 18:02:37.071 2019] [27876]

--- crashed invalid query ---

--- request dump end ---
Manticore 2.7.5 4a31c54@181204 release
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with 4.2.1
Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DUSE_LIBICONV=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.21 -DDATADIR=/usr/local/var/data -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=ON -DWITH_ICONV=ON -DWITH_MYSQL=1 -DWITH_ZLIB=ON
Host OS is FreeBSD test.instella.com 11.2-RELEASE-p4 FreeBSD 11.2-RELEASE-p4 #0: Thu Sep 27 08:16:24 UTC 2018     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
Stack bottom = 0x7fffdf9f179f, thread stack size = 0x100000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x7fffdf9f02b0)
Stack looks OK, attempting backtrace.
0x407f02
0x0
0x800c8feb2
0x7ffffffff193
0x4ea545
0x4ecd2b
0x73a1f8
0x45e480
0x5e7163
0x800c8ac06
0x0
Something wrong in frame pointers, manual backtrace failed (fp=0)
-------------- backtrace ends here ---------------
--- BT to source lines (depth 0): ---
--- BT to source lines finished ---
--- 2 active threads ---
thd 0, proto sphinxql, state handshake, command SYSTEM
thd 1, proto sphinxql, state handshake, command SYSTEM
------- CRASH DUMP END -------

sphinx.conf.zip

Changelog: fixed crash of daemon on RT optimize with MVA updated; prohibit optimize for RT index with disk chunks there MVA got updated; prohibit merge of plain indexes with MVA updated. That restrictions will be removed along with release of V3 version.

@manticoresearch

This comment has been minimized.

Copy link
Contributor

manticoresearch commented Feb 25, 2019

Thanks for the crash report. Please use our write-only FTP:
ftp: 88.99.251.227
user: manticorebugs
pass: shithappens

and upload into the github-issue-170 folder.

@drewblin

This comment has been minimized.

Copy link
Author

drewblin commented Feb 25, 2019

i've uploaded

@manticoresearch

This comment has been minimized.

Copy link
Contributor

manticoresearch commented Feb 25, 2019

Thanks, I've reproduced the crash

@surf148

This comment has been minimized.

Copy link

surf148 commented Mar 5, 2019

Hi drewblin,
could you rebuild all indexes from scratch with using latest code?
and then recheck "optimize index prod_addr_v2" ?

@drewblin

This comment has been minimized.

Copy link
Author

drewblin commented Mar 6, 2019

Ok, i'll check in 2-3 nearest days.
And some questions:

  1. Should I export code from master branch?
  2. And how is it correct to create fragmented index for this issue? If i'll call some FLUSH RAMCHUNK during sql inserts - is it enough?
  3. Will be possible in stable release to optimize my indexes without rebuild? It'll be difficult to rebuild such indexes in production. I have other indexes with such problem, but can't rebuild them via plain index because of #172.
@tomatolog

This comment has been minimized.

Copy link
Contributor

tomatolog commented Mar 6, 2019

Yes you could grab master HEAD \ 70b2576 or any revisions above.

There was fix recently that corrects use of large strings at index and that might fix how your data got marged and leads to such bad state that breaks OPTIMIZE.

To reproduce issue it might be better to create fragments as usual not just from only couple inserts.

In case #172 blocks this case we could prioritize that case to speed up check of this issue.

@drewblin

This comment has been minimized.

Copy link
Author

drewblin commented Mar 10, 2019

I've build from master, commit bd3e66e.
And call
/usr/local/bin/indexer --config /usr/local/etc/sphinx.conf prod_v2_plain
outputs
sigaction(): Invalid argument

What should i do?

@tomatolog

This comment has been minimized.

Copy link
Contributor

tomatolog commented Mar 13, 2019

could you provide full output of command - there might be BT printed.

Could you provide minimized example that reproduces this issue? As our tests show that indexer works fine.

@tomatolog

This comment has been minimized.

Copy link
Contributor

tomatolog commented Mar 13, 2019

seems crash on optimize due to updated MVA at RT disk chunks and merge operation cannot not handle persistent MVA's.

Could you use REPLACE of whole document instead of MVA UPDATE and check that crashes no more? After you recreates your indexes first as current index with mvp files could not be merged properly.

We still investigating issue and inform you on fix.

@drewblin

This comment has been minimized.

Copy link
Author

drewblin commented Mar 13, 2019

could you provide full output of command - there might be BT printed.

Could you provide minimized example that reproduces this issue? As our tests show that indexer works fine.

root@test:/home/drewblin # sudo -u _sphinx /usr/local/bin/indexer --verbose debugvv --print-queries --config /usr/local/etc/sphinx.conf prod_v2_plain
Manticore 2.8.2 bd3e66e0@190306 dev
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2019, Manticore Software LTD (http://manticoresearch.com)

sigaction(): Invalid argument

I've attached indexer, sphinx.conf and freebsd packet issue-170.zip. Maybe it can help.
Also i deleted all files from /db/sphinx/prod_v2_plain/ (index directory) and tried run again, but it is same result.

And here is more information:

root@test:/home/drewblin # /usr/local/bin/indexer --help
Manticore 2.8.2 bd3e66e0@190306 dev
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2019, Manticore Software LTD (http://manticoresearch.com)

Built by gcc/clang v 4.2.1,

Built on FreeBSD test.instella.com 11.2-RELEASE-p4 FreeBSD 11.2-RELEASE-p4 #0: Thu Sep 27 08:16:24 UTC 2018     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DUSE_LIBICONV=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.21 -DDATADIR=/usr/local/var/data -DFULL_SHARE_DIR=/usr/local/share/manticore -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=ON -DWITH_ICONV=ON -DWITH_MYSQL=1 -DWITH_ZLIB=ON

Usage: indexer [OPTIONS] [indexname1 [indexname2 [...]]]

Ask if i should try something else. I like manticore when it works)

@drewblin

This comment has been minimized.

Copy link
Author

drewblin commented Mar 13, 2019

seems crash on optimize due to updated MVA at RT disk chunks and merge operation cannot not handle persistent MVA's.

Could you use REPLACE of whole document instead of MVA UPDATE and check that crashes no more? After you recreates your indexes first as current index with mvp files could not be merged properly.

We still investigating issue and inform you on fix.

For this index i used only SQL REPLACE. It wasn't any mva update for it.

@tomatolog

This comment has been minimized.

Copy link
Contributor

tomatolog commented Mar 13, 2019

sound strange as RT index at our FTP has two disk chunks with mvp files both and mvp pops up only after mva attributes got updated and index got flushed

@drewblin

This comment has been minimized.

Copy link
Author

drewblin commented Mar 14, 2019

O, sorry... I found updates of this index in my code. I was inattentive.
Ok, i'll try not to use update.

@drewblin

This comment has been minimized.

Copy link
Author

drewblin commented Mar 14, 2019

seems crash on optimize due to updated MVA at RT disk chunks and merge operation cannot not handle persistent MVA's.

Could you use REPLACE of whole document instead of MVA UPDATE and check that crashes no more? After you recreates your indexes first as current index with mvp files could not be merged properly.

We still investigating issue and inform you on fix.

I made 2 test.

First one:

  1. Fill index with replace.
  2. Make replace for each record for some times
  3. Call flush ramchunk
  4. Make replace for each record for some times
  5. Call flush ramchunk
  6. Call OPTIMIZE INDEX
    There were no problems, both chunks were merged.

Second one:

  1. Fill index with replace.
  2. Make update for each record
  3. Call flush ramchunk. At this point no mvp file was created. So i restarted manticore and file appeared.
  4. Make replace for each record
  5. Make update for each record
  6. Call flush ramchunk, restart manticore. There were 2 mvp files - one for each chunk.
  7. Call OPTIMIZE INDEX
    Manticore crashed.

As i understand your assumption is right.

Crash info:

[Thu Mar 14 10:46:27.803 2019] [100497] DEBUG: progressive merge - merging 0 (223361 kb) with 1 (235066 kb)
------- FATAL: CRASH DUMP -------
[Thu Mar 14 10:46:27.583 2019] [79943]

--- crashed invalid query ---

--- request dump end ---
Manticore 2.8.2 bd3e66e0@190306 dev
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with 4.2.1
Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DUSE_LIBICONV=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.21 -DDATADIR=/usr/local/var/data -DFULL_SHARE_DIR=/usr/local/share/manticore -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=ON -DWITH_ICONV=ON -DWITH_MYSQL=1 -DWITH_ZLIB=ON
Host OS is FreeBSD test.instella.com 11.2-RELEASE-p4 FreeBSD 11.2-RELEASE-p4 #0: Thu Sep 27 08:16:24 UTC 2018     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
Stack bottom = 0x7fffdf9f179f, thread stack size = 0x100000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x7fffdf9f0280)
Stack looks OK, attempting backtrace.
0x4080a2
0x0
0x800cc1eb2
0x7ffffffff193
0x4f5aaa
0x4f807b
0x745c11
0x4583d0
0x5f3823
0x800cbcc06
0x0
Something wrong in frame pointers, manual backtrace failed (fp=0)
-------------- backtrace ends here ---------------
[Thu Mar 14 10:46:27.897 2019] [100187] watchdog: got USR1, performing dump of child's stack
--- 2 active threads ---
thd 0, proto sphinxql, state handshake, command SYSTEM
thd 1, proto sphinxql, state handshake, command SYSTEM
------- CRASH DUMP END -------
@surf148

This comment has been minimized.

Copy link

surf148 commented Mar 14, 2019

Hi drewblin,
A new version (3.x) of manticore has implemented feature which fix MVA UPDATE and don't have a crash on OPTIMIZE INDEX. We expect that new version will be released in near future.
Could you please to wait it and recheck again with new version?
Thank you.

@drewblin

This comment has been minimized.

Copy link
Author

drewblin commented Mar 14, 2019

Ok, i'll wait for nearest future)

@tomatolog tomatolog self-assigned this Mar 19, 2019

@tomatolog tomatolog added the bug label Mar 19, 2019

@tomatolog

This comment has been minimized.

Copy link
Contributor

tomatolog commented Mar 19, 2019

I've just added check at 57932ae that skips optimize for RT index that has updated MVA or disk chunks with updated MVA. And added same check for regular index merge.

Now you might use your regular schedule and have no crashes. However you could not optimize or merge such indexes or disk chunks of RT index. But that will be fixed at when we release next index version soon.

@githubmanticore githubmanticore removed the bug label Mar 19, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.