-
-
Notifications
You must be signed in to change notification settings - Fork 488
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Manticore crashes on frequent index updates #1458
Comments
I was not able to recreate this issue. After 15-20 minutes your sender crashes, manticore continues to work, without crash. Logs:
|
Very strange, manticore dropped the connection inside docker for some reason. Can you try again? I tried locally and let another developer test the script. We were able to replicate the crash |
Issue is being played on our server. Steps for reproduce.
Logs:
Comment:I can note the behaviour, crash is only reproduced after repeatedly running |
I try downgrade to 6.0.4 + rowwide / columnar engine - crashes everywhere. |
As discussed in Telegram, I can't reproduce this issue in the dev version, so most likely this bug has been already fixed. I'm closing this issue. Feel free to reopen if you can reproduce it in the dev version. |
In the dev version there's another problem with secondary indexes - manticoresoftware/columnar#36 |
Reopening. It seems that on the newer version the crash occurs less frequently, but still the same crash can happen:
|
It's also very unstable: sometimes the provided script works fine for the whole night, sometimes it crashes in a minute after started. |
A couple crash reports from a debug version of Manticore:
```
[Tue Oct 10 11:38:17.278 2023] [25] rt: table posts_idx: diskchunk 1179(168), segments 6 saved in 69.903959 (70.196098) sec, RAM saved/new 82581914/55061971 ratio 0.599968 (soft limit 80526329, conf limit 134217728)
------- FATAL: CRASH DUMP -------
[Tue Oct 10 11:38:34.078 2023] [ 1]
--- crashed HTTP request dump ---
�g �u_�����S� -------------- backtrace ends here --------------- --- crashed HTTP request dump ---
�g �u_�����S� -------------- backtrace ends here ---------------
|
Finally I could reproduce it more or less stably without docker. Here's how to do it on dev2: Take the etalon data dir (13G) on which it crashes stable:
(most likely you don't have to do it every time, but if it stops crashing again, you know where the etalon data dir is located) Start Manticore (latest dev):
Run the REPLACE load: 50 concurrent replaces, then a 30 second pause:
At the same time run the SELECT load: always 50 concurrent selects:
Wait a couple minutes until it crashes:
Inspect the log:
|
6 years since the fork and it is a buggy mess still. Same problem here. Was thinking to migrate from sphinx. |
Inserted by 1000 rows in a transaction. Result:
|
@glookka is there any workaround? |
@starinacool please try the latest dev version. As said in #1458 (comment) , some issues have been already fixed there. |
Oops. I've confused this issue with another one. But it still makes sense to make sure the issue can be reproduced in the latest dev version. |
@sanikolaev , made a separate issue. May be it is something different: #1602 |
Fixed in d67dbe6 |
I confirm I can't reproduce the issue anymore. |
To Reproduce
Steps to reproduce the behavior:
You can run script in repo, where i reproduce this logic:
https://github.com/Eclipsium/manticore_crash
docker compose up
Expected behavior
stable operation, or a warning of what we are doing wrong
Describe the environment:
Messages from log files:
link to repo
The text was updated successfully, but these errors were encountered: