OPTIMIZE fails, then it says no such table (rt index) #2097
Comments
I can't find |
sanikolaev
added
the
waiting
I see 27.8GB: am I missing smth? |
My bad, sorry.
This is from my history (so the last step is with And the contents: So only |
|
Actually, diskchunk save ( You can see after a |
|
This is in searchd.log after a restart And cannot connect with Then when I restart manticore again, this is the searchd.log At this point I can connect, but one of my indexes is "disabled at the JSON config" (I did not do that), and is not serving. (note, i always shut it down with Here is manticore.json It seems like manticore.json is created every time searchd starts. Im not sure if this is the file the log talks about when when it says "disabled at the JSON config", but maybe i need to do something with it, like directory permissions, or set something in the main manticore.conf file. |
|
Hi, is there an update you could give me by any chance? At this point manticore is very unstable: it crashes and even the restart fails (table disabled, and buddy). I don't know if its mlock, preopen_tables... |
|
Hi @Lot-Art Sorry for the delay. Unfortunately, we still don't have You did: which is wrong. Looks like you copied it to just
What's important in this case is the next line, e.g.: then you need to find out why the searchd can't lock
Unlikely in this case. |
|
I did the data with minio again, maybe this time i got it right. Could you please check? |
Still the same :( |
|
Ok, now I really think i did it right. Please check again (sorry for wasting your time with my fiddles). |
|
@Lot-Art I see the files now, thanks. We'll try to reproduce it locally. |
|
As suggested (thank you @sanikolaev ), i increased the ram to 48. Since then, I observed the following things in the log, twice:
|
|
@Lot-Art can you please share your actual configuration file? |
|
@Lot-Art pls also share your full searchd log. |
|
Your query log will be also helpful. |
|
searchd.log (that is be beginning yes, I had to empty it at one point because it got big with those warnings) So OPTIMIZE works in your environment... Ok, so i guess not much can be done with that. How about this... I provide here a script that adds data to the indexes: Maybe the DELETE should happen more often to make Could you please run it for about about ~400.000 docs (add ~400Mb): Then do the following on both indexes:
|
|
Thanks for the script and the other details, @Lot-Art . Unfortunately, I still can't reproduce the But for the warning it's important to have the queries they are linked with. A dummy
This way we can better understand the circumstances of the warning. |
|
Ok, thank you, i will get to that. This thread became long, i will close it and will make a new one when i have all that new info. |
|
Ok, I changed server completely (CentOS, different CPU, and no resource monitoring tools), and it works. If I have to make a guess, then Google Cloud's debian 11 with resource monitor tools creates an environment that bothers/limits the RAM or the SSD. Oh im such a happy camper, after 2 month my life is back to normal :) |
|
@Lot-Art Great! Interesting case. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The problem: OPTIMIZEing an RT index fails, then manticore says the table doesn't exist.
Debian, 4CPU, 32RAM
There is only 1 server, running 1 manticore instance with 2 RT indexes (and this issue is talking about 1 of the indexes only). Only using the mysql interface. There are no nodes, no distribution... its a simple setup.
conf
(Note that
rt_flush_periodis unorthodox but its fine.rt_mem_limitstill applies, it just takes 3 minutes which is ok for now. SoFLUSH RTINDEXandFLUSH RAMCHUNKare fine on both indexes.)create rt index
The table was filled with lots of bulk insert statments (sql). Then i did a clean restart. From searchd.log:
table status
indextool
Trying to OPTIMIZE... (it takes a few minutes to get the "ERROR 2013"). After the fail it says "requires existing table" as if the table didn't exist.
Then this is in searchd.log (the first line below is the same as the last line of the above searchd.log quote, just to show continuity). You can see around the 4 WARNINGs it goes a bit crazy.
After that, if I restart manticore, the tables come back (but the nr of
disk_chunksis still 15).Data with Minio according to your docs
RAM usage is not what i expected
Maybe this is not related, but
htoplooks like this:Questions about that htop: The data files are many GB in size. How come it only use less then 1GB RAM when everything is
mlockand preopened?(The high CPU is probably because i do lots of write and read operations.)
Manticore Search Version:
Manticore 6.2.12 dc5144d@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
Operating System Version:
Debian GNU/Linux 11 (bullseye)
Have you tried the latest development version?
Internal Checklist:
To be completed by the assignee. Check off tasks that have been completed or are not applicable.
The text was updated successfully, but these errors were encountered: