Moss panic with runaway disk usage #61
Comments
|
hi @connorgorman -- sorry to hear that. From the stack dump, can't tell much, other than moss is trying to merge segments together (e.g., compact files together to hopefully make them smaller). So, there's no "a-ha" that comes to mind, but some other thoughts -- were there perhaps multiple crashes over time, or just one? Would you happen to see whether moss is generating tons of files, or instead just a few (or even one file) is growing really large? Wondering about the application usage pattern? What's the app doing with moss? One case, for example, is if there are lots of so-called "slow-reader" goroutines, then they can be holding onto reference-counts on multiple moss "segments", which will keep the files around. An edge case of a "infinitely slow reader", for example, is if the application forgets to close a snapshot, so that also pops to mind. |
|
Thanks for the reply @steveyen I'm currently accessing moss + lower level store through Bleve. The app is basically writing/deleting batches of documents to Bleve periodically and then has some adhoc documents that flow in so it's a mix of both batch and individual indexing operations. Perhaps the query in the stacktrace below is particularly slow and could bog down the indexing? I'm currently running a cluster so I can let you know about moss generating tons of files vs a large file. It panics pretty frequently. Here are some more relevant stack traces. An interesting note is that the index isn't compacting, but when I run |
Perhaps -- can't tell from the just the stacktrace alone. Like if you're doing some wildcard searches that somehow return everything, or the regexp is for a huge # of terms, and the dataset (the everything) is large, then that might be a situation. btw, that goroutine 3232418 is in the early phases of search reques processingt (going through prep work & setup & constructors of various data structures), as opposed to being in the middle of actual iteration or Next()'ing through candidate hits, so that might mean the "slow searcher theory" isn't quite right.
Oh, that's interesting. Then that sounds like it might not be impossible to reach an easy-to-reliably-reproduce simple situation. Do step 1, 2, 3 then bam!
That's really weird! And, that's a bug, in "mossScope stats fragmentation", where it's supposed to open the file in read-only mode. Created a ticket in JIRA (what the full-text team happens to use at couchbase for day to day work) to track that -- https://issues.couchbase.com/browse/MB-31816 Maybe that's the issue, if you happen to be running mossScope regularly with "stats fragmentation", perhaps leading to some kind of concurrent file access issue. Not sure, but it's a legit mossScope bug.
Here's the way we did that in the couchbase cbft (full text search) project, where we just had a local interface with the Actual() method on it... https://github.com/couchbase/cbft/blob/master/pindex_bleve_moss_rollback.go#L41 |
|
@steveyen Thanks again!
Looks like there are lots of files generated. If I understand moss correctly (especially from the OpenStore code), that only the last written file is important if there are no other searchers referencing the other files (which is why OpenStore can prune the index). If this is the case, then the ref count stuff mentioned above makes sense. I'll take a look into that
We don't run moss scope frequently. It was mostly just to debug why things weren't compacting |
Hey everyone, I'm seeing the following panic which looks like it's run out of memory. However, I've attached a graph of Grafana which shows the disk usage by the Moss index (its from a file walker so it could have a race condition if Moss is generating lots of files while it is walking)
The disk usage grows for about 6 minutes then implodes when I assume the disk is completely filled. The green line after the blip is our service restarting and our indexes being rebuilt
The text was updated successfully, but these errors were encountered: