Autocomplete in 1.7.30 #3928
Replies: 8 comments 19 replies
-
|
I found suggester DBs for every project under the data directory, as well as the thesis, but it didn't help much in getting it to work :( |
Beta Was this translation helpful? Give feedback.
-
|
I'm totally lost with this. Apparently it works for some projects, but for others it doesn't. And for those projects that it does NOT work I see only WFST files in the suggester's project directory. I tried removing one of bad project directories and restarting Tomcat. It just regenerated WFST files (no version.txt, no search_count.db). I was adding some debug statements to see what's going on, and at some point after removing bad suggester project directory and restarting Tomcat it actually got regenerated correctly, with all the files. But subsequent attempt yielded the same problem with WFST only. |
Beta Was this translation helpful? Give feedback.
-
|
I think I see two Suggesters initialized and running in parallel from two threads once Tomcat is restarted: You can see how the two threads are generating about the same WFST files for the same PROJECT at the same time. I wonder if this shouldn't happen? |
Beta Was this translation helpful? Give feedback.
-
|
I wonder what triggers these 2 suggester inits for the same project. Could you try capturing this with |
Beta Was this translation helpful? Give feedback.
-
|
At which point does this happen ? During reindex ? |
Beta Was this translation helpful? Give feedback.
-
|
Actually, |
Beta Was this translation helpful? Give feedback.
-
|
I wonder where do This will eventually end up in Now, the problem with simultaneous init of suggester data for the same project could happen if Now, |
Beta Was this translation helpful? Give feedback.
-
|
Bumping the log level for the web application (to at least |
Beta Was this translation helpful? Give feedback.
-
|
Here are the stack traces. I don't do any API calls or run indexing in parallel. Just remove /opengrok/data/suggester/PROJECT and restart tomcat. |
Beta Was this translation helpful? Give feedback.
-
|
So this happens during web app startup ? |
Beta Was this translation helpful? Give feedback.
-
|
As for the stack traces, I mean something like https://stackoverflow.com/questions/1069066/how-can-i-get-the-current-stack-trace-in-java , in particular dumped in |
Beta Was this translation helpful? Give feedback.
-
|
Something like: diff --git a/opengrok-web/src/main/java/org/opengrok/web/api/v1/suggester/provider/service/impl/SuggesterServiceImpl.java b/opengrok-web/src/main/java/org/opengrok/web/api/v1/suggester/provider/service/impl/SuggesterServiceImpl.java
index e95115c67..1268fe577 100644
--- a/opengrok-web/src/main/java/org/opengrok/web/api/v1/suggester/provider/service/impl/SuggesterServiceImpl.java
+++ b/opengrok-web/src/main/java/org/opengrok/web/api/v1/suggester/provider/service/impl/SuggesterServiceImpl.java
@@ -309,6 +309,9 @@ public class SuggesterServiceImpl implements SuggesterService {
if (rebuildParalleismLevel == 0) {
rebuildParalleismLevel = 1;
}
+ logger.log(Level.WARNING, "dumping stack of thread {0} (id {1})",
+ new Object[]{Thread.currentThread(), Thread.currentThread().getId()});
+ Thread.dumpStack();
logger.log(Level.FINER, "Suggester rebuild parallelism level: " + rebuildParalleismLevel);
suggester = new Suggester(suggesterDir,
suggesterConfig.getMaxResults(), |
Beta Was this translation helpful? Give feedback.
-
|
I'm sorry the two threads running happened to be a misconfiguration. I had webapps/source-1.7.30 and then a webapps/source symlink, so Tomcat was starting two copies of the app. But after removing symlink it still cannot proceed past WFST init. It appears to be stuck in build(indexReader, "full") not taking any CPU. Added an Exception handler in SuggesterProjectData::build(). Nothing is getting thrown. |
Beta Was this translation helpful? Give feedback.
-
|
Two webapps with the same configuration was my hypothesis, nice that it is confirmed. |
Beta Was this translation helpful? Give feedback.
-
|
The above log snippet looks normal - the suggester was successfully initialized and the next rebuild scheduled. |
Beta Was this translation helpful? Give feedback.
-
|
It's never coming out of WSTCompletionLookup::build() for field="full" |
Beta Was this translation helpful? Give feedback.
-
|
Try to get a snapshot of thread stacks in the webapp using |
Beta Was this translation helpful? Give feedback.
-
|
Also, how big is the index for the project in question ? |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, if I exclude "full" from the suggester config, it all completes successfully. Once it gets into build() for "full", java takes about 100% CPU and the stack varies. E.g.: or Then I see this message in catalina.out: At this point the thread's dead and CPU inactive. It's worth mentioning again that control never leaves WFSTCompletionLookup::build(). IDK, can java throw something other than Exception? Can the thread just be killed? And the index is 3GB in total with individual files going up to 1.3GB. |
Beta Was this translation helpful? Give feedback.
-
|
How do you know the thread is still around when the |
Beta Was this translation helpful? Give feedback.
-
|
I didn't say that. After "Suggester successfully initialized" message the thread is no longer there in jstack output. So it's dead. |
Beta Was this translation helpful? Give feedback.
-
|
Right. If it was not gone, the opengrok/suggester/src/main/java/org/opengrok/suggest/Suggester.java Lines 189 to 190 in 7626bae waitTerminationTime has passed (half an hour by default).
What's the problem then ? |
Beta Was this translation helpful? Give feedback.
-
|
Woo-hoo. It was -Xmx1024 in Tomcat startup options. |
Beta Was this translation helpful? Give feedback.
-
|
I should have asked about that. See https://github.com/oracle/opengrok/wiki/Tuning-for-large-code-bases#web-application |
Beta Was this translation helpful? Give feedback.
-
|
So, to summarize, this ended up being a Tomcat misconfiguration. Suggester's index generator thread was silently killed due to OOM while generating WFST file for "full", so the /opengrok/data/suggester/PROJECT directory was left with partially generated files. And the solution was, as the Wiki says, pass a -Xmx8192M option to Tomcat during startup. Autocompletions do work now, and work really well. I think, they alone were well worth the upgrade. Thanks for your help! |
Beta Was this translation helpful? Give feedback.
-
|
As the tuning wiki says, the amount of Java heap should be computed (and recomputed as the instance changes, ideally based on heap monitoring) based on the indexed data. |
Beta Was this translation helpful? Give feedback.
-
|
The silent OOM I am not sure about - it might be Lucene (or something else in OpenGrok although I don't see the evidence for it in the code) actually swallowed the OutOfMemory Throwable, otherwise it should bubble all the way up I think, so it would be visible in Tomcat logs. Also, the tuning wiki has a Java option for the out-of-memory to dump the heap to disk in such event. For the production instance I maintain internally, I have Grafana alerts set up for JVM heap thresholds using stuff from https://github.com/OpenGrok/opengrok-monitoring-docker/tree/master/grafana/provisioning |
Beta Was this translation helpful? Give feedback.
-
Hi,
Does autocomplete of the main form search fields require some special configuration?
I see a wheel briefly spinning when I type, I even see requests like below in the logs, but no results show up.
GET /source/api/v1/suggest?projects%5B%5D=PROJECT&field=path&full=&defs=&refs=&path=boost&hist=&type=&caret=3 HTTP/1.1" 200 95
There are definitely matches though (plenty of files starting with "boost"), and if I copy/paste the above URL in the browser, I get back this:
{"time":0,"suggestions":[],"identifier":"boostlqsbw","queryText":"boostlqsbw","partialResult":true}
If I search for a variable definition with a name starting with "ccc", the query and result look like this:
/source/api/v1/suggest?projects%5B%5D=PROJECT&field=defs&full=&defs=ccc&refs=&path=&hist=&type=&caret=3
{"time":0,"suggestions":[],"identifier":"cccmqtze","queryText":"cccmqtze","partialResult":true}
I'm not sure if the 5-letter junk at the end of queryText and identifier is there on purpose or not. Every time I resubmit GET, I get back different 5-letter combinations.
Thanks!
Beta Was this translation helpful? Give feedback.
All reactions