JavaScriptAnalyzer fails to parse large js file and stops indexing of all other repositories

[whizard]

I have a java script file (used for testing - don't ask me why we would have such a file ;) that causes OpenGrok's JavaScriptAnalyzer to fail and after the failure all subsequent repositories are not processed nor index (it causes an Out of Memory error etc.)

I can send you the file that causes the issue.

Thanks,

Paul

[cdgz]

Hi whizard,

It's true that indexer halts once got Out of Memory... Would be great if it could carry on, not sure if there any issue on that.

Maybe you share the famous js file? It's just to reproduce the issue. Suppose dropbox link will be ok,
Thanks,
Chingis

[whizard]

Here are the two files that have issues. They have very long lines ;)

Paul

https://drive.google.com/file/d/0B_soVnwiiVQzT3lFMnlKaTFYSFE/edit?usp=sharing
https://drive.google.com/file/d/0B_soVnwiiVQzVFotQ3B5cDQ0VzA/edit?usp=sharing

[tarzanek]

huh
this is a very nice ctags "bug", which results in opengrok bug ...
I guess disabling local variables for js could help here ... (or fiddling with ctags options so ctag won't create such a huge file for opengrok to chew on)

[tarzanek]

you can try just running ctags on this file ... it will create a HUGE file which links of line to every line number, so this small file will become so big, it just kills the heap of opengrok
I'd suggest ignoring it ... alt. get some fixed ctags that won't create such mess
not sure if we can fix anything on opengrok side ...

[baurmatt]

Hi,

we ran into the exactly same problem today, also the exact same file ;)
@tarzanek you've mentioned a fixed version of ctags, but the project website (http://ctags.sourceforge.net/) shows, that it hasn't been updated since 09 July 2009. Could you please point us in the right direction?

@whizard a workaround would be to add the following line to the file '/usr/local/bin/opengrok':

export IGNORE_PATTERNS="-i big5.js -i gbk.js"

Thanks in advance! :)

Additional infos:
OS: Ubuntu 12.04.4 LTS
exuberant-ctags: 1:5.9~svn20110310-3
opengrok: 0.12

[tarzanek]

@baurmatt - I was under the impression most of distros put patches on top of old ctags
And then I was hoping Darren will pick up ctags, since he returned back to project end of last year ... but not many news since then ... :(
(they said something about github fork, but not sure which one will be picked as official)

[sshock]

I ran into this as well, also on a file called big5.js. OpenGrok choked on it with OutOfMemoryError. I had to remove it from my src folder to get past the problem. Here is a call stack:

2015-09-24 00:56:46.054-0600 SEVERE t30 IndexDatabase$1.run: Problem updating lucene index database:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2367)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:535)
at java.lang.StringBuffer.append(StringBuffer.java:322)
at java.io.BufferedReader.readLine(BufferedReader.java:363)
at java.io.BufferedReader.readLine(BufferedReader.java:382)
at org.opensolaris.opengrok.analysis.Ctags.readTags(Ctags.java:182)
at org.opensolaris.opengrok.analysis.Ctags.doCtags(Ctags.java:173)
at org.opensolaris.opengrok.analysis.plain.PlainAnalyzer.analyze(PlainAnalyzer.java:78)
at org.opensolaris.opengrok.analysis.plain.AbstractSourceCodeAnalyzer.analyze(AbstractSourceCodeAnalyzer.java:69)
at org.opensolaris.opengrok.analysis.AnalyzerGuru.populateDocument(AnalyzerGuru.java:307)
at org.opensolaris.opengrok.index.IndexDatabase.addFile(IndexDatabase.java:606)
at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:870)
at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:835)
at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:835)
at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:835)
at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:835)
at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:835)
at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:835)
at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:835)
at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:835)
at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:835)
at org.opensolaris.opengrok.index.IndexDatabase.update(IndexDatabase.java:383)
at org.opensolaris.opengrok.index.IndexDatabase$1.run(IndexDatabase.java:168)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

[tarzanek]

I also wonder if using universal ctags here on github can mitigate this issue.
Alt filing a bug against them might help, they are active and alive unlike old exuberant ctags.

[santeriv]

universal-ctags commit universal-ctags/ctags@ab9b5e9 seems to refer tackling of lengthy ctag files causing "OOM" and "disk-full" as referred in universal-ctags/ctags#163 (comment)

[tarzanek]

@santeriv this is very good news and it definitely means exuberant ctags is getting irrelevant
I will still add as part of #1014 a limiter on our side to also trim the line at the end (I will try to model it based on @masatake s fix, so we're in sync with universal ctags) and make it kick in for any other ctag than universal ...

[masatake]

I'm very happy to hear that universal-ctags solves the issue of OpenGrok.