Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StackOverflowError on each startup, when indexing Javascript #3669

Closed
mossroy opened this issue Feb 24, 2022 · 11 comments · Fixed by #3696
Closed

StackOverflowError on each startup, when indexing Javascript #3669

mossroy opened this issue Feb 24, 2022 · 11 comments · Fixed by #3696
Labels
kind:bug Bug report or fix

Comments

@mossroy
Copy link

mossroy commented Feb 24, 2022

Apache NetBeans version

Apache NetBeans 12.6

What happened

After upgrading NetBeans from 12.0 to 12.6 (and importing the settings from 12.0, as suggested by the UI), I have a StackOverflow error on each startup.

During the "background scanning of projects", an exception is thrown, and displayed by NetBeans with an infinite loop on org.netbeans.modules.javascript2.editor.index.JsIndexer.storeObject(JsIndexer.java:229)
(see stacktrace)

The background scanning of projects never ends, and consumes a lot of CPU in the background

How to reproduce

On my computer, it can be reproduced on each startup.

As a workaround, I could switch to another project group (with other non-javascript projects), then restart Netbeans.
In this case, there is no exception and no background CPU usage on startup. But the symptom comes back if I switch back to the project group where I have a specific javascript project.

This project is an open-source one: https://github.com/kiwix/kiwix-js
But this project is also open in a project group of another computer with no problem (same OS, same version of Netbeans, also upgraded from 12.0).

Did this work correctly in an earlier version?

Apache NetBeans 12.3 or earlier

Operating System

Ubuntu 20.04.3 amd64, with all updates installed

JDK

OpenJDK 11.0.13 (provided by Ubuntu). Same result with OpenJDK 17.0.1 (provided by Ubuntu)

Apache NetBeans packaging

Apache NetBeans provided installer

Anything else

Stacktrace:

java.lang.StackOverflowError
    at java.base/java.util.WeakHashMap.getTable(WeakHashMap.java:350)
    at java.base/java.util.WeakHashMap.get(WeakHashMap.java:398)
    at org.openide.util.lookup.MetaInfServicesLookup.beforeLookup(MetaInfServicesLookup.java:121)
    at org.openide.util.lookup.AbstractLookup.lookupItem(AbstractLookup.java:402)
    at org.openide.util.lookup.AbstractLookup.lookup(AbstractLookup.java:396)
    at org.openide.util.lookup.ProxyLookup.lookup(ProxyLookup.java:281)
    at org.netbeans.modules.javascript2.model.JsElementImpl.isInternalFile(JsElementImpl.java:194)
    at org.netbeans.modules.javascript2.model.JsElementImpl.isPlatform(JsElementImpl.java:188)
    at org.netbeans.modules.javascript2.model.api.IndexedElement$Flag.getFlag(IndexedElement.java:387)
    at org.netbeans.modules.javascript2.editor.index.JsIndexer.createDocument(JsIndexer.java:121)
    at org.netbeans.modules.javascript2.editor.index.JsIndexer.storeObject(JsIndexer.java:221)
    at org.netbeans.modules.javascript2.editor.index.JsIndexer.storeObject(JsIndexer.java:229)
    at org.netbeans.modules.javascript2.editor.index.JsIndexer.storeObject(JsIndexer.java:229)
    at org.netbeans.modules.javascript2.editor.index.JsIndexer.storeObject(JsIndexer.java:229)

(this last line is repeated around 1000 times)

Removing the cache and restarting Netbeans does not help.
I'd be happy to help to test/investigate on that issue.

(originally reported on https://issues.apache.org/jira/browse/NETBEANS-6467)

Are you willing to submit a pull request?

No response

Code of Conduct

Yes

@mossroy mossroy added kind:bug Bug report or fix needs:triage Requires attention from one of the committers labels Feb 24, 2022
@Chris2011
Copy link
Contributor

Someone said that the original indexer for JS was not that good enough to handle the amount of node_modules etc. It was not made for this. I think this is the problem now. If you just have a plain HTML, CSS, JS project, there will be no problem, but with npm, houston we have a problem.

@mossroy
Copy link
Author

mossroy commented Feb 24, 2022

Many thanks for this feedback!
Removing node_modules directory is indeed a workaround. So I can confirm it is the cause.

But if I remove this directory, NetBeans complains it's a "Project problem" that kiwix-js uses npm modules that are not installed. And if I let it install them, the problem is back.
In fact, this project only uses npm to run some unit tests for now: is there a way to exclude node_modules directory from the JS indexer?

@Chris2011
Copy link
Contributor

Yes you can add the folder node_modules to the options -> misc. -> files -> Files ignored by the IDE. But be aware that you will lost code completion for your node modules because they will not being indexed.

@Chris2011
Copy link
Contributor

But in general this is still a topic to do to fix/rewrite the indexer.

@mossroy
Copy link
Author

mossroy commented Feb 25, 2022

Appending |^node_modules$ to the setting mentioned above by @Chris2011 is a good workaround in my use case.
Note that the directory is completely ignored: it is not even visible in the projects or files explorer.

Many thanks @Chris2011 for the hints.
I agree with you that this infinite loop is still a bug that needs to be fixed

@Chris2011
Copy link
Contributor

Yes, it will be ignored completely and your code completion will not work for your npm modules, due to the missing index. Yes it will not be shown in the project but there is a PR for this, which was merged december 2021: #3209. This will allow to still have the folder to navigate through but w/o indexing.

@matthiasblaesing
Copy link
Contributor

matthiasblaesing commented Feb 27, 2022

Putting the discussion about the indexer in general aside the issue here is that the indexer does not handle self referential structures well. A minimal reproducer is this:

function mod(actions) {
    return actions.map((action) => {
        action = { action };
        return action;
    });
};

With that at least the occurrences finder:

java.lang.StackOverflowError
	at org.netbeans.modules.javascript2.editor.navigation.OccurrencesSupport.findOccurrence(OccurrencesSupport.java:59)
	at org.netbeans.modules.javascript2.editor.navigation.OccurrencesSupport.findOccurrence(OccurrencesSupport.java:80)
	at org.netbeans.modules.javascript2.editor.navigation.OccurrencesSupport.findOccurrence(OccurrencesSupport.java:80)
	at org.netbeans.modules.javascript2.editor.navigation.OccurrencesSupport.findOccurrence(OccurrencesSupport.java:80)

and the indexer are affected:

java.lang.StackOverflowError
	at java.base/java.util.WeakHashMap.get(WeakHashMap.java:398)
	at org.openide.util.lookup.InheritanceTree.searchInterface(InheritanceTree.java:846)
	at org.openide.util.lookup.InheritanceTree.lookup(InheritanceTree.java:215)
	at org.openide.util.lookup.DelegatingStorage.lookup(DelegatingStorage.java:149)
	at org.openide.util.lookup.AbstractLookup.lookupItem(AbstractLookup.java:411)
	at org.openide.util.lookup.AbstractLookup.lookup(AbstractLookup.java:396)
	at org.openide.util.lookup.ProxyLookup.lookup(ProxyLookup.java:281)
	at org.netbeans.modules.javascript2.model.JsElementImpl.isInternalFile(JsElementImpl.java:194)
	at org.netbeans.modules.javascript2.model.JsElementImpl.isPlatform(JsElementImpl.java:188)
	at org.netbeans.modules.javascript2.model.api.IndexedElement$Flag.getFlag(IndexedElement.java:387)
	at org.netbeans.modules.javascript2.editor.index.JsIndexer.createDocument(JsIndexer.java:122)
	at org.netbeans.modules.javascript2.editor.index.JsIndexer.storeObject(JsIndexer.java:222)
	at org.netbeans.modules.javascript2.editor.index.JsIndexer.storeObject(JsIndexer.java:230)
	at org.netbeans.modules.javascript2.editor.index.JsIndexer.storeObject(JsIndexer.java:230)
	at org.netbeans.modules.javascript2.editor.index.JsIndexer.storeObject(JsIndexer.java:230)

the immediate idea is using a Map/Set to store the visited maps would fix this, but that breaks intentional usages.

Here is the original offender (name of the property and the indexable being indexed):

X1: nm$_constant.exports.formatArgs.constantformatArgs#=>#8.action.action.action (node_modules/webdriverio/build/commands/constant.js)
X1: nm$_constant.exports.formatArgs.constantformatArgs#=>#8.action.action.action.action (node_modules/webdriverio/build/commands/constant.js)
X1: nm$_constant.exports.formatArgs.constantformatArgs#=>#8.action.action.action.action.action (node_modules/webdriverio/build/commands/constant.js)
X1: nm$_constant.exports.formatArgs.constantformatArgs#=>#8.action.action.action.action.action.action (node_modules/webdriverio/build/commands/constant.js)
X1: nm$_constant.exports.formatArgs.constantformatArgs#=>#8.action.action.action.action.action.action.action (node_modules/webdriverio/build/commands/constant.js)
X1: nm$_constant.exports.formatArgs.constantformatArgs#=>#8.action.action.action.action.action.action.action.action (node_modules/webdriverio/build/commands/constant.js)

@matthiasblaesing matthiasblaesing removed the needs:triage Requires attention from one of the committers label Mar 4, 2022
matthiasblaesing added a commit to matthiasblaesing/netbeans that referenced this issue Mar 4, 2022
JS structures, that reference themselves can cause unlimited recursion
in several places. In this changeset the JsIndexer and
OccurrencesSupport are modified, so that recursive structures can be
scanned without causing a StackOverflowError.

Closes: apache#3669
matthiasblaesing added a commit to matthiasblaesing/netbeans that referenced this issue Mar 4, 2022
JS structures, that reference themselves can cause unlimited recursion
in several places. In this changeset the JsIndexer and
OccurrencesSupport are modified, so that recursive structures can be
scanned without causing a StackOverflowError.

Closes: apache#3669
@matthiasblaesing
Copy link
Contributor

I created a PR with changes, that at least allow parsing of the structures. It is most probably not perfect, but at least allows working the files in a sane way. Please see the referenced PR for the changes themselves. If you can't build from source, here is a test build:

NetBeans-dev-dev-18f9648d4000a2f0f025e23cf728f16812f9774c-release.zip

It would be good if you could test it.

@mossroy
Copy link
Author

mossroy commented Mar 6, 2022

Many thanks @matthiasblaesing
The StackOverflowError seems to be fixed with your test build.
I first reproduced the issue with NetBeans 13, then imported the settings in your test build (as suggested on its first startup).
The StackOverflowError did not show up, and the background scanning could finish, without eating CPU forever. Cool!

However, I have another exception appearing in NetBeans UI (on each startup), that I do not have on NetBeans 13:

java.lang.AssertionError
	at com.oracle.js.parser.Parser.verifyDestructuringBindingPattern(Parser.java:1930)
	at com.oracle.js.parser.Parser.verifyDestructuringParameterBindingPattern(Parser.java:4310)
	at com.oracle.js.parser.Parser.verifyArrowParameter(Parser.java:5105)
	at com.oracle.js.parser.Parser.convertArrowFunctionParameterList(Parser.java:5055)
	at com.oracle.js.parser.Parser.arrowFunction(Parser.java:4992)
	at com.oracle.js.parser.Parser.assignmentExpression(Parser.java:4935)
	at com.oracle.js.parser.Parser.assignmentExpression(Parser.java:4948)
	at com.oracle.js.parser.Parser.expression(Parser.java:4777)
	at com.oracle.js.parser.Parser.expression(Parser.java:4773)
	at com.oracle.js.parser.Parser.expressionStatement(Parser.java:1987)
	at com.oracle.js.parser.Parser.statement(Parser.java:1186)
	at com.oracle.js.parser.Parser.sourceElements(Parser.java:949)
	at com.oracle.js.parser.Parser.program(Parser.java:881)
	at com.oracle.js.parser.Parser.parse(Parser.java:334)
	at com.oracle.js.parser.Parser.parse(Parser.java:288)
	at org.netbeans.modules.javascript2.editor.parser.JsParser.parseSource(JsParser.java:106)
	at org.netbeans.modules.javascript2.editor.parser.JsParser.parseSource(JsParser.java:33)
	at org.netbeans.modules.javascript2.editor.parser.SanitizingParser.parseContext(SanitizingParser.java:225)
	at org.netbeans.modules.javascript2.editor.parser.SanitizingParser.parseContext(SanitizingParser.java:205)
	at org.netbeans.modules.javascript2.editor.parser.SanitizingParser.parseSource(SanitizingParser.java:118)
	at org.netbeans.modules.javascript2.editor.parser.SanitizingParser.parse(SanitizingParser.java:78)
	at org.netbeans.modules.parsing.impl.TaskProcessor.callParse(TaskProcessor.java:598)
	at org.netbeans.modules.parsing.api.ResultIterator.getParserResult(ResultIterator.java:112)
	at org.netbeans.modules.parsing.api.ParserManager$MultiUserTaskAction.run(ParserManager.java:171)
	at org.netbeans.modules.parsing.api.ParserManager$MultiUserTaskAction.run(ParserManager.java:140)
	at org.netbeans.modules.parsing.impl.TaskProcessor$2.call(TaskProcessor.java:181)
	at org.netbeans.modules.parsing.impl.TaskProcessor$2.call(TaskProcessor.java:178)
	at org.netbeans.modules.masterfs.filebasedfs.utils.FileChangedManager.priorityIO(FileChangedManager.java:153)
	at org.netbeans.modules.masterfs.providers.ProvidedExtensions.priorityIO(ProvidedExtensions.java:335)
	at org.netbeans.modules.parsing.nb.DataObjectEnvFactory.runPriorityIO(DataObjectEnvFactory.java:118)
	at org.netbeans.modules.parsing.impl.Utilities.runPriorityIO(Utilities.java:67)
	at org.netbeans.modules.parsing.impl.TaskProcessor.runUserTask(TaskProcessor.java:178)
	at org.netbeans.modules.parsing.api.ParserManager.parse(ParserManager.java:85)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.indexEmbedding(RepositoryUpdater.java:3253)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.doIndex(RepositoryUpdater.java:2861)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.lambda$index$0(RepositoryUpdater.java:2626)
	at org.netbeans.modules.parsing.impl.indexing.errors.TaskCache.refreshTransaction(TaskCache.java:540)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.index(RepositoryUpdater.java:2625)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$AbstractRootsWork.lambda$scanSource$3(RepositoryUpdater.java:5719)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.lambda$runInContext$4(RepositoryUpdater.java:2119)
	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:279)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.runInContext(RepositoryUpdater.java:2117)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.runInContext(RepositoryUpdater.java:2098)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.access$1400(RepositoryUpdater.java:135)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$AbstractRootsWork.scanSource(RepositoryUpdater.java:5754)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$AbstractRootsWork.scanSources(RepositoryUpdater.java:5427)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$RootsWork.getDone(RepositoryUpdater.java:5059)
[catch] at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.doTheWork(RepositoryUpdater.java:3436)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task._run(RepositoryUpdater.java:6181)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task.access$3400(RepositoryUpdater.java:5839)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task$2.lambda$call$0(RepositoryUpdater.java:6100)
	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:279)
	at org.netbeans.modules.parsing.impl.RunWhenScanFinishedSupport.performScan(RunWhenScanFinishedSupport.java:83)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task$2.call(RepositoryUpdater.java:6100)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task$2.call(RepositoryUpdater.java:6096)
	at org.netbeans.modules.masterfs.filebasedfs.utils.FileChangedManager.priorityIO(FileChangedManager.java:153)
	at org.netbeans.modules.masterfs.providers.ProvidedExtensions.priorityIO(ProvidedExtensions.java:335)
	at org.netbeans.modules.parsing.nb.DataObjectEnvFactory.runPriorityIO(DataObjectEnvFactory.java:118)
	at org.netbeans.modules.parsing.impl.Utilities.runPriorityIO(Utilities.java:67)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task.run(RepositoryUpdater.java:6096)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1418)
	at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:45)
	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:278)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2033)

@matthiasblaesing
Copy link
Contributor

However, I have another exception appearing in NetBeans UI (on each startup), that I do not have on NetBeans 13:

java.lang.AssertionError
	at com.oracle.js.parser.Parser.verifyDestructuringBindingPattern(Parser.java:1930)
	

Thank you for the test. The java.lang.AssertionError is known and is rooted in the parser. Please open the file etc/netbeans.conf from the build and remove the -J-ea from the netbeans_default_options

@mossroy
Copy link
Author

mossroy commented Mar 6, 2022

Thanks, the AssertionError disappears with the config you mentioned.

matthiasblaesing added a commit to matthiasblaesing/netbeans that referenced this issue Mar 6, 2022
JS structures, that reference themselves can cause unlimited recursion
in several places. In this changeset the JsIndexer and
OccurrencesSupport are modified, so that recursive structures can be
scanned without causing a StackOverflowError.

Closes: apache#3669
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:bug Bug report or fix
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants