Unable to generate LSIF dump for intuit/karate

[ryan-blunden]

Platform

macOS Mojave 10.14.6 (18G103)

Java info

openjdk 13.0.1 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

Steps to reproduce

./lsif-java/build/install/lsifjava/bin/lsifjava -projectRoot karate -out karate.lsif

Exception in thread "main" spoon.compiler.ModelBuildingException: The type SslTest is already defined
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.reportProblem(JDTBasedSpoonCompiler.java:575)
	at spoon.support.compiler.jdt.TreeBuilderRequestor.acceptResult(TreeBuilderRequestor.java:27)
	at spoon.support.compiler.jdt.TreeBuilderCompiler.buildUnits(TreeBuilderCompiler.java:86)
	at spoon.support.compiler.jdt.JDTBatchCompiler.getUnits(JDTBatchCompiler.java:255)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnits(JDTBasedSpoonCompiler.java:403)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnitsAndModel(JDTBasedSpoonCompiler.java:362)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildSources(JDTBasedSpoonCompiler.java:330)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:113)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:96)
	at spoon.Launcher.buildModel(Launcher.java:766)
	at ProjectIndexer.index(ProjectIndexer.java:41)
	at Main.main(Main.java:15)

[chrismwendt]

I ran into this once before. All I have right now is a workaround: ignore test files.

Edit Spoon explicitly supports ignoring duplicate definitions INRIA/spoon#2772 Fixed in 45a3d27

[chrismwendt]

The indexer gets further, but there's a null pointer exception:

Exception in thread "main" java.lang.NullPointerException
	at DocumentIndexer.identifierRange(DocumentIndexer.java:312)
	at DocumentIndexer.access$1100(DocumentIndexer.java:16)
	at DocumentIndexer$ReferencesVisitor.visitCtFieldRead(DocumentIndexer.java:202)
	at spoon.support.reflect.code.CtFieldReadImpl.accept(CtFieldReadImpl.java:16)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:139)
	at spoon.reflect.visitor.CtScanner.visitCtNewArray(CtScanner.java:571)
	at spoon.support.reflect.code.CtNewArrayImpl.accept(CtNewArrayImpl.java:32)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:149)
	at spoon.reflect.visitor.CtScanner.visitCtAnnotation(CtScanner.java:183)
	at spoon.support.reflect.declaration.CtAnnotationImpl.accept(CtAnnotationImpl.java:85)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:139)
	at spoon.reflect.visitor.CtScanner.visitCtMethod(CtScanner.java:546)
	at spoon.support.reflect.declaration.CtMethodImpl.accept(CtMethodImpl.java:56)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:139)
	at spoon.reflect.visitor.CtScanner.visitCtClass(CtScanner.java:330)
	at spoon.support.reflect.declaration.CtClassImpl.accept(CtClassImpl.java:56)
	at DocumentIndexer.visitReferences(DocumentIndexer.java:55)
	at ProjectIndexer.index(ProjectIndexer.java:70)
	at Main.main(Main.java:15)

Fixed in 7acf525

[ryan-blunden]

I get a NullPointerException also, but of a different type:

Linking use to definition: /Users/rb/External/karate/karate-junit5/src/main/java/com/intuit/karate/junit5/FeatureNode.java:81:29-81:32 -> /Users/rb/External/karate/karate-junit5/src/main/java/com/intuit/karate/junit5/FeatureNode.java:78:31-78:53
Exception in thread "main" java.lang.NullPointerException
	at DocumentIndexer$ReferencesVisitor.visitCtVariableRead(DocumentIndexer.java:174)
	at spoon.support.reflect.code.CtVariableReadImpl.accept(CtVariableReadImpl.java:16)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.visitCtInvocation(CtScanner.java:497)
	at DocumentIndexer$ReferencesVisitor.visitCtInvocation(DocumentIndexer.java:181)
	at spoon.support.reflect.code.CtInvocationImpl.accept(CtInvocationImpl.java:44)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.visitCtIf(CtScanner.java:476)
	at spoon.support.reflect.code.CtIfImpl.accept(CtIfImpl.java:34)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:139)
	at spoon.reflect.visitor.CtScanner.visitCtBlock(CtScanner.java:294)
	at spoon.support.reflect.code.CtBlockImpl.accept(CtBlockImpl.java:55)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.visitCtLambda(CtScanner.java:608)
	at spoon.support.reflect.code.CtLambdaImpl.accept(CtLambdaImpl.java:54)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:139)
	at spoon.reflect.visitor.CtScanner.visitCtInvocation(CtScanner.java:499)
	at DocumentIndexer$ReferencesVisitor.visitCtInvocation(DocumentIndexer.java:181)
	at spoon.support.reflect.code.CtInvocationImpl.accept(CtInvocationImpl.java:44)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.visitCtReturn(CtScanner.java:670)
	at spoon.support.reflect.code.CtReturnImpl.accept(CtReturnImpl.java:25)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:139)
	at spoon.reflect.visitor.CtScanner.visitCtBlock(CtScanner.java:294)
	at spoon.support.reflect.code.CtBlockImpl.accept(CtBlockImpl.java:55)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.visitCtMethod(CtScanner.java:551)
	at spoon.support.reflect.declaration.CtMethodImpl.accept(CtMethodImpl.java:56)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:139)
	at spoon.reflect.visitor.CtScanner.visitCtClass(CtScanner.java:330)
	at spoon.support.reflect.declaration.CtClassImpl.accept(CtClassImpl.java:56)
	at DocumentIndexer.visitReferences(DocumentIndexer.java:55)
	at ProjectIndexer.index(ProjectIndexer.java:72)
	at Main.main(Main.java:15)

[chrismwendt]

This combination caused the NullPointerException:

• This repository multiple modules
• Some symbols in one module are referenced in others
• Spoon doesn't appear to support cross-module definitions

Specifically, this usage of result:

https://github.com/intuit/karate/blob/ecd63384bba0475c158d14c529d1c93b8babc771/karate-junit5/src/main/java/com/intuit/karate/junit5/FeatureNode.java#L82

refers to:

https://github.com/intuit/karate/blob/ecd63384bba0475c158d14c529d1c93b8babc771/karate-core/src/main/java/com/intuit/karate/core/ScenarioExecutionUnit.java#L42

But el.getVariable().getDeclaration() is null:

https://github.com/sourcegraph/lsif-java/blob/68d3bfbbd3afc791deddad7004a55191914c7949/src/main/java/DocumentIndexer.java#L174