Unable to generate dump on google/gson

[chrismwendt]

https://github.com/google/gson

./build/install/lsifjava/bin/lsifjava -projectRoot /Users/chrismwendt/github.com/google/gson -out /Users/chrismwendt/github.com/google/gson/dump.lsif
Exception in thread "main" spoon.SpoonException: Modules are only available since Java 9. Please set appropriate compliance level.
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.forEachCompilationUnit(JDTBasedSpoonCompiler.java:445)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildModel(JDTBasedSpoonCompiler.java:421)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnitsAndModel(JDTBasedSpoonCompiler.java:365)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildSources(JDTBasedSpoonCompiler.java:330)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:113)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:96)
	at spoon.Launcher.buildModel(Launcher.java:766)
	at ProjectIndexer.index(ProjectIndexer.java:41)
	at Main.main(Main.java:15)

[chrismwendt]

New exception coming from JDT https://github.com/eclipse/eclipse.jdt.core/commit/905307ba95aeaa4570286fd98cd5fee0aa7ea5d7#diff-1c6dd3b4a801c8d49da846e136f7f10cR129

Exception in thread "main" java.lang.IllegalStateException: Module should be known
        at org.eclipse.jdt.internal.compiler.batch.CompilationUnit.module(CompilationUnit.java:126)
        at org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration.module(CompilationUnitDeclaration.java:833)
        at org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.buildTypeBindings(LookupEnvironment.java:460)
        at org.eclipse.jdt.internal.compiler.Compiler.internalBeginToCompile(Compiler.java:845)
        at org.eclipse.jdt.internal.compiler.Compiler.beginToCompile(Compiler.java:394)
        at spoon.support.compiler.jdt.TreeBuilderCompiler.buildUnits(TreeBuilderCompiler.java:59)
        at spoon.support.compiler.jdt.JDTBatchCompiler.getUnits(JDTBatchCompiler.java:255)
        at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnits(JDTBasedSpoonCompiler.java:403)
        at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnitsAndModel(JDTBasedSpoonCompiler.java:362)
        at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildSources(JDTBasedSpoonCompiler.java:330)
        at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:113)
        at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:96)
        at spoon.Launcher.buildModel(Launcher.java:766)
        at ProjectIndexer.index(ProjectIndexer.java:43)
        at Main.main(Main.java:15)

Apparently this isn't supposed to happen 😕

[aidaeology]

Confirmed this issue is still occuring for google/gson repos:

Platform

Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
Java version: 14.0.1, vendor: N/A, runtime: /usr/local/Cellar/openjdk/14.0.1/libexec/openjdk.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.15.6", arch: "x86_64", family: "mac"

Command

% ~/Workspace/lsif-java/build/install/lsifjava/bin/lsifjava -projectRoot ~/Workspace/lsif-java-playground/projects-maven/gson -out gson-dump.lsif

Output

Running JVM 14.0.2
Arguments /Users/adewitt/Workspace/lsif-java-playground/projects-maven/gson ALL_SOURCE
Working Directory = /Users/adewitt/Workspace/lsif-java-playground/projects-maven
Absolute Path = /Users/adewitt/Workspace/lsif-java-playground/projects-maven
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details
Exception in thread "main" java.lang.IllegalStateException: Module should be known
	at org.eclipse.jdt.internal.compiler.batch.CompilationUnit.module(CompilationUnit.java:126)
	at org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration.module(CompilationUnitDeclaration.java:833)
	at org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.buildTypeBindings(LookupEnvironment.java:469)
	at org.eclipse.jdt.internal.compiler.Compiler.internalBeginToCompile(Compiler.java:845)
	at org.eclipse.jdt.internal.compiler.Compiler.beginToCompile(Compiler.java:394)
	at spoon.support.compiler.jdt.TreeBuilderCompiler.buildUnits(TreeBuilderCompiler.java:60)
	at spoon.support.compiler.jdt.JDTBatchCompiler.getUnits(JDTBatchCompiler.java:256)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnits(JDTBasedSpoonCompiler.java:417)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnitsAndModel(JDTBasedSpoonCompiler.java:369)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildSources(JDTBasedSpoonCompiler.java:337)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:114)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:97)
	at spoon.Launcher.buildModel(Launcher.java:755)
	at ProjectIndexer.index(ProjectIndexer.java:52)
	at Main.main(Main.java:18)

[Strum355]

Module should be known exception has been fixed upstream in INRIA/spoon#3549

Following that, the issue highlighted in INRIA/spoon#3550 has been fixed. PR pending at the time of writing.

Following that again, an issue in Spoon was detected where it trips over constructs with type params such as public static<T> where there is no space between the keyword and the type param. Fixed in INRIA/spoon#3552

The issue described in INRIA/spoon#3553 was encounted. This PR is still pending further investigation with Spoon maintainers to develop a better fix for it.

In combination, these PRs have brought lsif-java to the point where it is able to index google/gson without throwing an exception. There is still some investigation required wrt reducing the number of .java files that are not being picked up. PR to improve Spoons POM handling is in development at the time of writing.

[aidaeology]

@Strum355 It looks like INRIA/spoon#3567 resolves this item:

The issue described in INRIA/spoon#3553 was encountered. This PR is still pending further investigation with Spoon maintainers to develop a better fix for it.

Can you confirm if google/gson can be successfully indexed after the recent Spoon fixes?

[Strum355]

@aidaeology INRIA/spoon#3550 is still pending a proper review and to be merged.

[Strum355]

Indexes without exception with current latest Spoon snapshot (3.8.0-beta-8)
Some files are missing and a lot of "WARNING Skipping linking use to definition" but it runs otherwise 👍