Compilation bug, spoon silently doesn't compile any source code in environment where new File(".") points to the folder without any java sources (for example in tomcat environment where it points to the bin folder)

[gnom7]

Hello, I have a project that uses spoon 5.3.0. but as for now due to missing org.eclipse.jdt.core/3.12.0.v20160516-2131 (on which spoon relies) in maven repo (and because of that problem some builds on my CI have been failed) I consider switching to spoon 5.6.0, but it seems that I have faced a bug -- when I tested my code in IDE everything was fine, but the same code didn't work in tomcat environment - spoon silently didn't compiled any source code.

Debugging:
When running compile() on SpoonModelBuilder it calls batchCompiler.compile(args); so those args doesn't contain input sources, but "."

final String[] args = new JDTBuilderImpl() //
				.classpathOptions(new ClasspathOptions().encoding(this.encoding).classpath(getSourceClasspath()).binaries(getBinaryOutputDirectory())) //
				.complianceOptions(new ComplianceOptions().compliance(javaCompliance)) //
				.annotationProcessingOptions(new AnnotationProcessingOptions().compileProcessors()) //
				.advancedOptions(new AdvancedOptions().preserveUnusedVars().continueExecution().enableJavadoc()) //
				.sources(new SourceOptions().sources()) // no sources, handled by the JDTBatchCompiler
				.build();

		getFactory().getEnvironment().debugMessage("compile args: " + Arrays.toString(args));
		System.setProperty("jdt.compiler.useSingleThread", "true");
		batchCompiler.compile(args);

then in org.eclipse.jdt.internal.compiler.batch.Main it creates new File(".") from that argument and calls org.eclipse.jdt.internal.compiler.batch.FileFinder which recursively searchs for files with SuffixConstants.SUFFIX_STRING_JAVA (".java") extension, so when running in tomcat new File(".") points to tomcat bin folder wich does not contain any .java files (and running in IDE in sandbox project it points to project base dir -- so it contains .java files), then in Main class filesCount == 0 is true (there is no .java files in tomcat bin folder) and compilation termiates even not being started and in IDE filesCount != 0 (project base dir contains .java files) and compilation proceeds. I haven't debugged further more, and I don't know how then compiled files appeared from VALID sources (not from the project base)...
when filesCount == 0, the code below in Main class switches proceed flag to false and eclipse compiler dies.

	if (printUsageRequired || (filesCount == 0 && classCount == 0)) {
		if (usageSection ==  null) {
			printUsage(); // default
		} else {
			printUsage(usageSection);
		}
		this.proceed = false;
		return;
	}

/*
 *  Low-level API performing the actual compilation
 */
public boolean compile(String[] argv) {
	// decode command line arguments
	try {
		configure(argv); //!!!! code above is here
		if (this.progress != null)
			this.progress.begin(this.filenames == null ? 0 : this.filenames.length * this.maxRepetition);
		if (this.proceed) { //!!!! the proceed flag was already switched to false....

I have used InputType.FILES - compiles the java files from the file system, which were registered by {@link #addInputSource(File)} and {@link #addTemplateSource(File)}
and as was said in the above javaDoc compiler.addInputSource(spoonResource); method, so it seems to me that I have done all right.
(the same for v5.5.0)
(for v5.4.0 eclipse compiler is missing too)
The sample of my code is below.

SpoonModelBuilder compiler = new Launcher().createCompiler();
compiler.setSourceClasspath(classpath);
compiler.addInputSource(new File("/sourceFolder"));
compiler.setBinaryOutputDirectory(new File("/outputFolder"));
compiler.compile(SpoonModelBuilder.InputType.FILES);

Please note that the same code runs fine directly from IDE via main method, and doesn't work when running in tomcat in bean initialization code.

You can create small spring-boot project with dummy component with @PostConstruct method in which you can try to compile something, like I did (just place resulting jar with project and embedded tomcat in separate empty folder and run java -jar your.jar, then add to the folder some .java files and compilation will be performed).
So the matter is that with such implementation when new File(".") folder does not contain java source code spoon will not be able to compile wichever other source code.

[surli]

Hi @gnom7

we indeed spot this problem and fix it in Spoon 5.6.0 (see #1208). Thanks anyway for the complete bug report! 👍

You can use the following dependency for jdt.core to switch to version 5.6.0, Spoon 5.3.0 is not maintained anymore:

<!-- https://mvnrepository.com/artifact/org.eclipse.tycho/org.eclipse.jdt.core -->
<dependency>
    <groupId>org.eclipse.tycho</groupId>
    <artifactId>org.eclipse.jdt.core</artifactId>
    <version>3.12.2.v20161117-1814</version>
</dependency>

[gnom7]

we indeed spot this problem and fix it in Spoon 5.6.0

v5.6.0 silently does not compile sources, v5.5.0 throws NPE (I think for the same reason as in my issue) -- so this problem doesn't seem to be fixed in v5.6.0

The missing eclipse compiler dependency I was talking about is for v5.3.0
https://mvnrepository.com/artifact/org.eclipse.jdt/org.eclipse.jdt.core/3.12.0.v20160516-2131

You can use the following dependency for jdt.core to switch to version 5.6.0

The problem to switch to v5.6.0 is a bug, not a missing compiler dependency

[surli]

Ok my mistake.
Then could you give more information about the way you launch Spoon: to fix the bug with the NPE, we are using the path to the source files, so how do you give your source code to Spoon?

[gnom7]

Maybe I'm wrong, but for version 5.6.0 spoon still passes "." to eclipse compiler. The way I launch spoon was described in my first message - I use spring and tomcat8, and also I use spoon for code transformation and compilation, so in @PostConstruct method I call spoon compiler, but then as nothing appears in output directory I catch FileNotFoundException.

SpoonModelBuilder compiler = new Launcher().createCompiler();
compiler.setSourceClasspath(classpath);
compiler.addInputSource(new File("/sourceFolder"));
compiler.setBinaryOutputDirectory(new File("/outputFolder"));
compiler.compile(SpoonModelBuilder.InputType.FILES);

[surli]

The debugging session you show in your first message, are you sure it was with Spoon 5.6.0?
Because we changed the snippet of code you show in JDTBasedSpoonCompiler.java:

final String[] args = new JDTBuilderImpl() //
				.classpathOptions(new ClasspathOptions().encoding(this.encoding).classpath(getSourceClasspath()).binaries(getBinaryOutputDirectory())) //
				.complianceOptions(new ComplianceOptions().compliance(javaCompliance)) //
				.annotationProcessingOptions(new AnnotationProcessingOptions().compileProcessors()) //
				.advancedOptions(new AdvancedOptions().preserveUnusedVars().continueExecution().enableJavadoc()) //
				//.sources(new SourceOptions().sources()) // this line was removed to
                                .sources(new SourceOptions().sources(sourcesFolder.getAllJavaFiles()))
				.build();

Just to be sure, could you test with Spoon 5.7.0-SNAPSHOT?

[gnom7]

That's what I have from maven repo spoon v5.6.0.
spoon-core-5.6.0-sources.zip

	@Override
	public boolean compile(InputType... types) {
		factory.getEnvironment().debugMessage("compiling sources: " + factory.CompilationUnit().getMap().keySet());
		long t = System.currentTimeMillis();
		javaCompliance = factory.getEnvironment().getComplianceLevel();

		JDTBatchCompiler batchCompiler = createBatchCompiler(types);


		final String[] args = new JDTBuilderImpl() //
				.classpathOptions(new ClasspathOptions().encoding(this.encoding).classpath(getSourceClasspath()).binaries(getBinaryOutputDirectory())) //
				.complianceOptions(new ComplianceOptions().compliance(javaCompliance)) //
				.annotationProcessingOptions(new AnnotationProcessingOptions().compileProcessors()) //
				.advancedOptions(new AdvancedOptions().preserveUnusedVars().continueExecution().enableJavadoc()) //
				.sources(new SourceOptions().sources()) // no sources, handled by the JDTBatchCompiler
				.build();

		getFactory().getEnvironment().debugMessage("compile args: " + Arrays.toString(args));
		System.setProperty("jdt.compiler.useSingleThread", "true");
		batchCompiler.compile(args);

		reportProblems(factory.getEnvironment());
		factory.getEnvironment().debugMessage("compiled in " + (System.currentTimeMillis() - t) + " ms");
		return probs.size() == 0;
	}

[gnom7]

Just to be sure, could you test with Spoon 5.7.0-SNAPSHOT?

Ok, I'll do.

[surli]

Ok, I understand: we fix the bug for the model building, but not for the model compilation, it's in the same file but not at the same place. I'll propose a PR to fix it, so it will be in the next Spoon release.

[gnom7]

Thanks.

[gnom7]

But for now, have you any advice for me how to use v5.3.0 with missing
https://mvnrepository.com/artifact/org.eclipse.jdt/org.eclipse.jdt.core/3.12.0.v20160516-2131
Maybe you can switch to another version of this compiler or should I create local maven repo?

[surli]

You can use our own repository, we release this version of jdt core (see http://spoon.gforge.inria.fr/repositories/releases/org/eclipse/jdt/org.eclipse.jdt.core/3.12.0.v20160516-2131/)

<repository>
      <id>gforge.inria.fr-releases</id>
      <name>Maven Repository for Spoon</name>
      <url>http://spoon.gforge.inria.fr/repositories/releases/</url>
 </repository>

[gnom7]

Thanks again.

[surli]

I was looking to fix the bug, but in fact I'm not sure to understand what you're trying to do there: usually you don't use Spoon to compile files directly, first you build the model to manipulate the AST. Then why are you trying to compile files first? (Note, that your bug should not occur if you build the model first)

[gnom7]

I have a bunch of sources, some of them I compile first, pack to jar and use as part of classpath. Another sources require transformation, then, again, compilation and packaging to jar.
So with first ones I use just:

SpoonModelBuilder compiler = new Launcher().createCompiler();
compiler.setSourceClasspath(classpath);
compiler.addInputSource(new File("/sourceFolder"));
compiler.setBinaryOutputDirectory(new File("/outputFolder"));
compiler.compile(SpoonModelBuilder.InputType.FILES);

without any manipulations on code and therefore without building model, just compiling directly.