annotations processors support

[viktortnk]

Hi

I've got a project which Pants compiles fine, but IDEA highlights with red usage of generated classes.
I've tried to setup annotation processors directly in IDEA, but that doesn't help.
Specifically I'm trying to build example with Java and Immutables

Will it be possible to address it?

[wisechengyi]

Hi @viktortnk,

This issue should be directly related to the support for immutables in IntelliJ. immutables/immutables#404

Another example is https://projectlombok.org/, which generates code based on the annotations, and because of that, a special plugin is required for it to function properly in IntelliJ

Thanks,
Yi

[viktortnk]

Hi @wisechengyi

Thanks for quick reply. I had the same issue with AutoValue, but both Buck and Bazel's IDEA plugins discover generated classes after compiling

[wisechengyi]

Interesting, I have not tried Autovalue with Buck or Bazel before, but sounds like I should do that. If you have some examples, that'll be nice.

That said, I do see AutoValue has its own plugin https://plugins.jetbrains.com/plugin/8091-autovalue-plugin, do you think it's possible the plugin is masking the effect?

[viktortnk]

I haven't been using this plugin. Which suggests that the output of annotation processor for build tool was residing in IDEA classpath.

In Pants case I have both generated java and class files sitting in

./pants.d/compile/zinc/current/<lib>/current/classes/<lib>

But they are not being picked up by IDEA

[wisechengyi]

Can you zip up the example pants project with immutables and attach it here?

I didn't realize the code has been generated by Pants already. In that case it could be just an easy tweak, but I will have to fiddle with it a bit.

[viktortnk]

Sure, attached

pants-immutables.zip

[wisechengyi]

Thanks. The issue is that .pants.d/compile/zinc/c95338cea9cc/src.java.lib.lib/a89982c79b13/classes/lib/ImmutableFoobarValue.java is generated at compile stage, which is NOT triggered in ./pants export (the cmd intellij calls to import your project)

My speculation is that bazel/buck is doing something extra to inject compiled/generated code back to intellij. I did that manually in the example project and it was able to resolve.

Pants does have codegen, but that's in the sense of thrift or protobuf, whereas code generated by annotation processor needs actual compiler work.

I do not have a good answer for you at this time in terms of what Pants/its intellij plugin should do. Potentially I need to dig in how bazel is handling it.

Options:

1. Use autovalues or lombok, and there are intellij plugins for them, which probably generate code realtime.
2. Use Scala where immutables are builtin.
3. Keep bumping Intellij idea support  immutables/immutables#404 :)

[viktortnk]

Makes sense, thanks. I just though that if Pants internally using this folder as sources location, then maybe idea plugin should do the same.

[JasonRosenberg]

Any update on this?

[wisechengyi]

Hi @JasonRosenberg, my recommendation is still to use a specific IntelliJ plugin for a particular annotation processor, like lombok, if available.

The complication for Pants or Pants plugin to implement it are:

1. It would mean that all the intermediate classfiles will be put into indexing as well, and reindexing is needed likely after every pants run if anything changes. This would not scale very well in large repos.
2. Pants, unlike Gradle, does not have the feature to specify a dependency that is particularly used for annotation processing, so it would be difficult for Pants to narrow the scope of potential changes.

[JasonRosenberg]

hi @wisechengyi I tried installing the AutoValue plugin, it doesn't seem to help. Pants itself has no problem working with these annotations, which makes me wonder why the plugin can't be made to automatically know where the generated class files are being built to. Seems like it would just be a matter of adding the generated classes dir to the project class path?

[JasonRosenberg]

For example, in my project, pants is generating the compiled sources from the AutoValue annotation like so:

./.pants.d/compile/javac/252d64521cf9/myapp.src.main.java.lib/e5c8f2931085/classes/com/mypackage/AutoValue_MyEntry.java
./.pants.d/compile/javac/252d64521cf9/myapp.src.main.java.lib/e5c8f2931085/classes/com/mypackage/AutoValue_MyEntry.class
...
...

[wisechengyi]

Ah I see, the autovalue plugin does not help resolving the symbols, unlike the lombok plugin.

That said, what I mentioned above (#310 (comment)) is still valid.

There is also additional work for Pants to tell IntelliJ where the class files are after every compile via some kind of API.

Overall I don't think this issue is worth pursuing given cost vs benefit.

That said, PR still welcome.

[JasonRosenberg]

well, it looks like this same dir is where all compiled classes go for a pants module. so surely, it should be in a standard place known by the plugin? It seems to be the main output dir for all compiled classes (not just the ones that are from annotation processors).

It's the main "classes" dir for compiled ".class" files. And for generated classes, there are also ".java" files in there.

There's nothing magical about a particular annotation processor, I shouldn't think?

[wisechengyi]

The directories those classfiles live under are not static. e.g.

.pants.d/compile/zinc/
.pants.d/compile/zinc//a53b6931478d
.pants.d/compile/zinc//a53b6931478d/examples.src.java.org.pantsbuild.example.autovalue.autovalue-lib
.pants.d/compile/zinc//a53b6931478d/examples.src.java.org.pantsbuild.example.autovalue.autovalue-lib/59648d2af80c
.pants.d/compile/zinc//a53b6931478d/examples.src.java.org.pantsbuild.example.autovalue.autovalue-lib/59648d2af80c/backup
.pants.d/compile/zinc//a53b6931478d/examples.src.java.org.pantsbuild.example.autovalue.autovalue-lib/59648d2af80c/classes
.pants.d/compile/zinc//a53b6931478d/examples.src.java.org.pantsbuild.example.autovalue.autovalue-lib/59648d2af80c/classes/compile_classpath
.pants.d/compile/zinc//a53b6931478d/examples.src.java.org.pantsbuild.example.autovalue.autovalue-lib/59648d2af80c/classes/compile_classpath/examples.src.java.org.pantsbuild.example.autovalue.autovalue-lib.txt
.pants.d/compile/zinc//a53b6931478d/examples.src.java.org.pantsbuild.example.autovalue.autovalue-lib/59648d2af80c/classes/org
.pants.d/compile/zinc//a53b6931478d/examples.src.java.org.pantsbuild.example.autovalue.autovalue-lib/59648d2af80c/classes/org/pantsbuild
.pants.d/compile/zinc//a53b6931478d/examples.src.java.org.pantsbuild.example.autovalue.autovalue-lib/59648d2af80c/classes/org/pantsbuild/example
.pants.d/compile/zinc//a53b6931478d/examples.src.java.org.pantsbuild.example.autovalue.autovalue-lib/59648d2af80c/classes/org/pantsbuild/example/autovalue
.pants.d/compile/zinc//a53b6931478d/examples.src.java.org.pantsbuild.example.autovalue.autovalue-lib/59648d2af80c/classes/org/pantsbuild/example/autovalue/AutoValue_AutoValueMain_ValueType.class
.pants.d/compile/zinc//a53b6931478d/examples.src.java.org.pantsbuild.example.autovalue.autovalue-lib/59648d2af80c/classes/org/pantsbuild/example/autovalue/AutoValue_AutoValueMain_ValueType.java
.pants.d/compile/zinc//a53b6931478d/examples.src.java.org.pantsbuild.example.autovalue.autovalue-lib/59648d2af80c/classes/org/pantsbuild/example/autovalue/AutoValueMain$ValueType.class
.pants.d/compile/zinc//a53b6931478d/examples.src.java.org.pantsbuild.example.autovalue.autovalue-lib/59648d2af80c/classes/org/pantsbuild/example/autovalue/AutoValueMain.class

Components

• SHA a53b6931478d encodes the task version + some other inputs
• examples.src.java.org.pantsbuild.example.autovalue.autovalue-lib is the target id
• 59648d2af80c is SHA for the target content.

These SHAs are all computed by Pants, hence IntelliJ would need to query Pants for them.

You might also notice there are current symlinks

.pants.d/compile/zinc//a53b6931478d/examples.tests.java.org.pantsbuild.example.hello.greet.greet/current
.pants.d/compile/zinc//current

This would may be okay for a single project, but when multiple projects are open, things will get confusing quickly.

Potential option

The cleanest approach, I think, is to follow Gradle's pattern to create a separate doing annotation processing and generate files into a separate dir, before actual compilation.

[JasonRosenberg]

I see, so it would need to be something similar to what we do for generating java classes from protocol buffers, etc. Those generally result in new pants module dependencies, etc.....

It might be difficult in this case though, since the source files for these don't exist in separate modules (as we would do for proto modules).

Regardless, it does seem like since annotation processors are first-class java constructs, and the pants build system knows how to handle them in a first-class way, we should figure a way to solve this also for the plugin :)

[wisechengyi]

I am guessing this would be easier when Pants tasks shift to v2 implementation, so there is more flexibility of composing the tasks.

As of right now, the cost in various aspects are likely too high.

The indexing churn from generated files isn't insignificant when we are doing Android builds from Gradle, which is only a medium size project.