Annotation processing is stuck in eclipse

[WonderCsabo]

I use eclipse to develop and build my projects. From the command line, i can compile with maven, but unfortunately eclipse cannot do that. The annotation processing is started, but somewhere it is stuck. I debugged the processing, and i made it to TransactionProcessorPool.execute(). Here the ExecutorService.execute() method is called, but the execution is stuck at ExecutorService.awaitTermination(). I guess the Transaction goes into a forever loop or something. Unfortunately i could not debug it any deeper.

I am using eclipse 4.3.2 on Windows 8. This is the sample project what i used.

[johncarl81]

We've had a number of reports about eclipse compatibility (#9, #10).

As you may be aware, Eclipse implements JSR-269 in its own way, significantly different than the native Java platform. I have not focused on Eclipse compatibility much because I write Android applications in Intellij and Android Studio backed up by a build tool like Maven or Gradle. These typically use Java's implementation of JSR-269 for annotation processing.

Do you know where the Transaction enters an infinite loop? I know one of AA's strengths is that is runs in Eclipse, could you help make Parceler (and maybe Transfuse) compatible with Eclipse? I would love to see feedback from someone familiar with Eclipse annotation processing.

[WonderCsabo]

Yeah, we also encountered that eclipse APT is implemented in a different than the standard.

Do you know where the Transaction enters an infinite loop?

As i said, i try to debug deeper but for some reason i cannot go into the methods call by TransactionProcessorPool.

I know one of AA's strengths is that is runs in Eclipse, could you help make Parceler (and maybe Transfuse) compatible with Eclipse?

Unfortunately i am not familiar with that. But i think @DayS think can help us. Since we want to integrate parceler into AndroidAnnotations and AA supports eclipse, then parceler should do that, too.

[WonderCsabo]

Good news. Today morning the compiler decided to work properly, and it seems i found what is blocking the processing to finish. It seems the generated file cannot be closed. Here is the stack trace, the execution is stuck at this point:

Thread [pool-3-thread-1]
    owns: CodeWriter$1  (id=154)    
    owns: BufferedWriter  (id=155)  
    BufferedWriter.close() line: not available [local variables unavailable]    
    PrintWriter.close() line: not available 
    JFormatter.close() line: 137    
    JPackage.build(CodeWriter, CodeWriter) line: 439    
    JCodeModel.build(CodeWriter, CodeWriter) line: 311  
    CodeGenerationScopedTransactionWorker<V,R>.innerRun(V) line: 49 
    CodeGenerationScopedTransactionWorker<V,R>(AbstractCompletionTransactionWorker<V,R>).run(V) line: 35    
    ScopedTransactionWorker<V,R>.run(V) line: 55    
    Transaction<V,R>.run() line: 77 
    ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: not available 
    ThreadPoolExecutor$Worker.run() line: not available 
    Thread.run() line: not available

I can reproduce the problem on Windows and Ubuntu, too.

[WonderCsabo]

I dig a little bit even further. Your FilerSourceCodeWriter is responsible for writing out the source file. The equivalent for that class in AA is SourceCodewriter. Note that the AA class does have an empty close() method, but your class are invoking stream closing in its close() method. Maybe the AA class intentionally does not close the stream at this point?

[WonderCsabo]

Ok, i removed the calls from close(), but the problem is still here. I do not know why the stream cannot be closed.

[johncarl81]

That's really strange that Eclipse is failing on that. I'll try to tinker with this over the weekend and see if I can reproduce and investigate.

Would really appreciate any further attention on this from you @WonderCsabo or @DayS. Thanks.

[johncarl81]

Closing as there hasn't been progress on this issue and Im unsure it can be fixed. Reopen if a fix is proposed.

[WonderCsabo]

Unfortunetaly i did not have the time to investigate this further but i think i already found the root of the issue. Did you try to debug parceler in an eclipse setup like i did?

[johncarl81]

Shoot. I have messed around with Parceler in Eclipse, but I haven't set it up to debug. Doesn't Eclipse only perform partial compilation?

[WonderCsabo]

What do you mean by performing partial compilation?

[johncarl81]

I think I'm going to close this as there hasn't been any progress and Im not actively engaged in the eclipse platform.

Bottom line here: follow the advice of the Pragmatic Programmer and use a command line tool to build your Android project.

Also, if someone wants to take on this cause, please reopen.

[WonderCsabo]

Yeah, it is a pain about Eclipse that it is using its on compiler which is not strictly compatible with the JDK. On the other side, i love incremental compilation and also annotation processors can show messages in the Eclipse editor directly.

BTW, to answer your question: eclipse can run incremental (if you check build project automatically), and can perform full build (if you clean the project and build after that).

[WonderCsabo]

I do not have much time, but i extracted the complete stacktrace (including the eclipse internal classes), maybe it's a good start.

Thread [pool-3-thread-1] (Suspended)    
    owns: OutputStreamWriter  (id=17838)    
    owns: CodeWriter$1  (id=17839)  
    owns: BufferedWriter  (id=17840)    
    waiting for: Object  (id=17841) 
    Object.wait(long) line: not available [native method]   
    Object.wait() line: 502 
    ThreadJob.waitForRun(ThreadJob, IProgressMonitor, InternalJob, Thread) line: 270    
    ThreadJob.joinRun(ThreadJob, IProgressMonitor) line: 197    
    ImplicitJobs.begin(ISchedulingRule, IProgressMonitor, boolean) line: 92 
    JobManager.beginRule(ISchedulingRule, IProgressMonitor) line: 290   
    WorkManager.checkIn(ISchedulingRule, IProgressMonitor) line: 118    
    Workspace.prepareOperation(ISchedulingRule, IProgressMonitor) line: 2251    
    Workspace.run(IWorkspaceRunnable, ISchedulingRule, int, IProgressMonitor) line: 2308    
    CreatePackageFragmentOperation(JavaModelOperation).runOperation(IProgressMonitor) line: 794 
    PackageFragmentRoot.createPackageFragment(String, boolean, IProgressMonitor) line: 302  
    CompilationUnitHelper.createPackageFragment(String, IPackageFragmentRoot, IProgressMonitor) line: 144   
    GeneratedFileManager.generateFileDuringBuild(Collection<IFile>, String, String, boolean, IProgressMonitor) line: 530    
    IdeJavaSourceOutputStream.close() line: 67  
    StreamEncoder.implClose() line: 320 
    StreamEncoder.close() line: 149 
    OutputStreamWriter.close() line: 233    
    CodeWriter$1(FilterWriter).close() line: 104    
    BufferedWriter.close() line: 266    
    PrintWriter.close() line: 339   
    JFormatter.close() line: 137    
    JPackage.build(CodeWriter, CodeWriter) line: 439    
    JCodeModel.build(CodeWriter, CodeWriter) line: 311  
    CodeGenerationScopedTransactionWorker.innerRun(V) line: 49  
    CodeGenerationScopedTransactionWorker(AbstractCompletionTransactionWorker).run(V) line: 35  
    ScopedTransactionWorker.run(V) line: 55 
    Transaction.run() line: 77  
    ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1142  
    ThreadPoolExecutor$Worker.run() line: 617   
    Thread.run() line: 745  

[johncarl81]

Thanks @WonderCsabo for the details. I'm not going to pursue this one personally, but if anyone else wants to review further it would be appreciated.

[titanseason]

Hi, I find a project at https://github.com/baoyongzhang/ParcelableGenerator, using eclipse to generate Parcelable. It works well.

[WonderCsabo]

I debbuged a little bit again, and it seems two threads are waiting on the same object. I think Parceler creates a deadlock in JDT APT. I try to look into it later.

[johncarl81]

@WonderCsabo, I appreciate you looking. My guess is that the transactional approach that is used is messing with how Eclipse expects a processor to work. We do use a single thread executor, but it does use a thread off of the original processor:
https://github.com/johncarl81/transfuse/blob/master/transfuse-support/src/main/java/org/androidtransfuse/transaction/TransactionProcessorPool.java#L56
I guess we could try to use the original thread here to try and avoid deadlock.

@titanseason, there are a number of solutions out there. I think Parceler is the most complete and user friendly (except for the Eclipse integration I'm sorry to say). Ugh, looks like ParcelableGenerator uses reflection heavily behind the annotation processor and doesn't support Collections well. I can't say I approve.

[WonderCsabo]

@johncarl81 can you create a test Parceler which uses on thread? If you do so, i can check it in Eclipse.

[johncarl81]

Sure, I'll get something to you this afternoon.

[johncarl81]

Here's the branch: https://github.com/johncarl81/parceler/tree/single_thread Do you need an artifact?

[WonderCsabo]

mvn install should do it, shouldn't it?

[WonderCsabo]

Also, i just want to confirm that the branch only contains a change in the
POM. Is that correct?

[johncarl81]

yes, that's correct... I just updated the transfuse dependency.

I just deployed the artifact if that's more convenient: https://oss.sonatype.org/content/repositories/snapshots/org/parceler/parceler/1.0.2-SNAPSHOT/parceler-1.0.2-20150626.204834-2.jar

[WonderCsabo]

Unfortunetaly it still stuck at:

Thread [pool-4-thread-1] (Suspended)    
    owns: OutputStreamWriter  (id=566)  
    owns: CodeWriter$1  (id=567)    
    owns: BufferedWriter  (id=568)  
    waiting for: Object  (id=569)   
    Object.wait(long) line: not available [native method]   
    Object.wait() line: 502 
    ThreadJob.waitForRun(ThreadJob, IProgressMonitor, InternalJob, Thread) line: 270    
    ThreadJob.joinRun(ThreadJob, IProgressMonitor) line: 197    
    ImplicitJobs.begin(ISchedulingRule, IProgressMonitor, boolean) line: 92 
    JobManager.beginRule(ISchedulingRule, IProgressMonitor) line: 307   
    WorkManager.checkIn(ISchedulingRule, IProgressMonitor) line: 120    
    Workspace.prepareOperation(ISchedulingRule, IProgressMonitor) line: 2189    
    Workspace.run(IWorkspaceRunnable, ISchedulingRule, int, IProgressMonitor) line: 2236    
    CreatePackageFragmentOperation(JavaModelOperation).runOperation(IProgressMonitor) line: 794 
    PackageFragmentRoot.createPackageFragment(String, boolean, IProgressMonitor) line: 302  
    CompilationUnitHelper.createPackageFragment(String, IPackageFragmentRoot, IProgressMonitor) line: 144   
    GeneratedFileManager.generateFileDuringBuild(Collection<IFile>, String, String, boolean, IProgressMonitor) line: 530    
    IdeJavaSourceOutputStream.close() line: 67  
    StreamEncoder.implClose() line: 320 
    StreamEncoder.close() line: 149 
    OutputStreamWriter.close() line: 233    
    CodeWriter$1(FilterWriter).close() line: 104    
    BufferedWriter.close() line: 266    
    PrintWriter.close() line: 339   
    JFormatter.close() line: 137    
    JPackage.build(CodeWriter, CodeWriter) line: 439    
    JCodeModel.build(CodeWriter, CodeWriter) line: 311  
    CodeGenerationScopedTransactionWorker.innerRun(V) line: 49  
    CodeGenerationScopedTransactionWorker(AbstractCompletionTransactionWorker).run(V) line: 35  
    ScopedTransactionWorker.run(V) line: 55 
    Transaction.run() line: 77  
    ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1142  
    ThreadPoolExecutor$Worker.run() line: 617   
    Thread.run() line: 745  

[WonderCsabo]

OOps, i used the old processor, testing the new one.

[WonderCsabo]

@johncarl81 seems to be working! Where i can find test projects to test all situations?

[johncarl81]

@WonderCsabo, Great news! This will be covered extensively by building transfuse and parceler, which are passing ATM:

https://travis-ci.org/johncarl81/parceler/builds/68529594
https://travis-ci.org/johncarl81/transfuse/builds/68528990

If you want you can build the branch I referenced and build the example projects. That should exercise this functionality to the n-th degree.

FYI: johncarl81/transfuse#177

Let me know if you need anything else.

[WonderCsabo]

Great! I will import and build the example projects in Eclipse. However I can only work on next week on this.

[johncarl81]

No problem @WonderCsabo, any further input is much appreciated. I imagine the ultimate test will be trying an AA project with Parceler in Eclipse. In the mean time I'll try to get a PR put together in our other effort.

Thanks again for the help on this. It's great to work together!

[WonderCsabo]

@johncarl81 unfortunetaly i could not test the examples because of two problems with @AutoValue:

• AutoValue processor is crashing in Eclipse:

@AutoValue processor threw an exception: java.lang.NullPointerException
    at com.google.auto.value.processor.AutoValueProcessor.getTypeMirror(AutoValueProcessor.java:644)
    at com.google.auto.value.processor.AutoValueProcessor.defineVarsForType(AutoValueProcessor.java:406)
    at com.google.auto.value.processor.AutoValueProcessor.processType(AutoValueProcessor.java:390)
    at com.google.auto.value.processor.AutoValueProcessor.process(AutoValueProcessor.java:128)

• javax.annotation.Generated is not available on the classpath of Eclipse Android projects, hence the generated @AutoValue classess will have compilation errors.

BTW, i think you should not use @AutoValue in your example projects or other framework, you should focus only on your annotations. You may create an additional example which will use integrations, though.

[johncarl81]

I went ahead and removed AutoValue from the examples on master. Go ahead and try again.

[WonderCsabo]

Great! I built and deployed the example app from Eclipse, it works. One annoying thing thought that it will mark references like ExampleParcel$$Parcelable as an error in the editor, but not in the problems list, and let's building and deploying the app.

[johncarl81]

Fantastic.

I'm not surprised the generated code shows up as an error. Have you seen Eclipse pick up the generated classes before? Should we be doing something different in Parceler?

[WonderCsabo]

Eclipse can pickup the generated classes (we always use that we AA too) , actually it finds them with ctrl + K. I think the problem is with the dollar notation.

[johncarl81]

Strange. Well, I always try to keep the developer from referencing generated code. You should be using the Parcels utility to wrap and unwrap the annotated POJOs.

[WonderCsabo]

I know that. ;) That reference is in the example project, but that is OK.

[johncarl81]

I have no idea on this one... have you been able to build the Gradle project in Eclipse?

[WonderCsabo]

Sorry, but i am not sure what do you mean. Android Gradle projects are not working in Eclipse BTW, and simple Gradle projects just got support in the latest Mars release.

But i think there is no problem here... Users should not reference the generated code, and if they do,they just have to ignore the red marks in the editor.

[johncarl81]

Fair enough.

Thanks again for your efforts here @WonderCsabo, I really appreciate it! 🍻

[WonderCsabo]

BTW, i just tried to run the tests with the Eclipse JUnit 4 launcher and i got an NPE in findSuperInterfaces. I guess Eclipse is really not designed for this.

[WonderCsabo]

BTW, i found a bigger problem: Parceler does not really support incremental compilation, I just added a new class with @Parcel annotation, and it was added to the generated Parceler$$Parcels class, however the other, already present parcelable class disappeared from that class...

[johncarl81]

I've been wondering about partial compilation, and I'm surprised it hasn't come up sooner. It seems with a proper build in place partial compilation is a rarity these days. That being said I think there are ways that we can handle partial compilation.

I wonder if the findSuperInterfaces is throwing an NPE also because the target type isn't being compiled (because of partial compilation).

[WonderCsabo]

I can debug findSuperInterfaces tomorrow, but incremental compilation is more interesting. AFAIK both Android Gradle (Jack and Jill) and Maven (Takari life cycle) teams are planning to rely on incremental compiler heavily.

[WonderCsabo]

I think we should open a separate issue for incremental.

[johncarl81]

Which means this is a good time to tackle it.

Agreed.

[WonderCsabo]

Let's open a separate issue, then. BTW, Parceler can still work if some classes are missing from the generated Parcels class, can it?

[WonderCsabo]

findSuperInterfaces is no longer a problem in Eclipse Mars, and all 44 tests passed in Eclipse.

[johncarl81]

Yes, the Parceler$$Parcels generated class is just a dictionary for the Parcels utility to map generated classes to the original annotated classes. If the map does not contain an entry for the given annotated type then the Parcels utility will resort to reflection to lookup the generated class.

Thats good news... The changes we just made to Transfuse (in particular this one: johncarl81/transfuse#176) must have avoid the findSuperInterfaces call in question.

[WonderCsabo]

Should we open a separate issue for partial compilation?

[johncarl81]

Yes!