Extract ECJ compiler code from jdt.core to ecj bundle and make ecj a real plug-in

[iloveeclipse]

Follow up on eclipse-platform/eclipse.platform.ua#18 (comment).

Currently JDT has LOT of hacks around compiler part (dedicated source folders, dedicated build files, dedicated deployment steps, dedicated validation bundle) - all because the standalone Eclipse Compiler for Java (ECJ) code is located inside the jdt.core project that contains additionally all the IDE specific extensions.

Ideally ECJ could be just extracted from the jdt.core bundle so it can be used by jdt.core and other bundles as a regular bundle.

This would remove most of the custom hacks around building, testing and shipping ecj we have today.

I will provide a draft idea, but I have no enough time/knowledge to perform all the build/maven magic required to get that working.
Technically I believe this should work without breaking existing jdt.core clients.

[iloveeclipse]

@stephan-herrmann , @jarthana : WDYT about #182 ?

Do you have any idea why org.eclipse.jdt.compiler.apt and org.eclipse.jdt.compiler.tool are fragments of jdt.core? Just a hack to get the standalone part compiled separately?

Would there be any issue if we just merge all three code bases together (ecj + org.eclipse.jdt.compiler.apt + org.eclipse.jdt.compiler.tool) in one real bundle? At the end that is what ecj.jar does?

[stephan-herrmann]

Do you have any idea why org.eclipse.jdt.compiler.apt and org.eclipse.jdt.compiler.tool are fragments of jdt.core? Just a hack to get the standalone part compiled separately?

I can only guess / infer from what I see in our history. Both fragments implement additional capabilities connected to the compiler that were introduced around Java 6 (JSR 269 and JSR 199), right? As such the had to set BREE 1.6, while the rest of JDT/Core was still at 1.4 (!). That's why the implementation had to be kept separate.

If that was the only reason, then the reason is gone in times where JDT/Core requires 11.

For authoritative answers you'd have to ask Walter Harley or Olivier Thomann, I believe. I haven't heard from Walter for ages, but Olivier might still respond?

[guw]

+1 for merging the fragments into a new ECJ bundle. They make ECJ a better offering in the Java ecosystem.

My hypothesis is that they are separate because they have been created long time ago while JDT was much more strict with who and what goes into jdt.core.

[stephan-herrmann]

My hypothesis ...

The JDK 6 dependency was a sufficient reason to keep those fragments separate (see my previous comment). I just wasn't sure if that was the only reason.

As of today I cannot think of any reason against merging.

[jarthana]

@stephan-herrmann , @jarthana : WDYT about #182 ?

I am not sure I understand the changes in org.eclipse.jdt.core/.classpath. What does this mean to org.eclipse.jdt.core as a Java Project in the IDE?

Do you have any idea why org.eclipse.jdt.compiler.apt and org.eclipse.jdt.compiler.tool are fragments of jdt.core? Just a hack to get the standalone part compiled separately?

I agree with Stephan's reasoning.

Would there be any issue if we just merge all three code bases together (ecj + org.eclipse.jdt.compiler.apt + org.eclipse.jdt.compiler.tool) in one real bundle? At the end that is what ecj.jar does?

I can't think of a reason why this will be a big problem. Except that org.eclipse.jdt.core, when shipped with IDE, will contain some code that won't be useful. Or are you suggesting that we continue to make two bundles?

[iloveeclipse]

are you suggesting that we continue to make two bundles?

Yes. One "ecj" bundle for batch compiler (and apt) and one "core" bundle for the rest of jdt, that depends on ecj.

[stephan-herrmann]

are you suggesting that we continue to make two bundles?

Yes. One "ecj" bundle for batch compiler (and apt) and one "core" bundle for the rest of jdt, that depends on ecj.

I'm not sure this is the same split that @jarthana had in mind, see:

org.eclipse.jdt.core, when shipped with IDE, will contain some code that won't be useful.

This seems to refer to code from source folder antadapter, right?

[jarthana]

org.eclipse.jdt.core, when shipped with IDE, will contain some code that won't be useful.

This seems to refer to code from source folder antadapter, right?

I had that and "batch" in my mind. But just checked the jdt.core bundle and to my surprise found all those classes already in there. We really don't need them, do we? Perhaps there's some history I don't remember.

[stephan-herrmann]

org.eclipse.jdt.core, when shipped with IDE, will contain some code that won't be useful.

This seems to refer to code from source folder antadapter, right?

I had that and "batch" in my mind. But just checked the jdt.core bundle and to my surprise found all those classes already in there. We really don't need them, do we? Perhaps there's some history I don't remember.

Well, who remembers the decisions from 2001 or earlier :)

Just two small observations:

• org.eclipse.jdt.core.compiler.batch.BatchCompiler is API, so if it's in jdt.core now, it must remain there. It then pulls in all the rest from "batch".
• lot's of our tests use "batch", so they need a way to depend on it

[jarthana]

Thanks Stephan. So, that looks like it's only the antadapter then. In that case, then my previous concern of adding redundant code to jdt.core is no longer a big deal.

[iloveeclipse]

• org.eclipse.jdt.core.compiler.batch.BatchCompiler is API, so if it's in jdt.core now, it must remain there.

Why? It can be re-exported from JDT core. Do I miss something?

I've updated draft patch to actually move all the code to the final place, rename bundle etc.

Main point: we don't have 3 different jars now (ecj.jar / org.eclipse.jdt.compiler.apt / org.eclipse.jdt.compiler.tool), we don't need to "merge" them, we have everything inside org.eclipse.jdt.core.compiler.batch bundle.

jdt.core reexports API from org.eclipse.jdt.core.compiler.batch.

Note: I've used "org.eclipse.jdt.core.compiler.batch" name because that was used in MANIFEST.MF of the ecj.jar.

See #182

With this code I have no compile errors in my SDK workspace (with PDE,JDT UI etc) and I can start Eclipse from Eclipse and everything still works.

Here are current TODO's:

1. What I did NOT tried is to re-write all the magic scripts that build
   and package separated ecj library out of jdt.core. I have no idea about maven and current JDT build magic.
2. The 3 antadapter classes are now split over ecj and jdt core, because
   BuildJarIndex.java and CheckDebugAttributes.java depend on JDT core
   code.
3. ecj.1 and build_ecj.xml aren't touched yet. ecj.1 was touched last
   time 2017, it seem to be used for man pages.
4. pom from jdt core will need an adoption.
5. org.eclipse.jdt-feature need to be updated (trivial)
6. Current maven build fails with "chicken and egg" problem: jdt core needs ecj bundle, but the bundle isn't there yet, I have no idea how that is supposed to be done: Missing requirement: org.eclipse.jdt.core 3.32.0.qualifier requires 'osgi.bundle; org.eclipse.jdt.core.compiler.batch 3.32.0' but it could not be found.
7. TBC

@akurtakov , @sravanlakkimsetti , @jarthana , @mpalat : I need your help:

1. It would be great if people who know some of the magic behind building compiler could help to update scripts logic. I believe I've heard that we build some grammar somehow manually - that script/artifacts, if in jdt core, must be probebly moved to org.eclipse.jdt.core.compiler.batch bundle.
2. It would be great if people who know some of the magic behind building maven artifacts could check what needed for org.eclipse.jdt.core.compiler.batch module. I've copied pom.xml from jdt.core and adopted that a bit, but I guess there is lot more work need to be done.
3. I guess there must be something done for platform aggregator once jdt can be built.
4. SDK Download page lists "JDT Core Batch Compiler" - that need to be updated to include "org.eclipse.jdt.core.compiler.batch" bundle and its source.

Ideally we could get that in 4.26 M1.

[stephan-herrmann]

• org.eclipse.jdt.core.compiler.batch.BatchCompiler is API, so if it's in jdt.core now, it must remain there.

Why? It can be re-exported from JDT core. Do I miss something?

No objection. My comment was only trying to argue that clients of jdt.core must be able to use BatchCompiler (vs. Jay's worry that we're stuffing jdt.core with classes that don't belong there). This doesn't affect the question whether jdt.core directly contains that class, or pulls it in (and re-exports) from a new ecj plugin.

[stephan-herrmann]

Note: I've used "org.eclipse.jdt.core.compiler.batch" name because that was used in MANIFEST.MF of the ecj.jar.

Frankly, I don't know who needs that plugin name. If we keep it, then publish-to-maven-central will have to continue to translate this into the GA org.eclipse.jdt:ecj. But since 'ecj' is an exceptional artifactName anyway, that translation is probably OK / unavoidable.

[jarthana]

I am not sure if we still use build_ecj.xml. Of course, the ecj.1 is being built into the ecj*.jar. So, that one is needed.

If I had to take a wild guess, could something be wrong with the version of the compiler.batch bundle? I see that we replace the placeholders with actual version on the fly from the "prepare-package" phase in the jdt.core's pom.xml.

[stephan-herrmann]

could something be wrong with the version of the compiler.batch bundle? I see that we replace the placeholders with actual version on the fly from the "prepare-package" phase in the jdt.core's pom.xml.

To check we're on the same page: This is about patching the version into org/eclipse/jdt/internal/compiler/batch/messages.properties, right? Which is for the benefit of java -jar ecj.jar -version.

Are you saying that part is still missing in #182? Or is your point about the confusion / duplication(?) between build_ecj.xml and pom.xml?

[iloveeclipse]

@laeubi : see my comment above where I was stuck last time.

I've just rebased on master, see #182 build state where maven build will cry for sure.

18:30:19  [ERROR] Cannot resolve project dependencies:
18:30:19  [ERROR]   Software being installed: org.eclipse.jdt.core 3.32.0.qualifier
18:30:19  [ERROR]   Missing requirement: org.eclipse.jdt.core 3.32.0.qualifier requires 'osgi.bundle; org.eclipse.jdt.core.compiler.batch 3.32.0' but it could not be found

The org.eclipse.jdt.core.compiler.batch is new bundle. I have no idea how to say maven it exists in same git repo but not on any remote or whenever maven searches for it.

[iloveeclipse]

I consider the main task to be fixed now.

Remains

• Add N&N entry (I will do that)
• send a mail to cross-projects list, I think it is worth to do in case someone hacked around JDT core/compiler
• Find & fix possible leftovers in build scripts or elsewhere => in dedicated bugs please.
• Possible regressions etc should also be handled by dedicated bugs.
• Anything else I forgot?

[iloveeclipse]

Please review N&N entry eclipse-platform/www.eclipse.org-eclipse-news#74.
Should we add something like this to release notes?

[stephan-herrmann]

Does anyone know, why github failed to link the follow-up #578 and eclipse-pde/eclipse.pde#419 (which has a fix in jdt.core, not pde: #592) ? Without those issues its impossible to fully understand the story of this refactoring.

[laeubi]

What do you mean by

github failed to link the follow-up

e.g. none of those seem to mention the issue directly (except now as you mentioned them here) and also the commit message do not have a reference, so how do you suppose github to know about this issue?

In general there are two options:

1. You mention the issue number or link in the PR/isuue
2. You have the issue number or link in the commit message

[stephan-herrmann]

What do you mean by

github failed to link the follow-up

e.g. none of those seem to mention the issue directly

I just saw #578 mentioning this #181 right in its description, likewise eclipse-pde/eclipse.pde#419 saying eclipse-jdt/eclipse.jdt.core#181 - still I found neither by scanning this issue.

[stephan-herrmann]

@iloveeclipse I tried to incorporate your changes in the wiki. Do you have a minute to review my changes?

In particular, I wasn't able to explain how jdtCompilerAdapter.jar is created in maven builds. Perhaps you can fill in this detail?

[iloveeclipse]

Will do. And yes, maven is a PITA for me.

[iloveeclipse]

@stephan-herrmann : please check. I hope this is a bit more clear now.

[stephan-herrmann]

@stephan-herrmann : please check. I hope this is a bit more clear now.

Thanks @iloveeclipse . This helps (and I still hope, I'll never have to edit pom.xml in this regard ...).

Still I added one change on top as I felt confusion between creation and use of jdtCompilerAdapter. Well, this happens in true dogfooding situations, like exporting jdt.core w/ jdtCompilerAdapter using jdt.core and jdtCompilerAdapter :)