Simplify the build of SWT-binaries and make it more Maven-oriented

[HannesWell]

The build of the SWT-binaries is a quite complex process and I have the impression that it is even more complex than it would have to be, from the fact that o.e.swt contains the sources for all platforms but the build produces a fragment for each platform that contains all, the common and platform specific compiled code and native libraries.
I'm just about the learn the entire build-process of SWT (because of #490), so please correct me if I'm wrong or missing something.

One thing that makes it IMHO complex is the usage of multiple different ant-scripts that copy sources multiple times, back and fourth and that it looks like that the java-code is compiled multiple times and that besides the standard main- and source-jar there is also a zip produced for each platform-specific fragment.
The zip seems to be the one that is presented in the SWT section of the I-build drops:
https://download.eclipse.org/eclipse/downloads/drops4/I20221209-1800/
Is it correct that the o.e.swt and platform specific swt fragments that end up in the eclipse-updates repos (like https://download.eclipse.org/eclipse/updates/4.27-I-builds/) are the regular main- and sources-jar of the corresponding maven project?

I wonder why the code is compiled twice? And from my understanding of the build-process, as described above, this means that the class-files in the jars that end up in the p2-repo are different ones than those presented as swt-zips in the drops.

In general I think that it would be good if the swt.binaries build would be more maven oriented and I think it could be just like:

1. For each platform-specific fragment copy the corresponding java source-files from o.e.swt to the src folder
2. Build each fragment the usual maven way
3. Assemble the swt-*.zip for each fragment, but without recompiling the java-files.

This would also include that all ant based launch configs or procedures are replaced by ones that include executing a Maven build.

A second bigger point that could be improved is where/when the native-libs in each platform-specific fragment are build.
At the moment this happens in https://ci.eclipse.org/releng/job/SWT-Increment_if_needed, which is triggered in the beginning of the daily releng/I-build job.
I wonder if it would be better to migrate that job to steps that run as part of the master-branch build of this repo? The build of the native libraries (if necessary) could even be done in the beginning of the master-build so that the following verification always runs on the native libs that match it. At the moment (as far as I understand) the master branch is build against the native libs build in the last I-build. If the native libs have changed the changed lib could be committed to the swt.binaries repo in the end if the master-build succeeded.
This would allow to get rid of the SWT-Increment_if_needed and if we would use https://www.jenkins.io/doc/book/pipeline/syntax/#parallel maybe even all the platform-specific jenkins job could be removed as well (but I have not yet checked all).

This would also allow to remove the enfored qualifier for the o.e.swt Plugin:

eclipse.platform.swt/bundles/org.eclipse.swt/pom.xml

Line 32 in 1d81184

<forceContextQualifier>v20221208-2302</forceContextQualifier>

It could then just use the qualifier based on the git-timestamp, just like most other Plugins do it. If that qualifier ahd changed it could then be set and committed to the o.e.swt.binaries/binaries-parent/pom.xml, together with the changed native-libs/binaries in the end of this repos master-build.
In the I-builds pipeline the Swt build input stage could then be removed entirely.

@akurtakov, @mickaelistria, what do you think?
Maybe @SDawley or @niraj-modi are you interested in that as well?

[akurtakov]

One thing that makes it IMHO complex is the usage of multiple different ant-scripts that copy sources multiple times, back and fourth and that it looks like that the java-code is compiled multiple times and that besides the standard main- and source-jar there is also a zip produced for each platform-specific fragment. The zip seems to be the one that is presented in the SWT section of the I-build drops: https://download.eclipse.org/eclipse/downloads/drops4/I20221209-1800/ Is it correct that the o.e.swt and platform specific swt fragments that end up in the eclipse-updates repos (like https://download.eclipse.org/eclipse/updates/4.27-I-builds/) are the regular main- and sources-jar of the corresponding maven project?

Main and sources jars of fragments is what ends up in p2 repos, org.eclipse.swt host has no sources jar produced at all as having sources like we have it in git would mean having e.g. multiple Text/Button/Combo.java files and such sources jar would be practically useless.

I wonder why the code is compiled twice? And from my understanding of the build-process, as described above, this means that the class-files in the jars that end up in the p2-repo are different ones than those presented as swt-zips in the drops.

History. Standalone jars were produced with Ant back in the days when Eclipse was not using Maven to build and migrating this had lower priority than other work.

In general I think that it would be good if the swt.binaries build would be more maven oriented and I think it could be just like:

1. For each platform-specific fragment copy the corresponding java source-files from o.e.swt to the src folder

2. Build each fragment the usual maven way

3. Assemble the swt-*.zip for each fragment, but without recompiling the java-files.

This sounds like very good plan.

This would also include that all ant based launch configs or procedures are replaced by ones that include executing a Maven build.

A second bigger point that could be improved is where/when the native-libs in each platform-specific fragment are build. At the moment this happens in https://ci.eclipse.org/releng/job/SWT-Increment_if_needed, which is triggered in the beginning of the daily releng/I-build job. I wonder if it would be better to migrate that job to steps that run as part of the master-branch build of this repo?

That might work and it will simplify testing for people as sometimes we have misalignment between native binaries and sources till this job runs.

The build of the native libraries (if necessary) could even be done in the beginning of the master-build so that the following verification always runs on the native libs that match it. At the moment (as far as I understand) the master branch is build against the native libs build in the last I-build. If the native libs have changed the changed lib could be committed to the swt.binaries repo in the end if the master-build succeeded. This would allow to get rid of the SWT-Increment_if_needed and if we would use https://www.jenkins.io/doc/book/pipeline/syntax/#parallel maybe even all the platform-specific jenkins job could be removed as well (but I have not yet checked all).

These can't be removed as all natives have to be run on these platform specific jobs as they are setup with correct versions of system compilers (Visual Studio, XCode, GCC - pinned at min version SWT has to support) together will all they header files needed. I can't think of a reliable way to get this proper (or even at all) but running natively.

This would also allow to remove the enfored qualifier for the o.e.swt Plugin:

eclipse.platform.swt/bundles/org.eclipse.swt/pom.xml

Line 32 in 1d81184

<forceContextQualifier>v20221208-2302</forceContextQualifier>

It could then just use the qualifier based on the git-timestamp, just like most other Plugins do it. If that qualifier ahd changed it could then be set and committed to the o.e.swt.binaries/binaries-parent/pom.xml, together with the changed native-libs/binaries in the end of this repos master-build.
In the I-builds pipeline the Swt build input stage could then be removed entirely.

I would say this should be multi step process and trying to remove this should be kept for after the previous one is completed.

@akurtakov, @mickaelistria, what do you think? Maybe @SDawley or @niraj-modi are you interested in that as well?

One other idea that never came high enough in the todo list is eclipse-platform/eclipse.platform.swt.binaries#2 (comment) . It would be nice if you consider this one too as it has the potential to merge the repos and thus simplify things too.

[laeubi]

Also probably related:

• Support compile code for each environment separately eclipse-tycho/tycho#632

[HannesWell]

Glad you like the idea.

The build of the native libraries (if necessary) could even be done in the beginning of the master-build so that the following verification always runs on the native libs that match it. At the moment (as far as I understand) the master branch is build against the native libs build in the last I-build. If the native libs have changed the changed lib could be committed to the swt.binaries repo in the end if the master-build succeeded. This would allow to get rid of the SWT-Increment_if_needed and if we would use https://www.jenkins.io/doc/book/pipeline/syntax/#parallel maybe even all the platform-specific jenkins job could be removed as well (but I have not yet checked all).

These can't be removed as all natives have to be run on these platform specific jobs as they are setup with correct versions of system compilers (Visual Studio, XCode, GCC - pinned at min version SWT has to support) together will all they header files needed. I can't think of a reliable way to get this proper (or even at all) but running natively.

For each Parallel-stage you can define a distinct Jenkins-agent to run on. So the native builds itself can still run on these specialized nodes but all of that can be controlled by the Jenkinsfile in this repo.

I would say this should be multi step process and trying to remove this should be kept for after the previous one is completed.

@akurtakov, @mickaelistria, what do you think? Maybe @SDawley or @niraj-modi are you interested in that as well?

One other idea that never came high enough in the todo list is eclipse-platform/eclipse.platform.swt.binaries#2 (comment) . It would be nice if you consider this one too as it has the potential to merge the repos and thus simplify things too.

Yes, that should definitely be a multi-step process. My plan is currently to do the following changes:

1. Built the SWT-natives as part of the verification-build/master-branch (and only commit on the mater)
2. Mavenize the SWT(-binaries) build as much as it is reasonably possible as outlined above and remove ant as much as reasonably possible.
3. Convert SWT's platform-specific fragments to jar-typed Maven projects whose Manifest is built with BND as described in Introduce OSGi capability to wire o.e.swt to its native fragments #490
4. Introduce a o.e.swt.native capability to wire o.e.swt and a suitable fragment at runtime to resolve Introduce OSGi capability to wire o.e.swt to its native fragments #490.

Then the build should be much simpler to understand/follow (because everything is in this or the swt.binaries repo) and we can then consider further improvements like eclipse-platform/eclipse.platform.swt.binaries#2 (comment).

[mickaelistria]

Overall, simplifying is always worth it, but I'm not sure about what and how things can be best executed here.
Do we need to keep the swt-*.zip ? It seems to me that people willing to use SWT should either download the artifacts (eg from p2 repo) or use Maven. So maybe we can get rid of those and that's 1 less trouble to consider.

[laeubi]

osgi/osgi#520 seems really interesting! However, SWT is supposed to work without OSGi so I don't think an OSGi-only solution is enough

You can create a JEP for this as well :-)

[HannesWell]

in maven you usually have to use: https://github.com/jzy3d/jzy3d-swt-example/blob/main/pom.xml#L74-L144

With my current plan for #490 o.e.swt will have 'platform specific dependencies' (created with corresponding profiles) like these built-in. If those dependencies also end up in the poms published to Maven-Central, users in Maven won't have to do that anymore themself and can just reference o.e.swt and will get the fragment for their platform pulled in as transitive dependency.

IMHO that would be an advantage.

[laeubi]

I'm just not sure of this will work or maven ignores defined profiles in "dependency poms" (maybe if you import it with import scope but I have never tried that).

[laeubi]

@HannesWell I have to correct me here, I have checked with the maven-devs and actually OS enabled profiles from a parent should work...

[HannesWell]

OK, great. Thanks for investigating.
Then I will try it and we will see it then really works.

[akurtakov]

I'm closing this discussion as the new build is already on master for some time now.