Simplify the build of SWT-binaries and make it more Maven-oriented

[HannesWell]

Discussed in #497

^{Originally posted by HannesWell December 11, 2022}
The build of the SWT-binaries is a quite complex process and I have the impression that it is even more complex than it would have to be, from the fact that o.e.swt contains the sources for all platforms but the build produces a fragment for each platform that contains all, the common and platform specific compiled code and native libraries.
I'm just about the learn the entire build-process of SWT (because of #490), so please correct me if I'm wrong or missing something.

One thing that makes it IMHO complex is the usage of multiple different ant-scripts that copy sources multiple times, back and fourth and that it looks like that the java-code is compiled multiple times and that besides the standard main- and source-jar there is also a zip produced for each platform-specific fragment.
The zip seems to be the one that is presented in the SWT section of the I-build drops:
https://download.eclipse.org/eclipse/downloads/drops4/I20221209-1800/
Is it correct that the o.e.swt and platform specific swt fragments that end up in the eclipse-updates repos (like https://download.eclipse.org/eclipse/updates/4.27-I-builds/) are the regular main- and sources-jar of the corresponding maven project?

I wonder why the code is compiled twice? And from my understanding of the build-process, as described above, this means that the class-files in the jars that end up in the p2-repo are different ones than those presented as swt-zips in the drops.

In general I think that it would be good if the swt.binaries build would be more maven oriented and I think it could be just like:

1. For each platform-specific fragment copy the corresponding java source-files from o.e.swt to the src folder
2. Build each fragment the usual maven way
3. Assemble the swt-*.zip for each fragment, but without recompiling the java-files.

This would also include that all ant based launch configs or procedures are replaced by ones that include executing a Maven build.

A second bigger point that could be improved is where/when the native-libs in each platform-specific fragment are build.
At the moment this happens in https://ci.eclipse.org/releng/job/SWT-Increment_if_needed, which is triggered in the beginning of the daily releng/I-build job.
I wonder if it would be better to migrate that job to steps that run as part of the master-branch build of this repo? The build of the native libraries (if necessary) could even be done in the beginning of the master-build so that the following verification always runs on the native libs that match it. At the moment (as far as I understand) the master branch is build against the native libs build in the last I-build. If the native libs have changed the changed lib could be committed to the swt.binaries repo in the end if the master-build succeeded.
This would allow to get rid of the SWT-Increment_if_needed and if we would use https://www.jenkins.io/doc/book/pipeline/syntax/#parallel maybe even all the platform-specific jenkins job could be removed as well (but I have not yet checked all).

This would also allow to remove the enfored qualifier for the o.e.swt Plugin:

eclipse.platform.swt/bundles/org.eclipse.swt/pom.xml

Line 32 in 1d81184

<forceContextQualifier>v20221208-2302</forceContextQualifier>

It could then just use the qualifier based on the git-timestamp, just like most other Plugins do it. If that qualifier ahd changed it could then be set and committed to the o.e.swt.binaries/binaries-parent/pom.xml, together with the changed native-libs/binaries in the end of this repos master-build.
In the I-builds pipeline the Swt build input stage could then be removed entirely.

@akurtakov, @mickaelistria, what do you think?
Maybe @SDawley or @niraj-modi are you interested in that as well?

[mickaelistria]

Is it correct that the o.e.swt and platform specific swt fragments that end up in the eclipse-updates repos (like https://download.eclipse.org/eclipse/updates/4.27-I-builds/) are the regular main- and sources-jar of the corresponding maven project?

I cannot tell for sure, but I hope so and in case it's not the case, it would be a good target to have it that way ;)

For each platform-specific fragment copy the corresponding java source-files from o.e.swt to the src folder
Build each fragment the usual maven way
Assemble the swt-*.zip for each fragment, but without recompiling the java-files.

Sounds good.

I wonder if it would be better to migrate that job to steps that run as part of the master-branch build of this repo?

At the moment, the master build is not meant to be a production build, it's only for testing. The artifact it does generate are not going through the same amount of testing and validation as I-Build, they're not as reliable.
However, having the master build rebuilding the binaries instead of consuming those from I-Build as you're trying with #514 definitely sounds like an improvement.

[HannesWell]

I wonder if it would be better to migrate that job to steps that run as part of the master-branch build of this repo?

At the moment, the master build is not meant to be a production build, it's only for testing. The artifact it does generate are not going through the same amount of testing and validation as I-Build, they're not as reliable. However, having the master build rebuilding the binaries instead of consuming those from I-Build as you're trying with #514 definitely sounds like an improvement.

At the moment the sources for o.e.swt are committed to this repo (eclipse.platform.swt) the regular way through PRs. In the verification builds of this repo only the binaries for gtk.linux.x86_64 are build from scratch and tested with the tests in this repo. The resulting binaries are not committed to eclipse.platform.swt.binaries.

In the beginning of each I-Build the Jenkins-Job releng/SWT-Increment_if_needed is triggered:
https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/c2c0b2e1f87c239d1d784d6c6094f85dd7401453/JenkinsJobs/Builds/I_build.groovy#L232-L236

If there are changes in o.e.swt, that require recompilation of the SWT-binaries, then this job triggers six other jobs, one to build the swt-binaries for each platform: https://ci.eclipse.org/releng/view/SWT%20Natives/
In that case the results build by those jobs are collected and then committed to the eclipse.platform.swt.binaries repo. The extensive testing on all platforms happens in the end of the I-build (similar to and together with all other components of the Eclipse platform TLPs).

The main goal of #514 is to perform compilation and committing of the SWT-binaries as part of the eclipse.platform.swt pipeline. This a first step of this issue to make the build of SWT as a whole simpler. But since every task performed in regards of changing the eclipse.platform.swt and swt.binaries is then done in that pipeline this make subsequent simplifications much simpler, because one does not have to consider other jobs defined elsewhere. And it makes all seven jobs in https://ci.eclipse.org/releng/view/SWT%20Natives/ obsolete (as far as I understood them).

We don't loose much in testing here, because at the moment the swt.binaries are build and committed in the very beginning of the I-builds and the tests are performed in the very end. So if there are test-failures the committed binary changes have to be reverted any way. #514 is just moving the binary compilation step from the beginning of the I-builds to the committing process of this repo (which btw. probably also allows to use git-timestamp based qualifier at least for o.e.swt). The extensive testing still happens at the end of the I-builds (just like for other eclipse TLP components as well).

If your main concern is to duplicate the agents of the releng JIPP/JIRO used to build SWT-binaries to the platform JIPP, we could consider to move the jobs of this repo simply from platform to the releng. While this would be a break the logical 'sorting'/'assignment' of those jobs it would simplify the management of the agents (but maybe would require more other work).
Personally I would choose to take the burdon to duplicate some agents in the platform JIRO.

[akurtakov]

What about running SWT pipeline on releng JIPP? That way we should be able to have all benefits you listed without the extra work.

[akurtakov]

One more reason to go for releng JIPP is the fact that it is configure to push to git repos which I'm not sure is done for Platform JIPP nor whether it's even desired.

[HannesWell]

What about running SWT pipeline on releng JIPP? That way we should be able to have all benefits you listed without the extra work.

Yes, that should work as well, I even suggested that already and would be fine with that :)

If your main concern is to duplicate the agents of the releng JIPP/JIRO used to build SWT-binaries to the platform JIPP, we could consider to move the jobs of this repo simply from platform to the releng. While this would be a break the logical 'sorting'/'assignment' of those jobs it would simplify the management of the agents (but maybe would require more other work).

There might be some extra configuration work necessary to move this repos pipeline to releng JIPP, but it is probably less then setting up the agents. The freeze-period check works without extra credentials/tpkens and tokens for the dash-licenses tools or alike are not necessary in this repo.
So yes from all I know and can see moving https://ci.eclipse.org/platform/job/eclipse.platform.swt/ (and maybe even https://ci.eclipse.org/platform/job/eclipse.platform.swt.binaries/) to the releng JIPP would be the simplest way.
@fredg02 do you know any obstacles?

One more reason to go for releng JIPP is the fact that it is configure to push to git repos which I'm not sure is done for Platform JIPP nor whether it's even desired.

That's a point I didn't consider yet, but yes it makes sense.
So moving to the releng JIPP seems like the best way.

[fredg02]

@fredg02 do you know any obstacles?

If the mentioned Platform SWT jobs do not deploy artifacts to Maven Central... then no.

[HannesWell]

@fredg02 do you know any obstacles?

If the mentioned Platform SWT jobs do not deploy artifacts to Maven Central... then no.

They don't. AFAIK SWT is deployed to Maven-Central together with the other parts of eclipse-platform via separate Jobs in the releng JIPP.

@akurtakov, @fredg02 so do we have consensus to move https://ci.eclipse.org/platform/job/eclipse.platform.swt/ to the releng JIPP? The binaries repo is not necessary as far as I can tell at the moment.

If yes, @fredg02 can you perform the move? Should I create a new helpdesk issue for that or can this be done under eclipsefdn/helpdesk#2448?
I can then create a PR to https://github.com/eclipse-cbi/jiro to add the required agent-labels for the releng JIPP as suggested by you in https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/2448.

[akurtakov]

Let's move it!

[fredg02]

If yes, @fredg02 can you perform the move? Should I create a new helpdesk issue for that or can this be done under eclipsefdn/helpdesk#2448?

I can move it, yes. I will actually copy the job and disable it on the platform JIPP until everything works as expected.

[fredg02]

=> https://ci.eclipse.org/releng/job/eclipse.platform.swt/

[HannesWell]

If yes, @fredg02 can you perform the move? Should I create a new helpdesk issue for that or can this be done under eclipsefdn/helpdesk#2448?

I can move it, yes. I will actually copy the job and disable it on the platform JIPP until everything works as expected.

Sounds great. Thank you! I will let you know once #514 is fully implemented and working.

[HannesWell]

=> https://ci.eclipse.org/releng/job/eclipse.platform.swt/

Thank you again! That was quick 😄

[HannesWell]

=> https://ci.eclipse.org/releng/job/eclipse.platform.swt/

I just noticed that all jobs expect the one for my PR #514 are currently starving waiting for an executor. I assume that is because currently the swt-pipeline uses a kubernetes agent:
https://github.com/eclipse-platform/eclipse.platform.swt/blob/master/Jenkinsfile#L9-L80

In #514 I changed that to be a agent labeled with centos-latest. Therefore I assume that the kubernetes agents cannot run at the releng Jenkins. I hope that I can continue with a centos-latest agent in #514 so that eventually the master (and all other PRs after a rebase) use that as well, but for the meantime and maintanance branches that should probably be enabled?

[HannesWell]

=> https://ci.eclipse.org/releng/job/eclipse.platform.swt/

I just noticed that all jobs expect the one for my PR #514 are currently starving waiting for an executor. I assume that is because currently the swt-pipeline uses a kubernetes agent: https://github.com/eclipse-platform/eclipse.platform.swt/blob/master/Jenkinsfile#L9-L80

@fredg02 the print-out of the build log e.g. of the swt-master is listed below. Could it be that the kubernetes agents request more resources than available?

Furthermore I noticed that needed mac agents is not completing its start-up.

Branch indexing
08:32:54 Connecting to https://api.github.com/ using GitHub bot (username/token)
Obtained Jenkinsfile from da2d2e7718eaa22442f94c2bcff893892147b7a4
[Pipeline] Start of Pipeline
[Pipeline] echo
[WARNING] label option is deprecated. To use a static pod template, use the 'inheritFrom' option.
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Created Pod: kubernetes releng/swtbuild-pod-5lfnk-k9c0n
Created Pod: kubernetes releng/swtbuild-pod-5lfnk-nm5js
Created Pod: kubernetes releng/swtbuild-pod-5lfnk-0pgk8
Still waiting to schedule task
All nodes of label ‘[swtbuild-pod](https://ci.eclipse.org/releng/label/swtbuild-pod/)’ are offline
Created Pod: kubernetes releng/swtbuild-pod-5lfnk-fmct8
Created Pod: kubernetes releng/swtbuild-pod-5lfnk-tlbc7
WARNING: Unable to create pod: kubernetes releng/swtbuild-pod-5lfnk-zc44v because kubernetes resource quota exceeded. 
Failure executing: POST at: https://10.30.0.1/api/v1/namespaces/releng/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "swtbuild-pod-5lfnk-zc44v" is forbidden: exceeded quota: jenkins-instance-quota, requested: limits.memory=4608Mi,requests.cpu=1100m, used: limits.memory=49664Mi,requests.cpu=14450m, limited: limits.memory=49664Mi,requests.cpu=14950m.
Retrying...

WARNING: Unable to create pod: kubernetes releng/swtbuild-pod-5lfnk-5pcvf because kubernetes resource quota exceeded. 
Failure executing: POST at: https://10.30.0.1/api/v1/namespaces/releng/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "swtbuild-pod-5lfnk-5pcvf" is forbidden: exceeded quota: jenkins-instance-quota, requested: limits.memory=4608Mi,requests.cpu=1100m, used: limits.memory=49664Mi,requests.cpu=14450m, limited: limits.memory=49664Mi,requests.cpu=14950m.
Retrying...

WARNING: Unable to create pod: kubernetes releng/swtbuild-pod-5lfnk-fspk0 because kubernetes resource quota exceeded. 
Failure executing: POST at: https://10.30.0.1/api/v1/namespaces/releng/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "swtbuild-pod-5lfnk-fspk0" is forbidden: exceeded quota: jenkins-instance-quota, requested: limits.memory=4608Mi,requests.cpu=1100m, used: limits.memory=49664Mi,requests.cpu=14450m, limited: limits.memory=49664Mi,requests.cpu=14950m.
Retrying...

[fredg02]

@fredg02 the print-out of the build log e.g. of the swt-master is listed below. Could it be that the kubernetes agents request more resources than available?

That's possible.

Furthermore I noticed that needed mac agents is not completing its start-up.

MacOS agent b9h15-macos10.15 went AWOL and had to be reprovisioned, see also https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/2454

[HannesWell]

@fredg02 the print-out of the build log e.g. of the swt-master is listed below. Could it be that the kubernetes agents request more resources than available?

That's possible.

Is there anything I can do to fix that?

Furthermore I noticed that needed mac agents is not completing its start-up.

MacOS agent b9h15-macos10.15 went AWOL and had to be reprovisioned, see also https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/2454

Thanks for that hint. But according to the releng Jenkins overview the mac agent is still launching (since yesterday evening).

[fredg02]

Is there anything I can do to fix that?

Demand less resources. :)

Thanks for that hint. But according to the releng Jenkins overview the mac agent is still launching (since yesterday evening).

I've tried to take it offline, but it kept reconnecting. Should be fixed now. I'm trying to get the agent back online today.

[HannesWell]

Is there anything I can do to fix that?

Demand less resources. :)

If that is sufficent to compile that SWT-natives for linux-x86_64 that's indeed the simplest solution. :)
Initially I was thinking about to raise the quorum, but now created #522 in an attempt to implement your suggestion.

Thanks for that hint. But according to the releng Jenkins overview the mac agent is still launching (since yesterday evening).

I've tried to take it offline, but it kept reconnecting. Should be fixed now. I'm trying to get the agent back online today.

Thank you!

[HannesWell]

@fredg02 besides the resources issue it looks like the eclipse.platform.swt job on the releng Jenkins is not notified about git-events in this repo. For example when I push a new commit to a PR or even open a new PR the commits/branches are not picked up and build by Jenkins by default and I have to trigger builds manually respectively have to scan the repository manually. Could it be that the webhooks are not configured correctly for the new Jenkins home of this git repos pipeline?

[fredg02]

I have added a webhook to the eclipse.platform.swt repo that triggers the Releng JIPP. Please test it.

[HannesWell]

I have added a webhook to the eclipse.platform.swt repo that triggers the Releng JIPP. Please test it.

Tested it with a new commit on a existing PR and that works well now. Thanks again. :)

[HannesWell]

The main work for this issue is now completed.

• Since the native binaries are build as part of SWT's Jenkins Pipeline (Perform SWT-natives build as part of the master/verification build #514) the seven SWT natives jobs in the Releng-Jenkins instance are obsolete and I will delete them soon:
  • Delete now obsolete Jobs in Releng Jenkins eclipse.platform.releng.aggregator#1060
• The build of SWT now doesn't use ANT-scripts anymore, only very few executions of the maven-antrun-plugin are left, but those don't require JavaScript (see also SWT build still requires Java 11 #626) and there is no ANT script left in org.eclipse.swt. All tasks encoded in the former scripts where either obsolete or have been migrated to Maven, the Jenkins pipeline (which uses Groovy) or to one JEP 330 single source Java script.
• Short after the native binaries have been moved into this repo with Merge SWT native binaries into this repository stored in Git LFS #956, I have also changed the version qualifier computation to happen on the fly during builds so that the native fragments have the same qualifier as the org.eclipse.swt host, that in turn uses the usual git-based qualifier computation of Tycho, which all together makes it obsolete to commit the enforced qualifiers each time something changes within org.eclipse.swt.
  • This means less noise in the git-history and less aborted master builds because the last commit was pushed by the releng-bot
  • Also only commits that really change the binaries are tagged

There are still a few minor things to stream-line or simply thus I leave this open until that is done, but since all ANT scripts are now gone I wanted to share this intermediate update.

[iloveeclipse]

@HannesWell : could you please update wiki https://github.com/eclipse-platform/eclipse.platform.swt/wiki/SWT-Native-Builds-on-Foundation ?

Especially I'm interested in "how can one build native SWT fragments locally".

[HannesWell]

@HannesWell : could you please update wiki https://github.com/eclipse-platform/eclipse.platform.swt/wiki/SWT-Native-Builds-on-Foundation ?

Yes I will do that soon.

Especially I'm interested in "how can one build native SWT fragments locally".

You can build them as part of a local SWT maven build by setting the native property to the platform to build (usually the running one):
mvn clean verify -Dnative=${ws}.${os}.${arch}

If you just want to build the natives as fast as possible using maven you can use:
mvn clean process-resources -pl org.eclipse.swt:org.eclipse.swt.${ws}.${os}.${arch} -am -Dnative=${ws}.${os}.${arch} -DskipTests

If you don't want to use maven running the following two commands from the directory of the project org.eclipse.swt to collect the native sources and build them for the desired platform as it is done in that antrun-plugin execution for the binaries:

eclipse.platform.swt/binaries/pom.xml

Lines 141 to 169 in bd81b48

	<property name="build_dir" value="${project.build.directory}/natives-build-temp"/>
	<exec executable="${java.home}/bin/java" dir="${swtMainProject}" failonerror="true">
	<arg line="-Dws=${ws} build-scripts/CollectSources.java -nativeSources '${build_dir}'"/>
	</exec>
	<property name="SWT_JAVA_HOME" value="${java.home}"/><!-- Not overwritten if already set, e.g. as CLI argument -->
	<echo>Compile SWT natives against headers and libraries from JDK: ${SWT_JAVA_HOME}</echo>
	<if><equals arg1="${ws}" arg2="win32" />
	<then>
	<exec dir="${build_dir}" executable="${build_dir}/build.bat" failonerror="true">
	<env key="SWT_JAVA_HOME" value="${SWT_JAVA_HOME}"/>
	<env key="OUTPUT_DIR" value="${project.basedir}"/>
	<arg line="${targets}"/>
	<arg line="clean"/>
	</exec>
	</then>
	<else>
	<property name="gtk_version" value="3.0" />
	<exec dir="${build_dir}" executable="sh" failonerror="true">
	<arg line="build.sh"/>
	<env key="SWT_JAVA_HOME" value="${SWT_JAVA_HOME}"/>
	<env key="OUTPUT_DIR" value="${project.basedir}"/>
	<env key="GTK_VERSION" value="${gtk_version}"/>
	<env key="MODEL" value="${arch}"/>
	<arg line="${targets}"/>
	<arg line="clean"/>
	</exec>
	</else>
	</if>
	</then>

The exact commands depend on the platform and maybe can also be simplified in the future. Therefore using maven is probably the simplest way.
This topic was also discussed: #973 (comment)

[HannesWell]

If you just want to build the natives as fast as possible using maven you can use:
mvn clean process-resources -pl org.eclipse.swt:org.eclipse.swt.${ws}.${os}.${arch} -am -Dnative=${ws}.${os}.${arch} -DskipTests

An even faster way and a Maven launch configuration that has it encoded is about to be added in #991.

[SyntevoAlex]

eclipse.platform.swt/binaries/pom.xml

Line 50 in 2378112

<id>build-natitve-fragment</id>

Typo: build-nati*t*ve-fragment
Maybe it makes sense to fix it before it becomes legacy code?

[HannesWell]

eclipse.platform.swt/binaries/pom.xml

Line 50 in 2378112

<id>build-natitve-fragment</id>

Typo: build-nati*t*ve-fragment Maybe it makes sense to fix it before it becomes legacy code?

That profile is not activated explicitly, only automatically where needed, so it would not be dramatic. Nevertheless thanks for spotting this, just created #1141 to fix it.

[fedejeanne]

If you just want to build the natives as fast as possible using maven you can use:
mvn clean process-resources -pl org.eclipse.swt:org.eclipse.swt.${ws}.${os}.${arch} -am -Dnative=${ws}.${os}.${arch} -DskipTests

An even faster way and a Maven launch configuration that has it encoded is about to be added in #991.

--> #1167