Preparation work for 4.31 (2024-03) and open master for development

[MohananRahul]

This preparation work involves the following tasks. For previous issue please refer to-#1282

• Create R4_30_maintenance branch #1568
• Update R4_30 maintenance branch with release version for 4.30+ changes #1622
• Move 4.30-I builds to R4_30_maintenance branch #1569
• Update Parent pom and target sdk deployment jobs for R4_30_maintenance branch #1571
• Create new test jobs for 4.31
• Configure SWT build scripts for 4.31
• Splash Screen for 4.31 (2024-03)
• Create 4.31-I-builds repo
• POM and product version changes for 4.31 release #1579
• Update product version number to 4.31 across build scripts
• Move previous version to 4.30RC2 across build scripts
• Update version number in Mac's Eclipse.app for 4.31
• Disable the freeze report for 4.31
• Clean forceQualifierUpdate files for doc bundles
• Cleanup approved api list
• Update builds and repo cleanup scripts for 4.31
• Create new I-build job for 4.31 release https://ci.eclipse.org/releng/job/Builds/job/I-build-4.31/
• Update check composites script to verify 4.31 repositories
• Update Comparator repo and eclipse run repo to 4.31-I-builds repo
• Version bumps for 4.31 stream #1590
• Update previous release version to 4.30 GA across build scripts #1619
• Update build calendar for platform 4.31 release #1561
• Deploy ecj compiler from 4.30 GA and use it for 4.31 M1 build
• Update generic repos I-builds, Y-builds, P-builds to point to 4.31 repos

[HannesWell]

• Version bumps for 4.31 stream

@MohananRahul can you please make sure that for this release the prepareRelease workflow is used to carry out all the preparation work and version bumps in the sub repositories automatically:

• https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/master/.github/workflows/prepareRelease.yml

If this works this should automate a big portion of the work and therefore speed it up.

@laeubi since you created these workflows, can you please again give a summary what needs to be done or point to where it has been described? IIRC the workflows are triggered by closing the GH milestones?

[laeubi]

A while back I added a readme for the workflows here:

https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/master/.github/workflows/README.MD

so if anything is unclear or should be documented please add it there.

1. the prepareRelease.yml workflow is triggered by the creation of the GH Milestones and then creates a PR with increasing the version in the master pom what the is deployed to the snapshot repository so it can be found by other repositories right away. This is the most important part so we don't have failing verification builds!
2. In "regular" repositories the milestones then trigger updateRelease.yml workflow what set the parent version to the next release and consumes the one produced by this and tries for a while to bump the versions and then creates a PR, please note that last time there where some problems with access right of the different bots not sure if @fredg02 has fixed that but just in case watch out for such failures and report them.

Beside that it might that are other glitches, I once have created a workflow to automatically update the news here:
https://github.com/eclipse-platform/www.eclipse.org-eclipse/blob/master/.github/workflows/updateRelease.yml

this is to make the N&N automatically update but it has for sure not been tested after the move and based on shell scripts, so maybe one wants to make a test if the pathes are all still correct and the shell script works for the new layout.

[HannesWell]

1. the prepareRelease.yml workflow is triggered by the creation of the GH Milestones and then creates a PR with increasing the version in the master pom what the is deployed to the snapshot repository so it can be found by other repositories right away.

Wouldn't it be more suitable to trigger the workflow if the maintenance branch is created?
Because from that point on all work for the upcoming release (if there is any) goes to that branch, doesn't it?

This is the most important part so we don't have failing verification builds!

Agree on this. 👍🏽

@MohananRahul can you please confirm that you consider to use these workflows for that task?

[laeubi]

Wouldn't it be more suitable to trigger the workflow if the maintenance branch is created?
Because from that point on all work for the upcoming release (if there is any) goes to that branch, doesn't it?

The workflow is for the master not for the maintenance branch. Also the maintenance branch name itself does not contain enough information but only the milestones does (e.g. end-date).

[HannesWell]

Wouldn't it be more suitable to trigger the workflow if the maintenance branch is created?
Because from that point on all work for the upcoming release (if there is any) goes to that branch, doesn't it?

The workflow is for the master not for the maintenance branch. Also the maintenance branch name itself does not contain enough information but only the milestones does (e.g. end-date).

Sure it is for the master. Creating the maintenance should only be the trigger to create the necessary version bumps for the master (at the moment the maintenance branch is created that branch and the master usually point to the same commit any ways, so it is mainly a question against which branch to create the PR).

Why is the end-date relevant?

[laeubi]

If you look at https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/master/eclipse-platform-parent/pom.xml it has two "variables" for each release:

1. releaseName
2. releaseNumberSDK

where only the second might be reconstructed from a branch name easily.
The milestone on the other hand has a name, a date and it has a description that says what it is used for:
https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/milestone/43

and that's what is extracted by the workflow here
https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/master/.github/workflows/prepareRelease.yml

So if one wants to automate that more, a milestone should better trigger the maintenance and what else is desired, if something goes wrong its also easier to recreate a milestone, also milestone have states, so closing of a milestone can even trigger more actions all this is not possible for a branch where only the name has a quite weak meaning...

Please also note that the prepare workflow do NOT bump any versions, the updateRelease workflow does that!

[laeubi]

The prepare workflow runs successful here

• Prepare Release 4.31 #1577

but for example PDE failed

• https://github.com/eclipse-pde/eclipse.pde/actions/runs/6993464045/job/19026166842

this is due usage of (vanished) SNAPSHOT version of Tycho, fixed here:

• Update updateRelease.yml to use released Tycho version #1578

I'll now merge this and try to trigger the workflow for PDE again to verify it is working, for this I need to delete the Milestone once and will recreate it afterwards.

[laeubi]

PDE Milestone is recreated and workflow is running now:

• https://github.com/eclipse-pde/eclipse.pde/actions/runs/6993748017/job/19026735725

PR was successful created:

• Update for release 4.31 eclipse-pde/eclipse.pde#935

will now do the same for P2/Equinox...

[MohananRahul]

@laeubi where I need to check update release workflow run ?

[laeubi]

P2 Workflow is running:

• https://github.com/eclipse-equinox/p2/actions/runs/6993766906

PR is here:

• Update for release 4.31 eclipse-equinox/p2#387

Equinox has missing the workflow at all, created:

• Add update release workflow eclipse-equinox/equinox#408

Equinox Workflow is running:

• https://github.com/eclipse-equinox/equinox/actions/runs/6993787119

But it seems it uses non trivial ant stuff/setup so automatic bump does not work, I would recommend to do it manually there for this release and investigate this independently (but we can wait untill the workflow is done I'm not sure if it will create PR with things that worked anyways ...), will now do Platform/UI.

@laeubi where I need to check update release workflow run ?

You can go to the repository (e.g. PDE) then click on the Actions Tab and you will (if setup) see Update to next release entry (for PDE one failed and now one with success).

[laeubi]

Platform Workflow run now:

• https://github.com/eclipse-platform/eclipse.platform/actions/runs/6993839632

PR:

• Update for release 4.31 eclipse.platform#884

Platform UI Workflow runs now:

• https://github.com/eclipse-platform/eclipse.platform.ui/actions/runs/6993846643

PR:

• Update for release 4.31 eclipse.platform.ui#1327

@MohananRahul can you add a task for the next preparation todo list that we check to update Maven/Tycho versions for these workflows? I think we should (maybe at RC1) simply upgrade to maven/Tycho used in regular platform builds there.

[laeubi]

For the news: https://github.com/eclipse-platform/www.eclipse.org-eclipse it seems actions are not enabled for that repository, @akurtakov @vogella can any of you enable that directly or create a help desk ticket?

@MohananRahul It seems the old milestones are not closed for that repository: https://github.com/eclipse-platform/www.eclipse.org-eclipse/milestones

[MohananRahul]

Platform Workflow run now:

• https://github.com/eclipse-platform/eclipse.platform/actions/runs/6993839632

PR:

• Update for release 4.31 eclipse.platform#884

Platform UI Workflow runs now:

• https://github.com/eclipse-platform/eclipse.platform.ui/actions/runs/6993846643

PR:

• Update for release 4.31 eclipse.platform.ui#1327

@MohananRahul can you add a task for the next preparation todo list that we check to update Maven/Tycho versions for these workflows? I think we should (maybe at RC1) simply upgrade to maven/Tycho used in regular platform builds there.

es Sure, and will update release.md, for remaining repo will create bump manually.

[laeubi]

JDT is not using the workflows and I have no committer power there, so if anyone is interested to enable this for JDT here are the steps:

• Make sure Actions are enable (e.g. Action tab is showing in the repo)
• add the release update workflow to the repo like in Add update release workflow eclipse-equinox/equinox#408
• delete the 4.31 milestone
• create it again with these data:

4.31
13.03.2024
4.31 Release

After that the workflow should run and create a PR if everything works fine.

[laeubi]

SWT, due to its special structure also requires manual bumps currently, maybe also SWT binaries ... once @HannesWell has enabled LFS and we merged binaries with main swt repo I can propose a new Tycho based build to maybe support this in the future.

[laeubi]

@MohananRahul if you notice anything else that could/should be automated after these workflows have run please add an issue in this repository so we can track it and hopefully tackle for next release.

[laeubi]

This might be merged before we do any further changes to the doc bundles:

• Remove PDE user docs from aggregator #1575

[akurtakov]

For the news: https://github.com/eclipse-platform/www.eclipse.org-eclipse it seems actions are not enabled for that repository, @akurtakov @vogella can any of you enable that directly or create a help desk ticket?

@MohananRahul It seems the old milestones are not closed for that repository: https://github.com/eclipse-platform/www.eclipse.org-eclipse/milestones

I think we are supposed to provide PRs for https://github.com/eclipse-platform/.eclipsefdn/blob/main/otterdog/eclipse-platform.jsonnet for such changes which webmaster should approve after that. I'm still not sure whether

 workflows+: {
        default_workflow_permissions: "write",
      },

is enough though.

[HannesWell]

It is great to see that the workflows are running 🎉

If you look at https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/master/eclipse-platform-parent/pom.xml it has two "variables" for each release:

1. `releaseName`

2. `releaseNumberSDK`

where only the second might be reconstructed from a branch name easily.

Ah, yes. I missed the releaseName, this is indeed easier to get from the milestone.

So if one wants to automate that more, a milestone should better trigger the maintenance and what else is desired, if something goes wrong its also easier to recreate a milestone, also milestone have states, so closing of a milestone can even trigger more actions all this is not possible for a branch where only the name has a quite weak meaning...

Agree.
In general I wonder if it would be more clear/simpler if the updateRelease and prepareRelease workflows would be triggered directly through the GitHub API by the job that also creates the Milestones?
One can also pass arguments in such API calls:

• https://docs.github.com/en/rest/actions/workflows?apiVersion=2022-11-28#create-a-workflow-dispatch-event
• https://www.enekoalonso.com/articles/issue-7

This would avoid the need to extra the values from a milestone due date.
IIRC the Milestone creation is already automated in by a (private?) Jenkins job, that calls the GH API, isn't it?
The updateRelease workflow could then also be enhanced to create the maintenance branches at the same time in the repo it runs. If I understand it correctly this is currently done manually?

I think with that all the RelEng work distributed over many repositories would already be automated (expect for the special ones like SWT).

[laeubi]

The advantage of Milestones is that they can be created/managed without special permissions by the committers, of course one can also use the API for many tasks. But as you see it has its drawbacks (e.g. special security considerations), so one better can think on how to automate things with regular workflows if possible. e.g. a maintenance branch might be created by closing a milestone ...

[HannesWell]

The advantage of Milestones is that they can be created/managed without special permissions by the committers, of course one can also use the API for many tasks. But as you see it has its drawbacks (e.g. special security considerations), so one better can think on how to automate things with regular workflows if possible. e.g. a maintenance branch might be created by closing a milestone ...

This is all right. But at the same time triggering workflows by closing/creating milestones can also be surprising.
From this POV I would say that calling the workflow directly with explicit parameters would IMHO be more clear.

And there is already an GH-API call (I assume) to create the milestones from Jenkins and I think we won't get rid of that, even if the workflow would run entirly at GH, since (AFAIK) you cannot simply call workflows in other repositories (especially in different organizations).
It could also be done the other way round, that the updateRelease workflow creates the milestones too, so that milestone and maintance-branch creation and the PR for version updates is all done in one workflow run, which is triggered for each SDK repo from Jenkins. That workflow run could then get all the required parameters explicitly?
Maybe it would be good to have the Jenkins job as Jenkins file in the releng.aggr repo, too? And only restrict the execution right instead of hiding it completely?

But however it is done, I think it is better to have this automated than doing it manually. :)

[laeubi]

Maybe it would be good to have the Jenkins job as Jenkins file in the releng.aggr repo, too? And only restrict the execution right instead of hiding it completely?

The Job contains the credentials, so it is not that a good idea, anyways this is part of the infra/releng team and not automation, it does not really matter and bring much value to overcomplicate this now where we have a lot of other work to do, especially this time where things went wrong it was incredible useful to have a (manual) way to trigger the workflow again instead of failing completely.

So maybe one can have a workflow that creates the milestones and is then trigger somehow externally but putting everything in one big workflow has it shortcomings.

[akurtakov]

@laeubi Following the instructions for jdt repo failed https://github.com/eclipse-jdt/eclipse.jdt/actions/runs/7001093546 .
I see that @noopur2507 tried it for jdt.ui and got same issue https://github.com/eclipse-jdt/eclipse.jdt.ui/actions/runs/7001058019 . We seem to be missing some config but I'm not sure whether it's in the workflow or the repo or even the org.

[MohananRahul]

@laeubi Following the instructions for jdt repo failed https://github.com/eclipse-jdt/eclipse.jdt/actions/runs/7001093546 . I see that @noopur2507 tried it for jdt.ui and got same issue https://github.com/eclipse-jdt/eclipse.jdt.ui/actions/runs/7001058019 . We seem to be missing some config but I'm not sure whether it's in the workflow or the repo or even the org.

Yes , we tried to create workflow. Seems required permission to create PR, branch got created
https://github.com/eclipse-jdt/eclipse.jdt.ui/actions/runs/7001058019/job/19042596690#step:8:145

[laeubi]

It needs to be enabled on the organization see for example:

https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/2845

[fedejeanne]

fedejeanne closed this as completed in fedejeanne/eclipse.platform.swt@6a83d0c 4 minutes ago

Is this task really done?

How did this happen? I didn't close anything

Closed due to "fixes" mentioned in PR.

It's still odd that it says I closed it since I had nothing to do with the original commit... :-\

[laeubi]

You merged it to master ...

[iloveeclipse]

Build is unstable: https://download.eclipse.org/eclipse/downloads/drops4/I20231127-0750/

https://download.eclipse.org/eclipse/downloads/drops4/I20231127-0750/buildlogs/comparatorlogs/buildtimeComparatorUnanticipated.log.txt

Comparator differences from current build
	/home/jenkins/agent/workspace/Builds/I-build-4.31/eclipse.platform.releng.aggregator/eclipse.platform.releng.aggregator/cje-production/siteDir/eclipse/downloads/drops4/I20231127-0750
compared to reference repo at 
	https://download.eclipse.org/eclipse/updates/4.30-I-builds

1.  eclipse.platform/team/bundles/org.eclipse.team.core/.polyglot.META-INF
   no-classifier: different
      org/eclipse/team/internal/core/subscribers/BatchingLock$IFlushOperation.class: different
    The main artifact has been replaced with the baseline version.
    The following attached artifacts have been replaced with the baseline version: [sources]

2.  eclipse.jdt.ui/org.eclipse.jdt.ui/pom.xml
   no-classifier: different
      org/eclipse/jdt/internal/ui/preferences/FilteredPreferenceTree.class: different
      org/eclipse/jdt/internal/ui/preferences/formatter/ModifyDialog$PreferenceBuilder.class: different
    The main artifact has been replaced with the baseline version.
    The following attached artifacts have been replaced with the baseline version: [sources]

[akurtakov]

@iloveeclipse would you please look into org/eclipse/jdt/internal/ui/preferences/formatter/ModifyDialog$PreferenceBuilder.class ? It's a real bytecode change while the FilteredPreferenceTree is lambda order bug.
The team one is also a "real" change.

[fedejeanne]

You merged it to master ...

Nope, niraj-modi did (eclipse-platform/eclipse.platform.swt#907 (review)). It wasn't even my branch (https://github.com/deepika-u/eclipse.platform.swt/tree/Configure_SWT_build_scripts_for_4.31)

[laeubi]

The I have no clue :-D
Github references a commit from your repository though ...

[iloveeclipse]

The change on FilteredPreferenceTree is a usual problem with lambda visit/generation order.
Should be fixed by "touching" org.eclipse.jdt.ui.

However, I have no explanation for the difference below yet.

diff BatchingLock\$IFlushOperation.class-baseline BatchingLock\$IFlushOperation.class-build
7a8,9
>   // access flags 0x9
>   public static INNERCLASS org/eclipse/team/internal/core/subscribers/BatchingLock$ThreadInfo org/eclipse/team/internal/core/subscribers/BatchingLock ThreadInfo

The "changed" IFlushOperation class has an additional reference to innerclass ThreadInfo, the old one doesn't.
Compiling code with IDE (4.30 RC2) I also don't see second innerclass reference.

It could be related to the wrongly merged eclipse-jdt/eclipse.jdt.core#1626 PR (which is not in the RC2 build but on master!), but I also don't see the extra innerclass using the unstable build itself.

Also I don't see the second reference by opening class from org.eclipse.team.core_3.10.200.v20231106-1240.jar.

@srikanth-sankaran , @stephan-herrmann : may I ask you to look into the difference?

Expected version (also same in the old/newIDE):

// class version 61.0 (61)
// access flags 0x601
public abstract interface org/eclipse/team/internal/core/subscribers/BatchingLock$IFlushOperation {

  NESTHOST org/eclipse/team/internal/core/subscribers/BatchingLock
  // access flags 0x609
  public static abstract INNERCLASS org/eclipse/team/internal/core/subscribers/BatchingLock$IFlushOperation org/eclipse/team/internal/core/subscribers/BatchingLock IFlushOperation

  // access flags 0x401
  public abstract flush(Lorg/eclipse/team/internal/core/subscribers/BatchingLock$ThreadInfo;Lorg/eclipse/core/runtime/IProgressMonitor;)V throws org/eclipse/team/core/TeamException 
}

"new" version (seen on unstable build only)

// class version 61.0 (61)
// access flags 0x601
public abstract interface org/eclipse/team/internal/core/subscribers/BatchingLock$IFlushOperation {

  NESTHOST org/eclipse/team/internal/core/subscribers/BatchingLock
  // access flags 0x609
  public static abstract INNERCLASS org/eclipse/team/internal/core/subscribers/BatchingLock$IFlushOperation org/eclipse/team/internal/core/subscribers/BatchingLock IFlushOperation
  // access flags 0x9
  public static INNERCLASS org/eclipse/team/internal/core/subscribers/BatchingLock$ThreadInfo org/eclipse/team/internal/core/subscribers/BatchingLock ThreadInfo

  // access flags 0x401
  public abstract flush(Lorg/eclipse/team/internal/core/subscribers/BatchingLock$ThreadInfo;Lorg/eclipse/core/runtime/IProgressMonitor;)V throws org/eclipse/team/core/TeamException 
}

The logs/diffs are:

• https://download.eclipse.org/eclipse/downloads/drops4/I20231127-0750/buildlogs.php
• https://download.eclipse.org/eclipse/downloads/drops4/I20231127-0750/buildlogs/comparatorlogs/artifactcomparisons.zip

[iloveeclipse]

OK, that's not compiler issue. @srikanth-sankaran , @stephan-herrmann: please sorry for the noise, and ignore my previous comment here.

Edit: this below was wrong :-(
The problem appears if I revert org.eclipse.team.internal.core.subscribers.BatchingLock to the previous version.
So it looks like the version used to check baseline for org.eclipse.team.core uses wrong bundle version.

I guess that we compare wrong repositories state here. We seem to compare latest state from the "team" repo with latest state from the "platform" repo into this project was moved via eclipse-platform/eclipse.platform#77.

[iloveeclipse]

OK, it is compiler issue, and it is a difference between clean vs incremental compile, so basically similar to the lambda issue but not same.

If I clean compile org.eclipse.team.core, I see "expected" version. If I add an empty line in the BatchingLock and save, the "unexpected" version is generated.

I will create a JDT ticket, but for now we probably can try to "touch" org.eclipse.team.core.

Thinking a bit more, I wonder why do we see such "order" issues on branch changes? Do we compile incrementally in most cases and "clean" compile happens on new releases only?

[iloveeclipse]

Created eclipse-jdt/eclipse.jdt.core#1631

[akurtakov]

@iloveeclipse I see that jdt.ui bundle has been touched but is this the correct thing to do now? Should we wait for jdt.core analysis?

[iloveeclipse]

@iloveeclipse I see that jdt.ui bundle has been touched but is this the correct thing to do now? Should we wait for jdt.core analysis?

Looking at the compiler behavior I assume it falls into same category as lambda numbering problem - dependency on compilation order or scope. Therefore if "touching" bundle workarounds the problem for now, I think it is OK to go with that.

[akurtakov]

Looking at the compiler behavior I assume it falls into same category as lambda numbering problem - dependency on compilation order or scope. Therefore if "touching" bundle workarounds the problem for now, I think it is OK to go with that.

OK, hopefully we don't find more "ordering" issues as we may have to disable comparator checks entirely if this becomes common.

[akurtakov]

So there is no java anymore on download.eclipse.org (https://ci.eclipse.org/releng/job/akurtakov%20test/5/console) which made our scripts to fail at adding in to composite sites(https://ci.eclipse.org/releng/job/Builds/job/I-build-4.31/4/console).
I've opened to get info from webmaster what/how/when should we do https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/4001 .
In general downloading eclipse on download.eclipse.org in order to modify an xml file is wrong and risky in every possible way so redoing how composites are managed is highly overdue - if someone can spend the time on that please say so!

[MohananRahul]

-Deploy ecj compiler from 4.30 GA and use it for 4.31 M1 build : no change in ecj version 3.36.0.v20231114-0937

All Subtask Completed