Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Native support for Eclipse SDKs on Windows Arm64 #577

Open
22 of 28 tasks
chirontt opened this issue Sep 19, 2022 · 41 comments
Open
22 of 28 tasks

Native support for Eclipse SDKs on Windows Arm64 #577

chirontt opened this issue Sep 19, 2022 · 41 comments
Labels
help wanted Extra attention is needed

Comments

@chirontt
Copy link
Contributor

chirontt commented Sep 19, 2022

Update: now that adding support for Windows on Arm64 is under way for various Eclipse repositories, here's the central TODO list to keep track of various tasks done and/or under way:


I'm trying to build the Eclipse SDKs for Windows 11 on Arm64 (WoA), using this repo. I hope this is the right repo to start with, which is the umbrella repo for all others.

The environment triplet I use for it is win32/win32/aarch64, following the existing win32/win32/x86_64 triplet as a guide.

After re-compiling many *.exe and *.dll native files for WoA in various repos (from existing C code mostly unchanged), and adding many new win32.aarch64 folders (most of them are duplicates of the existing win32.x86_64 ones), I've managed to build successfully this aggregator repo, and got the following new zip files in eclipse.platform.releng.tychoeclipsebuilder/eclipse.platform.repository/target/products (in addition to other supporting os/arch zip files there):

org.eclipse.platform.ide-win32.win32.aarch64.zip
org.eclipse.sdk.ide-win32.win32.aarch64.zip

After unzipping the org.eclipse.sdk.ide-win32.win32.aarch64.zip file, I discover that, in its eclipse/plugins folder, the following 6 bundles are missing:

org.eclipse.core.filesystem.win32.aarch64_<version>.jar
org.eclipse.core.net.win32.aarch64.source_<version>.jar
org.eclipse.core.net.win32.aarch64_<version>.jar
org.eclipse.core.resources.win32.aarch64_<version>.jar
org.eclipse.equinox.security.win32.aarch64.source_<version>.jar
org.eclipse.equinox.security.win32.aarch64_<version>.jar

When compared to the org.eclipse.sdk.ide-win32.win32.x86_64.zip distribution, those bundles (of relevant x86_64 arch) do exist there. So somehow they got missing from the final zip file for aarch64 during the build.

The missing bundles are crucial for the Eclipse SDK to properly start up, but I have no idea how they got missed out from the build artifacts. I think I must have skipped over some crucial file(s) somewhere which produced this missing result.

Is there a file somewhere in the repo to indicate which bundles to be included (in the plugins folder of the zip file) for each SDK?

PS. The missing bundles are actually produced in their own Maven modules during the build, in the eclipse.platform & equinox submodules. Once I manually add 4 of them (the other 2 are source bundles) to the plugins folder, and also add their missing entries to the configuration\org.eclipse.equinox.simpleconfigurator\bundles.info file, starting it up with eclipse.exe works nicely!

PPS. Especially and crucially for WoA, eclipse.exe can only start up after I add the -Dorg.osgi.framework.os.name=win32,win entry to the eclipse.ini file, to help it resolve the critical com.sun.jna bundle at runtime.

@merks
Copy link
Contributor

merks commented Sep 19, 2022

Did you update the features to include the new fragments?

image

@chirontt
Copy link
Contributor Author

@merks thanks for the tip. Your picture is worth more than a thousand words to me. Now I can build the Eclipse SDKs for WoA and they work!

Next is where/what is the repo to build the Eclipse IDE (which should include other tools like JGit/EGit, m2e, etc.) for WoA?

@merks
Copy link
Contributor

merks commented Sep 20, 2022

The repository used to build the IDE are composed from many projects. There's information about that here:

https://ci.eclipse.org/simrel/

This file has information about all the projects involved:

https://git.eclipse.org/c/simrel/org.eclipse.simrel.build.git/tree/simrel.aggran

Quite of a few of the project has Oomph setups that you can use to setup an an IDE; many of them you'd have to do manually.

@mickaelistria
Copy link
Contributor

I don't think you'll need to change many other projects. The very vast majority of them wouldn't have Platform specific artifacts, and I don't have the impression fixing those is an immediate requirement for you.
As @merks pointed, you might want to make the SimRel build capable of including the win32/aarch64 artifacts; adding a relevant line at https://git.eclipse.org/c/simrel/org.eclipse.simrel.build.git/tree/simrel.aggr#n74 . But SimRel references p2 repositories, so you'll need to tweak it to reference a p2 repository that contains the win32/aarch64 SDK, which might not be easy.
Then, from SimRel, the "Eclipse IDE" is build with https://git.eclipse.org/c/epp/org.eclipse.epp.packages.git/ ; there are some pom.xml and .product files in here you'll also need to modify for this new platform to be considered.

But this is a multi-step process. My advice would be that -if you're willing to make your work sustainable and to contribute it to Eclipse Project-, now that you have a build working, you find a way to get a win32/aarch64 machine contributed to the build farm and contribute your changes to the Eclipse Platform/SDK project to get this Platform supported. Going to "official" SDK and p2 repo will already be a major milestones and will make it easier to cascade later on to SimRel/EPP...

@chirontt
Copy link
Contributor Author

As to the problem with JNA bundle in WoA:

PPS. Especially and crucially for WoA, eclipse.exe can only start up after I add the -Dorg.osgi.framework.os.name=win32,win entry to the eclipse.ini file, to help it resolve the critical com.sun.jna bundle at runtime.

I've raised an issue with the JNA project: java-native-access/jna#1465

@chirontt
Copy link
Contributor Author

@mickaelistria My Arm64 box is an entry-level, slow box, suitable for some light, manual testing only. I had started building this repo with this Arm64 box, and it was a big exercise in patience. I eventually gave it up and switched to building on a Linux (x64) box instead. It takes around 1 hour to build this repo on the Linux box without running the tests (i.e. with -DskipTests=true). The Arm64 box becomes a test machine for some manual testing of the resulting SDKs. So I don't think it's a good addition to your build farm.

The good news is that you don't need an Arm64 box to build these SDKs for WoA. I can build and produce all SDKs for all os/arch (WoA included) with my Linux (x64) box, so your existing build farm can build the WoA distribution too, once my work on WoA are accepted in to the official repos. The recompiled binaries for Arm64 would be checked in to the repos like other existing binaries currently for other os/arch, so that the build wouldn't depend on any platform-specific tools to run.

From the look of it, building the Eclipse IDE for WoA is more than I can chew at the moment, so for now I will concentrate on submitting PRs for this WoA work to various repos (9 total, including this repo), to get the official Platform/SDK build started for WoA!

@mickaelistria
Copy link
Contributor

@chirontt OK, thanks for the details. So if you have things that you could already contribute, please do so and we'll see what we can merge easily. But at some point, to guarantee good enough quality, we'll need to have the SDK tests running automatically on a win32/aarch64 machine. Without it, then the project cannot prevent from potential regressions easily, and we basically cannot provide good enough support ot make it reliable; which IMO kind of question the value of hosting this specific code in the Eclipse project instead of eg in some fork of yours.

@chirontt
Copy link
Contributor Author

Can this aggregator build be run for Windows on Arm64 only, i.e. for win32/win32/aarch64 environment triplet only? How should it be configured?

If so, I would try to build just for this WoA platform on my Arm64 box, including running the tests, to see how it would fare in producing the SDKs for WoA. I can't give any promise at the moment, though.

BTW, I've started sending out the PRs for this WoA work.

@mickaelistria
Copy link
Contributor

Some of the CI issues such as the ones about not found bundles in https://ci.eclipse.org/equinox/job/equinox/job/PR-121/1/console are caused by the missing win32/win32/aarch64 declaration in the parent pom: https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/master/eclipse-platform-parent/pom.xml#L254 . Please try adding the environment there and if build is fine, submitting a PR for that.
If you want to try a local build skipping other environments, you can comment them out.

@akurtakov
Copy link
Member

We can have the bundles but can't include them in features/products until we have the builder to ensure that they are rebuild on source changes.

@chirontt
Copy link
Contributor Author

Sorry for the confusion in my partial submission of PRs. I ran out of time yesterday, and only managed to submit the PRs for eclipse-equinox repos.

Now I've done all other PRs for eclipse-platform repos, to complete this WoA submission to mainline.

My guess is that all these PRs need to go in together for a successful build, as they are inter-related, but you may have better idea about them.

@akurtakov
Copy link
Member

I'll repeat it:
We can not include native win32 aarch fragments in features unless we have a builder in our JIPPs allowing to rebuild them on e.g. source change in SWT. Until this happens we can only have sources that can't end up in deliverables.

@akurtakov
Copy link
Member

@daliasheasha You was interested in Eclipse on Windows AArch64. Maybe you can help with some builder?

@chirontt
Copy link
Contributor Author

@akurtakov I'll resubmit the PRs if you can show me what can be submitted now, and what should be submitted later when the build farm would have a builder available to (re)build Windows on Arm64 artifacts. This requires in-depth knowledge of the overall build requirements of all the repos, knowledge that I don't have.

Although I've withdrawn all the PRs, for the records, here was the list of them again that contains all of the WoA work in my forks:

eclipse-equinox/equinox#121
eclipse-equinox/equinox.binaries#2
eclipse-equinox/p2#147
eclipse-platform/eclipse.platform#189
eclipse-platform/eclipse.platform.releng#117
#595
eclipse-platform/eclipse.platform.swt#413
eclipse-platform/eclipse.platform.swt.binaries#16
eclipse-platform/eclipse.platform.ui#365

The branches in my forks for them are still available, to be resubmitted to mainline if needed. From my point of view, submitting them all together would lessen the chance of build breakage, but you may have better idea.

I know that all changes in those PRs would produce a successful Eclipse SDK for Windows on Arm64 platform, but separating them to do partial submits to mainline is a tough job for me, not to mention (God forbid!) build breakage. I'm willing to create new branches in my forks to submit the PRs again, if you can give me a precise list of what can be accepted, or wouldn't be needed (natives?), out of the above (withdrawn) PRs.

@chirontt
Copy link
Contributor Author

FYI, my fork of this umbrella repo, for Windows-on-Arm64 support, is now available:

https://github.com/chirontt/eclipse.platform.releng.aggregator

The fork includes links to other forked submodules, to make up my whole Windows-on-Arm64 work.

To build it is like normal (it can be built on any OS, be it Linux, Windows, or MacOS (haven't actually tried this myself)):

> git clone --recurse-submodules https://github.com/chirontt/eclipse.platform.releng.aggregator
> cd eclipse.platform.releng.aggregator
> mvn verify -DskipTests=true

The eventual build artifacts for all Eclipse SDKs of all os/arch will be produced in the eclipse.platform.releng.tychoeclipsebuilder/eclipse.platform.repository/target/products/ directory.

And the specific artifact of my interest would be org.eclipse.sdk.ide-win32.win32.aarch64.zip, which should be created in that directory. Note that, in order to run this artifact successfully on a WoA machine, you must apply the workaround I mentioned above, for Eclipse to successfully resolve the com.sun.jna bundle at startup.

I will try to keep this fork tracking the upstream repo closely, to keep it from becoming outdated, perhaps by merging it with upstream repos every weekend.

@akurtakov
Copy link
Member

I did one of them in eclipse-platform/eclipse.platform.ui#365 (review)

@akurtakov
Copy link
Member

@waynebeaton Maybe you can try foundation MS connections for having win/aarch64 machine on the build farm?

@SyntevoAlex
Copy link
Member

There's no need to have win/aarch64 machine to build bundles, see eclipse-platform/eclipse.platform.swt#413 (comment)

Without such machine, we won't be able to run tests against built binaries, but this is only a minor issue: it's unlikely to have any new bugs compared to x64 bundle.

@merks
Copy link
Contributor

merks commented Oct 7, 2022

I expect we'll want to be able to test that this stuff actually works. Also, if we want SimRel/EPP to deliver these things, will need a JustJ JRE for which we need to run jlink natively. If no one is willing to commit to providing hardware, what does that say about depth of the commitment?

@akurtakov
Copy link
Member

I said it already - it's fine to have sources for anyone willing to work on it. But it's not fine to ship any user binaries without being able to at least smoketest it.
We have went through such hoops with linux/[aarch64|ppc64le] - binaries produced were not even starting in real world until smoketesting infrastructure was present.

@karianna
Copy link

karianna commented Oct 7, 2022

Hi folks - Martijn from Microsoft (and Eclipse Adoptium) here. I'm wondering if some of the Windows on ARM hosts we have at Eclipse Adoptium could be utilized for this project? I'll check with George Adams when he's back from leave and see what we can do.

@akurtakov
Copy link
Member

@karianna so is there progress on your arm hosts being available for releng jipp?

@karianna
Copy link

An option is Project Volterra - https://blogs.windows.com/windowsdeveloper/2022/10/24/available-today-windows-dev-kit-2023-aka-project-volterra/

Not sure if we can do the Adoptium loan - CC @gdams

@akurtakov
Copy link
Member

I'm closing this one as there is clearly no interest from anyone to step up and do some work. If/when one does please reopen or create new issue.

@chirontt
Copy link
Contributor Author

chirontt commented Dec 8, 2023

FYI: In the meantime, I'm putting out unofficial Eclipse SDK builds for Windows on Arm64 in my fork.

I guess my fork will exist until Eclipse starts officially supporting Windows on Arm64 in its IDE releases.

@akurtakov
Copy link
Member

FYI: In the meantime, I'm putting out unofficial Eclipse SDK builds for Windows on Arm64 in my fork.

I guess my fork will exist until Eclipse starts officially supporting Windows on Arm64 in its IDE releases.

Thanks a lot, @chirontt ! That's the way forward as long as no one with resources cares enough to provide a single Windows Arm64 builder to the project to use.

@merks
Copy link
Contributor

merks commented Dec 13, 2023

Once the Platform provides it, most likely EPP will want to provide it and then JustJ will want to provide JREs for it, and it needs to run JLink natively. I really think we need some hardware. I would also think we'd want to run tests with it natively as well. I wonder that Adoptium wouldn't have such hardware. I'll ask around again...

@akurtakov
Copy link
Member

For the record - getting support in the whole stack is/should be the end goal. But if cross compiling allows getting support at Platform level now and there are people to do the work and test it- we should get that in Platform now and further work can/should be handled later. After all Linux AArch64 support has been in Platform for years before it made it to EPPs.
If/when someone starts doing the work for cross compiling and needs tracking issue for the various things to change feel free to reopen this one or create a new issue for that purpose.

@SyntevoAlex
Copy link
Member

@chirontt please prepare a PR and I could have a look at it.
Not soon, though. Maybe in a month.

@hmartinez82
Copy link

@merks @mickaelistria How can I contribute a a Windows on ARM CI runner to the project? At least for a few months until you guys get a permanent one from ARM (they are donating hardware for projects trying to be ported).

@SyntevoAlex I got here actually because I need SmartGit for Windows on ARM 😉

@laeubi
Copy link
Contributor

laeubi commented Feb 11, 2024

@hmartinez82 great to hear you want help out, to contribute a runner you need to open a ticket on the helpdesk here is an example:

there you should note that you want your runner be added to the Platform Releng Jenkins:

the IT team will then ask you for additional details in my case it was

  • server name
  • server description (e.g. OS, arch, etc)
  • server IP
  • user name (e.g. jenkins)

Additionally it would be good to mention how many concurrent build should be allowed on this node, you maybe also want to take a look here to get some idea what needs to be setup:
https://www.jenkins.io/doc/book/managing/nodes/#installing-a-jenkins-agent-on-windows

At least for a few months until you guys get a permanent one from ARM (they are donating hardware for projects trying to be ported).

If you have further information please share them here, or you can even mention them in the help-desk so the IT team probably can take the necessary actions!

@laeubi laeubi reopened this Feb 11, 2024
@laeubi laeubi added the help wanted Extra attention is needed label Feb 11, 2024
@laeubi laeubi pinned this issue Feb 11, 2024
@laeubi laeubi changed the title Building Eclipse SDKs for Windows on Arm64 [help wanted] Native support for Eclipse SDKs on Windows Arm64 Feb 11, 2024
@HannesWell
Copy link
Member

After unzipping the org.eclipse.sdk.ide-win32.win32.aarch64.zip file, I discover that, in its eclipse/plugins folder, the following 6 bundles are missing:

org.eclipse.core.filesystem.win32.aarch64_<version>.jar
org.eclipse.core.net.win32.aarch64.source_<version>.jar
org.eclipse.core.net.win32.aarch64_<version>.jar
org.eclipse.core.resources.win32.aarch64_<version>.jar
org.eclipse.equinox.security.win32.aarch64.source_<version>.jar
org.eclipse.equinox.security.win32.aarch64_<version>.jar

First of all the source bundles are created by the build, so you don't have to take care about them.
And org.eclipse.core.filesystem.win32.aarch64 should also not be necessary to make Eclipse work on WoA. The native fragment for Windows x86_64 is only to improve the performance of filesystem access but there is also a default handler available, which is IIRC 5~10% slower.
If org.eclipse.core.resources.win32.aarch64 is not available you only loose auto-refresh with native polling, so unless you change files outside Eclipse you shouldn't notice it is missing.
I haven't checked org.eclipse.core.net.win32.aarch64 and org.eclipse.equinox.security.win32.aarch64 yet, but maybe this one can also be saved with only some conveniences lost.

On the long run we can probably replace all the native binaries with Java's new Foreign Function and Memory Access API and solve all of this in pure Java without the need to compile for each architecture. Therefore I'm not sure its worth the effort to work towards that direction in the meantime.

The only definitive essentials I'm aware of are SWT and the Equinox launcher, but for both you have already created PRs 👍🏽

@HannesWell
Copy link
Member

Once the basic are available we should also set-up an I-build for win32.aarch64.

@HannesWell
Copy link
Member

With all the recent changes tonight's I-build should produce a complete Eclipse-SDK product for Windows on ARM.
@chirontt could you please test them and report back if they work fine?

@merks since all requirements on win32.aarch64 fragments are now strict again (in SWT it was optional for some time), could you test if the eclipse sdk contribution for SimRel still works?

Explicit win32.aarch64 fragments for the org.eclipse.core.filesystem and org.eclipse.core.resources plugins are the only things that have not yet been implemented, but they are not strictly necessary.

If no o.e.core.filesystem.win32.X fragment is available instead a Java NIO file API based file-system NativeHandler is used. The custom native filesystem-access implementations used by Eclipse were created to be faster, but I don't know if it has been reevaluated recently and if this is still the case respectivly worth the effort.
It would be great if you could benchmark with your custom builds?

If no o.e.core.resources.win32.X fragment is available only the installation of the Win32RefreshProvider fails, which leads to the following error log and the auto-refresh feature being not available:
grafik

Both should not block one from using Eclipse on Windows on ARM.
The auto-refresh feature would be nice to have, but to implement a native file-system access I think we should really have some numbers first if it is worth the effort. If a custom native file-system access is still significantly faster, in some time in the future it will probably be replaced by the Java-22 FFM API any ways and then will be automatically available on all architecturs for which JVMs are available.

@chirontt
Copy link
Contributor Author

chirontt commented May 9, 2024

I've downloaded the Eclipse SDK for WoA from the latest I-build and ran it on my WoA box; seems to work fine, but here's its screenshot in dark mode:

image

Note the lighted scrollbars, they require the fix in eclipse-platform/eclipse.platform.swt#1172 to be dark again.

Seems both the o.e.core.filesystem.win32.aarch64 and o.e.core.resources.win32.aarch64 fragments are not needed for the SDK to work.

How do I check the auto-refresh feature? like, checking the auto-sync of the IDE with changes in the file system?

If no o.e.core.filesystem.win32.X fragment is available instead a Java NIO file API based file-system NativeHandler is used. The custom native filesystem-access implementations used by Eclipse were created to be faster, but I don't know if it has been reevaluated recently and if this is still the case respectivly worth the effort.
It would be great if you could benchmark with your custom builds?

How should I benchmark this filesystem feature? Import a sizable project into the IDE and time it? (I have a sizable RCP project here to test it if needed.) Also, I have a slow box, so it may clearly show the difference as compared to a faster box. But I agree that we should remove these dependencies in native code as much as possible, given reasonably fast hardware available nowadays.

@laeubi
Copy link
Contributor

laeubi commented May 9, 2024

How do I check the auto-refresh feature?

Go to Preferences > Workspace > Refresh Using native Hooks or polling

What will happen is that you open expand a project folder, then you go to the Windows Explorer and create a new file there. If Refresh/Polling is activated the file will be shown in Eclipse after some (hopefully short) time, otherwhise you need to press F5 on the folder.

I think that's a part worth porting to aarch64 as there is no NIO Impl and the Java abstraction is rather complicated to be used while the native windows allows to watch a whole folder including its subfolders.

You can find some discussions here:

How should I benchmark this filesystem feature?

There was recently a benchmark and the NIO is slightly fasterslower (a few seconds over a very large count of operations) because NIO uses Exceptions to signal non existing files while we use in the native integration a boolean flag:

You can find some discussions here including the benchmarks:

I think its up to you if you think its worth to migrate this for aarch64 or not, or wait unless there are real complaints, inside an "usual" IDE setup I suspect one would never notice as there are other factors that slow down things considerably more than such micro optimization.

@merks
Copy link
Contributor

merks commented May 9, 2024

The https://download.eclipse.org/eclipse/updates/4.32-I-builds/I20240508-1800 aggregates:

image

@HannesWell
Copy link
Member

The https://download.eclipse.org/eclipse/updates/4.32-I-builds/I20240508-1800 aggregates:

Thank you Ed for checking.

I've downloaded the Eclipse SDK for WoA from the latest I-build and ran it on my WoA box; seems to work fine, but here's its screenshot in dark mode:
Note the lighted scrollbars, they require the fix in eclipse-platform/eclipse.platform.swt#1172 to be dark again.

That's great news. The SWT issue will hopefully be resolved soon.

If no o.e.core.filesystem.win32.X fragment is available instead a Java NIO file API based file-system NativeHandler is used. The custom native filesystem-access implementations used by Eclipse were created to be faster, but I don't know if it has been reevaluated recently and if this is still the case respectivly worth the effort.
It would be great if you could benchmark with your custom builds?

How should I benchmark this filesystem feature? Import a sizable project into the IDE and time it? (I have a sizable RCP project here to test it if needed.) Also, I have a slow box, so it may clearly show the difference as compared to a faster box. But I agree that we should remove these dependencies in native code as much as possible, given reasonably fast hardware available nowadays.

Yes that would one option. You could for example compare the runtime of an explicit refreseh of all projects in the workspace or of a full clean+rebuild both should include a lot of IO (altough I have to admit I have not yet checked for which operations exactly the native file-system handler is used). And yes using a slow computer will probably show any difference better.
The king-way would be to create a JMH benchmark case of the different org.eclipse.core.internal.filesystem.local.NativeHandler implementations, but that is probably more work.
But lets discuss that further in eclipse-platform/eclipse.platform#1311 (or its successors) lets keep this for high-level discussions.

I think all other questions have been answered by Christoph already.

With that I think support for WoA is ready to be announced on the cross-project mailing list so that more people can try it out and give feedback.

@iloveeclipse
Copy link
Member

There was recently a benchmark and the NIO is slightly faster (a few seconds over a very large count of operations) because NIO uses Exceptions to signal non existing files while we use in the native integration a boolean flag:

You can find some discussions here including the benchmarks:

Per occasion, we re-run "our" real life application benchmark on Eclipse 4.30 platform and we still see native Eclipse code performing faster as Java 21 NIO on xfs.

JNI vs Java 21 NIO 50x runs RH7 xfs, Xeon W-2145 RH9 xfs, Xeon w5-2465X
JNI eclipse.filesystem.useNatives=true 146.9 s (σ 3.9)  90.3 s (σ 1.7)
Java I/O eclipse.filesystem.useNatives=false 153.9 s (σ 3.7) 92.5 s (σ 1.5)
NIO vs JNI 104,8% 102,4%

I think its up to you if you think its worth to migrate this for aarch64 or not, or wait unless there are real complaints, inside an "usual" IDE setup I suspect one would never notice as there are other factors that slow down things considerably more than such micro optimization.

Correct.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

10 participants