Merge swt.binaries repo in the swt one

[iloveeclipse]

There was a good question asked on eclipse-platform/eclipse.platform#7 - why don't we use git LFS for SWT binaries? As of today, SWT is about 250 MB and it seems that the github provides 1 GB free bandwidth for git LFS repos, so it should be enough.

The benefit is surely that we don't need to synchronize two repos and have platform code & binaries in one, plus much faster clone.

[nnemkin]

Is the SWT binaries repo really necessary? What's the harm of building the natives on demand? Github provides build machines for the three main platforms, right? I was hoping to radically simplify SWT build process after the Github migration.

Also, if we combine SWT repos, we could attach platform specific source to the platform specific fragments and leave the main bundle empty. Physically, the source might stay in o.e.swt, but logically every binary project like o.e.swt.win32.win32_x86_64 would contain both the binaries and the source. No more .classpath switcheroo and it would be possible to work on all platforms from the same workspace simultaneously.

[iloveeclipse]

If you bisect SWT bugs, you won't (and probably can't) build on every commit. Building natives locally is a non trivial task (requires extra dependencies to be installed with root rights etc).

[nnemkin]

You can't bisect across two repos anyway, can you? TBH I never tried to bisect SWT bugs.

We can make the natives build almost a trivial task, but dependencies are of course unavoidable. JDK is a dependency too, why single out native code?

[iloveeclipse]

You can't bisect across two repos anyway, can you?

It is not the question of "can" but it is a "must have" because if you bisect SWT issues your native library version is essential to be able to start anything. And yes, I've bisected two repos and that is NOT FUN, because you have to look which commit corresponds to which native commit. Having both the code & binaries in LFS and in one repo would simplify that A LOT.

why single out native code

Because a "simple" java compile with maven doesn't need to install extra gtk/win native packages etc, and in order to do that you must be able to run tasks as a root / have access to the required packages at all. Once you have setup that, it's fine, but it is not trivial to setup. Most people don't even need that at all if they fix something in simple Java code.

[vogella]

Adding build artifacts to a Git repo also sounds wrong to me. We can publish the artifacts somewhere else to help with searching for errors. WDYT @akurtakov

[akurtakov]

There have been already a discussion with @sravanlakkimsetti about that with potentially just using something like downloads.eclipse.org/eclipse/swt-natives/4567a/win32/ and having maven/ant download them.
If Git LFS could be used in an easier way - I'm all for that. The repository o.e.platform.swt.binaries is just an abomination no one dared to start the fight with. Thanks @iloveeclipse for starting this discussion.
@nnemkin let's keep the issue dedicated at getting rid of the binaries repo and merging both as what you speak of would be nice but has this one as a strong prereq.

[sravanlakkimsetti]

Corporates rebuild older versions of eclipse including SWT to release eclipse based products. Building on demand is not exactly possible as many of the dependencies would not be available.

One solution is to store the built libraries in a separate repo(that is the current solution).

Regarding git LFS, The individual libraries are not that big to start with less than 1 MB. I am not sure we are qualified for the LFS here.

Adding build artifacts to a Git repo also sounds wrong to me. We can publish the artifacts somewhere else to help with searching for errors. WDYT @akurtakov

What we are doing is not wrong. These are intermediate files used for the build purposes. I feel most of the thought process is going towards building master. for master we will have machines and other dependencies available for the build purposes. But that cannot be said when we want to build older release.

For example IBM builds 4.23, 4.19, 4,15 regularly. By removing these artifacts you are forcing the end users to maintain native build infrastructure for different versions. A lot of duplicate effort and it turn into cost to the end users.

We will definitely need to have the binaries stored. Where is going to be a question. Also we should make building aggregator simpler. Before triggering a build we should not fetch artifacts by some other means.

Same problem exists for team, equinox.framework, filesystem components as well.

[laeubi]

So probably instead of check them into the git, simply deploy them as an additional attached artifact along with the maven deployment?

[iloveeclipse]

So probably instead of check them into the git, simply deploy them as an additional attached artifact along with the maven deployment?

This is what git LFS seamlessly does for you without any 3rd party tool chain.

The individual libraries are not that big to start with less than 1 MB. I am not sure we are qualified for the LFS here.

The number of files in history does it. Cloning SWT binaries will move over wire ~1 GB just because every binary file is in the history. The sources itself are minimal, and with git LFS only last native libraries version will be needed to copy from server, not all of them.

[nnemkin]

@sravanlakkimsetti
Yeah, without a binaries repo, rebuilding historical releases will get more complicated, that's a real problem. But why are they rebuilding historical releases at all? Why not build them once and store in a maven/p2 repo forever?

[akurtakov]

We will definitely need to have the binaries stored. Where is going to be a question. Also we should make building aggregator simpler. Before triggering a build we should not fetch artifacts by some other means.

I believe we are all in agreement that for everyone but seasoned SWT developer having the native bits downloadable is beneficial. But this comes at a price - almost every release we have one broken build by bump of version of the host without doing so for the fragments. Not to mention other missed opportunities to simplify things. So the question is how do we organize a change in order to get more benefits than complications?

Same problem exists for team, equinox.framework, filesystem components as well.

One step at a time. Let's keep these discussions for other places. If we fix it in one place we will have better idea how to fix it in the other one. Btw, the team one is gone now after Linux moved to JNA 2 releases ago and Windows did so in master (https://bugs.eclipse.org/bugs/show_bug.cgi?id=578341). So we are getting there although the solution is totally different.

[merks]

@iloveeclipse A big plus from migrating to github is that cloning is much faster. For me SWT binaries clones in just under 3 minutes...

[niraj-modi]

Coping from(eclipse-platform/eclipse.platform#7 (comment)):

Merging of SWT repos will break all previous SWT build setups and back-porting of patches which will become challenging if we merge repos:

• Also all old Eclipse build setups which we manage to work with might break(be it cherry-pick or SWT native builds)
• And it's no brainier to rework any of the old Eclipse build setups(build scripts, libraries etc...)

Currently both of SWT-Sources & SWT-Binaries repo itself is combination of code from below 3 different platforms:

• Windows
• MAC
• Linux

In general and specifically existing Eclipse/SWT wiki articles for setup/configuration are written with SWT(Source & Binaries) repo wise and it may also need to be fixed.

Sorry it's a NoGo for SWT(Sources & Binaries) repo merging.

Not only new comers but also all the existing corporate customers who just depend on Eclipse/SWT sources to build their products(on non-supported versions or non-supported platforms) are prone to break and may get out of picture for same set of reasons.

[sravanlakkimsetti]

@sravanlakkimsetti Yeah, without a binaries repo, rebuilding historical releases will get more complicated, that's a real problem. But why are they rebuilding historical releases at all? Why not build them once and store in a maven/p2 repo forever?

One of the reasons could be a bug fix required in older releases. Atleast in IBM a product needs to be supported for 5 years. Any bugs reported needs to be fixed. so they end up rebuilding historical releases.

Making it difficult to build will make customers to backdown. This would a very bad idea.

I am all for optimizing. At the same time we should also make it simpler for customers to build their own products and make contributing easy. to me both are equally important one gets us funding and another gets us good fixes.

[iloveeclipse]

@merks : that depends on your provider / country.

@niraj-modi : I see your point.

Is moving SWT binaries to git LFS alone worth it then?

If we will go for LFS, we would need to:

1. Rewrite history for SWT binaries (which shouldn't be a problem)?
2. There should be no extra changes needed after that step (I think)
   Conversion info: https://github.com/git-lfs/git-lfs/blob/main/docs/man/git-lfs-migrate.1.ronn

Pros:

1. Smaller clone size / faster checkout
2. anything else?

Cons:

1. One technology layer more
2. anything else?

[vogella]

@niraj-modi is SWT doing a lot of backporting? Looks to me that 4.23, 4.22, 4.21 and 4.20 maintenance branch still points to commit tagged for the release.

[akurtakov]

So let's try to summarize discussion so far to see whether we can come to an agreement.

1. Biggest concern seems to be backporting fixes. If eclipse.platform.swt.binaries repository stays as it is for years, backports could be done without any breakage for old releases.
2. When we speak about merging swt.binaries into swt repo I think no one had the intention to put actual dll/so files in swt repo but rather the rest manifests, pom, etc. I haven't looked into Git LFS so can't speak for it but one setup could be actual binaries being put in a structure like downloads.eclipse.org/eclipse/swt-natives/4567a/[gtk,mac,win] and on maven build hook download-maven-plugin to prepare-resources stage so it downloads the needed native bits for the fragment (e.g. https://github.com/eclipse/wildwebdeveloper/blob/master/org.eclipse.wildwebdeveloper/pom.xml#L34). Same could be achieved as ant builder for in workbench experience.
3. With an approach as in 2. there will always be binaries available for people that can't/won't setup environments to rebuild natives.
4. Fixing documentation - that is inevitable and we just have to do it.

Does it sound more applicable now?

[iloveeclipse]

When we speak about merging swt.binaries into swt repo I think no one had the intention to put actual dll/so files in swt repo

Nope, exact that is the proposal. Have everything in one repo will simplify bisecting / changes a lot, we would not need to synchronize anything across two different repos / different locations, because everything will be in same git repository.

I haven't looked into Git LFS

Git LFS stores the text url to the file in a server in the commit, not the binary itself in the repository (it goes to the extra server storage). This means, we can push 10 MB binaries but the commit record itself will be few bytes.

[akurtakov]

When we speak about merging swt.binaries into swt repo I think no one had the intention to put actual dll/so files in swt repo

Nope, exact that is the proposal. Have everything in one repo will simplify bisecting / changes a lot, we would not need to synchronize anything across two different repos / different locations, because everything will be in same git repository.

I haven't looked into Git LFS

Git LFS stores the text url to the file in a server in the commit, not the binary itself in the repository (it goes to the extra server storage). This means, we can push 10 MB binaries but the commit record itself will be few bytes.

Thanks for explaining. What does it mean for clone time? When is the binary for given commit downloaded ? At clone time? If it practically means no slowdown for clone while having the binary when needed sounds exactly what we need.

[iloveeclipse]

I think on clone you get only the last version of the binaries (with the checkout).
So clone should only transfer ~10 MB for SWT (8MB for all latest binaries and few MB for the entire history) I guess.

• https://git-lfs.github.com
• https://www.atlassian.com/git/tutorials/git-lfs
• There is also additional speedup possible by using git lfs clone http://manpages.ubuntu.com/manpages/jammy/man1/git-lfs-clone.1.html

[niraj-modi]

@iloveeclipse
Answering to you question posted on eclipse-platform/eclipse.platform#7 (comment): There is huge difference in sizes of SWT Source and SWT Binaries repo.

Your above comments suggests that the old SWT Binaries repo would be read-only.
Read only SWT binaries repo might be a problem with SWT build input(as we commits tag & updated binaries during native build), so we will need old SWT binaries repo in writable state to work with. Thanks!

[iloveeclipse]

@niraj-modi : the whole point of git LFS is to avoid huge repo size.

So binaries are all on server and only fetched on checkout of a commit, in the "placeholder" there is just plain text link to the server url. I expect the SWT repo size after merge with SWT binaries be ~5 - 10 MB higher only (but I haven't tested, just looking how much space binaries taking on the actual binaries repo and how much the rest).

And for the read-only repo I meant the old one, not the new one (main SWT), which will be of course writable.

[niraj-modi]

@niraj-modi : the whole point of git LFS is to avoid huge repo size.

So binaries are all on server and only fetched on checkout of a commit, in the "placeholder" there is just plain text link to the server url. I expect the SWT repo size after merge with SWT binaries be ~5 - 10 MB higher only (but I haven't tested, just looking how much space binaries taking on the actual binaries repo and how much the rest).

And for the read-only repo I meant the old one, not the new one (main SWT), which will be of course writable.

We will need writable repo not only for main master but also for maintenance branches as well(for SWT build input to work)

Note: At-least in IBM a product needs to be supported for 5 years and please check below log:
https://git.eclipse.org/c/platform/eclipse.platform.swt.binaries.git/log/?h=R4_8_maintenance

Hope you understand my concerns w.r.t. to SWT binaries repo and reason/importance of it to be writable on master/old as well :)

[akurtakov]

@iloveeclipse There is no requirement for old repo to become readonly, right? We can e.g. clean master stating that it's no longer used and is there only for support or old releases.
Do you think you can play a bit with some such repo of your own which we can play with to get better feeling of the possible implementation?

[iloveeclipse]

Regarding read only: sure. I assumed no one need write access to the main repo and all rebuilds happen on local clones.

Regarding playing with git lfs: I surely can try if that all worka at all as planned.

[akurtakov]

Regarding read only: sure. I assumed no one need write access to the main repo and all rebuilds happen on local clones.

@niraj-modi Do you have more principal concerns?

Regarding playing with git lfs: I surely can try if that all worka at all as planned.
Thanks!

[SyntevoAlex]

github provides 1 GB free bandwidth for git LFS repos, so it should be enough

Note that this is 1GB bandwidth. Here are my calculations, correct me if I'm wrong:

1. 7.28mb = sum of current *.so, *.dll, *.jnilib files in a single checkout.
2. 1.53mb = same files compressed in .zip. Gives a rough ballpark of how much LFS bandwidth is used per checkout. Here I'm assuming that GitHub charges bandwidth in terms of transferred (compressed) git objects and not in terms of uncompressed data.
3. This makes some 600 checkouts, for all people combined, per month.
4. Now consider build bots. How many checkouts per month do they make? My estimate: too many.

My conclusion: Free GitHub bandwidth is not sufficient for SWT binaries repo. We could probably use a separate LFS server, but this approach sounds quite weird in the grand picture of trying to move things to GitHub.

[SyntevoAlex]

Here's my attempt to summarize (and discuss) mentioned benefits:

1. Faster clone - While I agree that cloning entire eclipse.platform.swt.binaries is slow, I need to mention that shallow clone (basically cloning just the top commit) of the repo is super fast. Takes around 1 second for me. See also Bug 562937.
2. Easier bisecting - that's somewhat nice. On the other hand, I bisected a few times, and I can't say that manually checking out matching commit in binaries repo was a pain to me. For those who do bisecting more often, this can be further automated with a simple script (look up commit matching regex in SWT, look up same commit message in binaries, check it out there).
3. The general "let's merge stuff" thing, which I see as a motto, but not as a justification for any action.

I'm not sure if merging is worth the trouble.

[iloveeclipse]

@HannesWell : many thanks, it was really huge amount of hard work.

[HannesWell]

@HannesWell : many thanks, it was really huge amount of hard work.

Thanks and you are welcome. It was indeed a lot of work but also a lot to learn.

Since there were no reports about any problems over the day I have completed this and merged #70 to clear and retire this repository.