[🐛 Bug]: Git pack file is really large

[jasperstone]

What happened?

Cloning SeleniumHQ is really large.

It appears that there are a large number of binary files stored in cpp, third_party and website.

Suggest moving any references to Git LFS or just referencing a url to download from the vendor.

How can we reproduce the issue?

1. Clone the repo
2. Run the following command from this page
https://stackoverflow.com/questions/10622179/how-to-find-identify-large-commits-in-git-history

git rev-list --objects --all |   git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |   sed -n 's/^blob //p' | awk '$2 >= 2^20' |  sort --numeric-sort --key=2 |   cut -c 1-12,41- |   $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest > output.txt

### Relevant log output

```shell
Some of the larger files...
277ca571dcb3  5.1MiB common/manager/linux/selenium-manager
87a299d8ba46  5.1MiB third_party/closure/bin/compiler-20120123.jar
d958a61882cf  5.1MiB third_party/closure/bin/compiler-20120305.jar
24ae5637d301  5.2MiB third_party/java/android/android-8.jar
b7111d8b95e9  5.2MiB lib/buildtime/jruby-complete-r5974.jar
52a920f28c74  5.2MiB third_party/closure/bin/compiler-20120710.jar
7e0290349482  5.3MiB third_party/selenium/selenium-server-1.0.1.jar
fb4eb39f0119  5.4MiB third_party/dotnet/nuget/NuGet.exe
96430f7a5ee9  5.5MiB third_party/py/Sphinx-1.8.2.tar.gz
e23352ff0bdb  5.6MiB third_party/closure/bin/compiler-20120917.jar
22025c98c19a  5.6MiB website/download/selenium-rc-1.0-SNAPSHOT.zip
c12a17d3e227  5.7MiB website/webapps/ROOT/selenium_v0.1.3.zip
a12f24be30b3  5.7MiB third_party/closure/bin/compiler-20140814.jar
e2338676d56a  6.0MiB third_party/closure/bin/compiler-20130227.jar
d969a3313dca  6.0MiB third_party/closure/bin/compiler.jar
33b44ed43902  6.0MiB third_party/closure/bin/compiler.jar
a297dc0f126d  6.0MiB third_party/closure/bin/compiler.jar
0b0e68c3dd16  6.0MiB third_party/closure/bin/compiler-20130603.jar
e4e6756afc65  6.1MiB third_party/closure/bin/compiler.jar
9cae6957d4c0  6.1MiB third_party/closure/bin/compiler.jar
0d42389c08b7  6.1MiB third_party/closure/bin/compiler.jar
0f83b17f510b  6.2MiB third_party/dotnet/nuget/NuGet.exe
57859f0b2f04  6.3MiB third_party/closure/bin/compiler.jar
5943f680ceae  6.4MiB third_party/dotnet/nuget/NuGet.exe
4fed060629ee  6.5MiB website/download/selenium-remote-control-0.7.zip
67aa45afcffd  6.5MiB third_party/closure/bin/compiler-20140407.jar
a1d81349816d  6.5MiB website/download/selenium-remote-control-0.7.1-SNAPSHOT.zip
ec9f5b0784e2  6.5MiB website/download/selenium-remote-control-0.7.1.zip
d621604551a5  6.9MiB third_party/closure/bin/compiler.jar
239bf9fbae07  7.4MiB third_party/closure/bin/compiler.jar
27358741fe71  7.6MiB third_party/py/Babel-2.6.0.tar.gz
35effd7af200  8.3MiB third_party/py/jython.jar
3f2c98347240  9.0MiB docs/api/java/index-all.html
3800352dd9de   10MiB website/download/selenium-remote-control-0.7.2-SNAPSHOT-924-1023.zip
db0e1befa4a2   11MiB third_party/jruby/jruby-complete-1.4.0RC1.jar
859a8c03220f   11MiB third_party/jruby/jruby-complete-1.5.0.RC2.jar
7572a14d1723   11MiB third_party/java/browsermob_proxy/browsermob-proxy-1.0-SNAPSHOT-release.jar
f32c46305db1   11MiB third_party/java/htmlunit/htmlunit-driver-2.20.0-SNAPSHOT.jar
40289c9469dd   12MiB third_party/jruby/jruby-complete.jar
35f6451ec32f   12MiB third_party/jruby/jruby-complete.jar
15d50e102d34   13MiB third_party/jruby/jruby-complete-1.5.0.RC2.jar
9e01e5caa0c3   14MiB third_party/py/jython.jar
476761b860fc   14MiB third_party/jruby/jruby-complete.jar
7dfaeb5e621e   14MiB third_party/jruby/jruby-complete.jar
e88a9eb7e281   15MiB third_party/jruby/jruby-complete.jar
4a979b82f080   16MiB third_party/jruby/jruby-complete.jar
edd1c82164f5   17MiB third_party/java/android/android-14.jar
96dd3dd7b503   18MiB third_party/gecko-1.9.0.11/mac/lib/XUL
10128805fa57   18MiB third_party/java/dossier/dossier-0.1.1.jar
112b31b46496   18MiB third_party/java/dossier/dossier-0.2.1.jar
fb5d97499e26   18MiB third_party/java/dossier/dossier-0.3.0.jar
4d630f4a8c54   18MiB third_party/java/dossier/dossier-0.4.0.jar
4c9e78044e34   18MiB third_party/java/dossier/dossier-0.4.1.jar
be4e36b9093f   18MiB third_party/gecko-1.9.0.11/linux/lib/libxul.so
ad0da1fa57ba   19MiB third_party/jruby/jruby-complete.jar
5d52d406e53d   19MiB third_party/jruby/jruby-complete.jar
f25bc68c91d8   19MiB third_party/java/dossier/dossier-0.7.1.jar
ac78b3a4923e   19MiB third_party/java/dossier/dossier-0.7.2.jar
58977b359e30   19MiB third_party/java/dossier/dossier-0.6.1.jar
3fa4e9c04cec   19MiB third_party/java/dossier/dossier-0.6.2.jar
4b968cc2db46   19MiB third_party/jruby/jruby-complete.jar
f82d46909648   25MiB third_party/jruby/jruby-complete.jar
eb3e6639c466   25MiB third_party/jruby/jruby-complete.jar
e090d1926e24   27MiB third_party/java/htmlunit/htmlunit-2.11-src.zip
f886fb00a4fb   34MiB third_party/jruby/jruby-complete.jar
fccd11760146   34MiB third_party/jruby/jruby-complete.jar
613bb32ba35d   34MiB third_party/jruby/jruby-complete.jar

Operating System

Windows 11 22H2

Selenium version

N/A

What are the browser(s) and version(s) where you see this issue?

N/A

What are the browser driver(s) and version(s) where you see this issue?

N/A

Are you using Selenium Grid?

No

[github-actions]

@jasperstone, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[titusfortner]

Yeah, it won't affect git clone, but removing some of the things that aren't actively being used in our current build process would at least improve --depth 1

[diemol]

Thank you for this issue, I've changed the contribution guidelines to highlight people should clone with --depth 1, as the repository contains the history for over 10 years of project changes.

If folks want to remove things that are not used, please help us with a PR.

[jasperstone]

@titusfortner, removing them from the repo won't affect the clone size but removing them from the git history should. Since the files are binary, every complete checked in version is stored in history instead of just the text diff.

For instance, just looking at the worst offender, removing this history would save 292MB in the pack file.
40289c9469dd 12MiB third_party/jruby/jruby-complete.jar
35f6451ec32f 12MiB third_party/jruby/jruby-complete.jar
476761b860fc 14MiB third_party/jruby/jruby-complete.jar
7dfaeb5e621e 14MiB third_party/jruby/jruby-complete.jar
e88a9eb7e281 15MiB third_party/jruby/jruby-complete.jar
4a979b82f080 16MiB third_party/jruby/jruby-complete.jar
ad0da1fa57ba 19MiB third_party/jruby/jruby-complete.jar
5d52d406e53d 19MiB third_party/jruby/jruby-complete.jar
4b968cc2db46 19MiB third_party/jruby/jruby-complete.jar
f82d46909648 25MiB third_party/jruby/jruby-complete.jar
eb3e6639c466 25MiB third_party/jruby/jruby-complete.jar
f886fb00a4fb 34MiB third_party/jruby/jruby-complete.jar
fccd11760146 34MiB third_party/jruby/jruby-complete.jar
613bb32ba35d 34MiB third_party/jruby/jruby-complete.jar

[diemol]

@jasperstone, what are the consequences of rewriting the git history when several people have the project already cloned?

[titusfortner]

We used a custom JRuby compilation to run our build system for many years, so it isn't surprising to see those.

Regardless, we aren't going to rewrite the git history, that would be a disaster.

[jasperstone]

Some technical debt is harder to pay off than others. I wouldn't want to deal with a 2GB repo for my developers and CI/CD pipelines but if it isn't causing your team headaches, let it ride.

[diemol]

It is not because most developers are cloning it once and forgetting about it. For spontaneous contributions we enable GitPod or (thanks to this issue) we recommend cloning with --depth 1.

[titusfortner]

CI pipeline only checks out HEAD by default, and average cost of 2 GB looks to be about 3 cents at this point.

If there are files we can delete in HEAD, we should, but changing git history maroons everything anyone has ever cloned or forked; it breaks all PRs and prevents everyone from updating their code. It should never be done in a public repo, ever.

[titusfortner]

To an extent, this is the cost of a mono-repo, which has a lot of advantages. All the big companies (including Microsoft) have repos that are much larger than Selenium's. Recent git updates that were designed to significantly optimize large repos didn't affect Selenium code at all because it isn't actually considered a large repo.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.