Lean git clone – move large files to LFS

[adamziel]

Cloning the Playground repo can be slow when done without --depth=1 --shallow-clone because git will download the full history, including all historical .wasm builds and WordPress builds. One way to make it smaller is rewriting the history to move all the large files to LFS. Now, I don't like the idea of rewriting the history, but we could create a new branch off trunk and rewrite it to effectively have a historical, large trunk and a lean new main. We could then change the main branch to main.

Major kudos to @brandonpayton @JanJakes @zaerl @mho22 @akirk @fellyph for brainstorming these ideas

[adamziel]

Here's some commands from GPT we may be able to use for this process:

LFS is controlled by .gitattributes, which lives in each branch. So a branch can track files with LFS while others don’t.

What that means in real life:

• If you add .gitattributes on main and rewrite only main to move big files to LFS pointers, then set main as the only active branch (delete/hide the rest and heavy tags), a default git clone becomes lean.
• If you keep old branches/tags that still reference big blobs, they remain reachable, and default clones still pull the weight—LFS-on-one-branch won’t fix that.

How to do “branch-only LFS” cleanly:

1. On main, add tracking rules:

git checkout main
git lfs install
git lfs track "*.png" "*.jpg" "*.zip" "*.pdf" "*.mp4"
git add .gitattributes
git commit -m "Track large files with LFS on main"

2. Rewrite just main history to move existing large files to LFS:

git lfs migrate import --include="*.png,*.jpg,*.zip,*.pdf,*.mp4" --include-ref=refs/heads/main
git push --force origin main

3. Make the clone small without user flags:

• Delete stale branches and heavyweight tags you don’t need:

# example
git push origin :old-big-branch
git tag -d v0.1 v0.2; git push origin :refs/tags/v0.1 :refs/tags/v0.2

• If you must keep deep archaeology, move it to a separate “-archive” repo.

[thelovekesh]

Here is the list of first 50 biggest blobs in the repo:

$ git rev-list --all --objects | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | sed -n 's/^blob //p' | sort --numeric-sort --key=2 -r | head -n 50 | awk '{ printf "%.2f MB\t%s\t%s\n", $2/1048576, $1, $3 }' 

60.74 MB        319bae1b70c52c83c08ab4317425547d6d632aca        packages/playground/website/playwright/deploy-e2e-mid-release.zip
59.36 MB        46bc032c2388082b25e70b7cee05107234c4e9a6        packages/playground/website/playwright/deploy-e2e-old-release.zip
59.36 MB        da95ec8afd19d1bec162732eb331b58ad4c8cd8f        public/php-web-wordpress.data
47.16 MB        a4994db7a8732d2f4f1d66772ef7b9c5045b07cb        public/php-web-wordpress.data
43.52 MB        e325d4606154ac4d8b5676f7cfe25b2aed2accd0        dist-web/wp.data
35.23 MB        ff34fd9ad43ce03075e5429eb16b4b4c4bbbd957        packages/php-wasm/node/asyncify/8_4_11/php_8_4.wasm
35.23 MB        5c4af57888c52efc934a6515170e186dd5d2a2ae        packages/php-wasm/node/asyncify/8_4_11/php_8_4.wasm
35.23 MB        e087fc134dcf84b463fbfe360972be2fe18c8a39        packages/php-wasm/node/asyncify/8_4_10/php_8_4.wasm
35.23 MB        56b80b5c4dfc0126b137a675d8e9d062e3f999e9        packages/php-wasm/node/asyncify/8_4_10/php_8_4.wasm
35.08 MB        a7733707cae14bdcd0a562601ed388a120082d8f        packages/php-wasm/node/asyncify/8_4_10/php_8_4.wasm
35.06 MB        3fd69ae787eb1d9e5cd74204d591802e67563603        packages/php-wasm/node/asyncify/8_4_0/php_8_4.wasm
35.06 MB        1a6d8e4c149cb11bc559d7acac882d474d3bee2e        packages/php-wasm/node/asyncify/8_4_0/php_8_4.wasm
35.06 MB        5a2b1ae650f24a6e0c7717a2f9b7dfd667c6ca89        packages/php-wasm/node/asyncify/8_4_0/php_8_4.wasm
35.06 MB        36329c9a671937bacbc5625066548ab268dee214        packages/php-wasm/node/asyncify/8_4_0/php_8_4.wasm
35.04 MB        59c6eca519e3b39cb009c9dba6b9e56c9ec91a54        packages/php-wasm/node/asyncify/8_4_0/php_8_4.wasm
34.69 MB        99e4cfe8ac6434a90c36dec80040707b5e92e945        packages/php-wasm/node/jspi/8_4_11/php_8_4.wasm
34.69 MB        e5071cf59da18b29d59d6319eb77348c72a86614        packages/php-wasm/node/jspi/8_4_10/php_8_4.wasm
34.69 MB        62b14f231be1f06855012a84385999cef3a54105        packages/php-wasm/node/jspi/8_4_10/php_8_4.wasm
34.53 MB        e972a4e032d3d6088bb2893033a20054233e5a61        packages/php-wasm/node/jspi/8_4_0/php_8_4.wasm
34.53 MB        54558ec4eb08b83282887e9c0b89b20f9e370995        packages/php-wasm/node/jspi/8_4_0/php_8_4.wasm
34.52 MB        8df64949d7090f4ea5507bdc2c872def8506e33e        packages/php-wasm/node/jspi/8_4_0/php_8_4.wasm
34.52 MB        609f6ac16dcab892440b076c8e7cd4d336b916af        packages/php-wasm/node/jspi/8_4_0/php_8_4.wasm
34.52 MB        551312cebe09ca468e6baed2b26501eaf8dfcc4b        packages/php-wasm/node/jspi/8_4_0/php_8_4.wasm
31.61 MB        90f8044bca3f7ddeb0d30efd75e2c880a84c1687        packages/php-wasm/node/asyncify/8_3_24/php_8_3.wasm
31.61 MB        cc1220cfc821735a7ea7d499c542fc9a7754faaa        packages/php-wasm/node/asyncify/8_3_23/php_8_3.wasm
31.61 MB        5447a59427e891ba4bd9bc332f43bac33de6f17f        packages/php-wasm/node/asyncify/8_3_23/php_8_3.wasm
31.61 MB        849bfa3926b8324b2df7cac02d900691d9110bb1        packages/php-wasm/node/asyncify/8_3_24/php_8_3.wasm
31.46 MB        e04b47ec547168d8fec15dc7deada2108fe303ee        packages/php-wasm/node/asyncify/8_3_23/php_8_3.wasm
31.42 MB        22d39bf945071c5c489f565694e3158dfe084aa1        packages/php-wasm/node/asyncify/8_3_0/php_8_3.wasm
31.42 MB        0d686f46cc14b2ce4cbffbd4c892f07ac8e1d04f        packages/php-wasm/node/asyncify/8_3_0/php_8_3.wasm
31.42 MB        0c716f86d110652a0c6060551755a4a8ebcf0898        packages/php-wasm/node/asyncify/8_3_0/php_8_3.wasm
31.42 MB        6b58c5f2931250f8a0629ddd209612dad1bfdd3f        packages/php-wasm/node/asyncify/8_3_0/php_8_3.wasm
31.41 MB        40dce2b700f0494c2babfe46724076073c494b91        packages/php-wasm/node/asyncify/8_3_0/php_8_3.wasm
31.06 MB        76fd277e7da8a6b01215f5c6f79515f680163964        packages/php-wasm/node/jspi/8_3_24/php_8_3.wasm
31.06 MB        cf6928345bb911c59d7a055b93fcd36448fc709a        packages/php-wasm/node/jspi/8_3_23/php_8_3.wasm
31.06 MB        8d6e91e59c9fb43d9216b26e0f36575a0c2fdc23        packages/php-wasm/node/jspi/8_3_23/php_8_3.wasm
30.89 MB        f2111b5b70b1415d9fb81a119a3a03a00adb6ee9        packages/php-wasm/node/jspi/8_3_0/php_8_3.wasm
30.89 MB        bcab5513c495df82825548be6d86d6300889be20        packages/php-wasm/node/jspi/8_3_0/php_8_3.wasm
30.89 MB        b9d44485d08e0652cdec2038a5c0d62fdffa2ebe        packages/php-wasm/node/jspi/8_3_0/php_8_3.wasm
30.89 MB        96ba3b538ce989fe08b954a2cf18bfc1da079cd5        packages/php-wasm/node/jspi/8_3_0/php_8_3.wasm
30.89 MB        16ab71951ca0168d79ae3a94e7f1c46c762112e0        packages/php-wasm/node/jspi/8_3_0/php_8_3.wasm
30.56 MB        b74fd68e7c3f5974e4a7ef4a1561d57852381be9        packages/php-wasm/node/asyncify/8_2_29/php_8_2.wasm
30.56 MB        9f482d567bf2b29f5542b74a6abe1e414cc68fc6        packages/php-wasm/node/asyncify/8_2_29/php_8_2.wasm
30.56 MB        819070471c90db93a49fc98ffbb1d69a98506325        packages/php-wasm/node/asyncify/8_2_29/php_8_2.wasm
30.56 MB        60f5af5ec0ed3affa337c600b14ee104876baa7d        packages/php-wasm/node/asyncify/8_2_29/php_8_2.wasm
30.42 MB        3b615c7a0511f5396d470dd656bdb63ff988c3ed        packages/php-wasm/node/asyncify/8_2_29/php_8_2.wasm
30.38 MB        5937a8036f5d72ae9fa012989293c3fee015988b        packages/php-wasm/node/asyncify/8_2_10/php_8_2.wasm
30.38 MB        a9a1e2cfe804f23b83b1c6c8df7dbc61b602ee9d        packages/php-wasm/node/asyncify/8_2_10/php_8_2.wasm
30.38 MB        af06dc89d6b44d06588d314e9ec2cc38a2ada1cb        packages/php-wasm/node/asyncify/8_2_10/php_8_2.wasm
30.38 MB        20ca3bb0f8e0c4314d85356499308f15da6cfb94        packages/php-wasm/node/asyncify/8_2_10/php_8_2.wasm

and I think just adding them to git lfs won't cut it rather we need to prune them from the git history which can be done by:

git filter-repo --path-glob '*.wasm' --path-glob '*.zip' --invert-paths

or using the above proposed solution to migrate big files to lfs.

[thelovekesh]

Also, what about not including build files in the git repo at all and attach them as the release assets?

[adamziel]

Also, what about not including build files in the git repo at all and attach them as the release assets?

I was thinking about that, too! Unfortunately, that approach seems less practical of the two:

• Creating and reviewing Pull Requests would get more difficult – where would the rebuilt wasm assets live? They need to be included in the PR as it's not reasonable to ask every reviewer to wait 10 hours for a full rebuild.
• Every wasm-affecting commit would require a release, otherwise git checkout old-commit wouldn't reliably reflect the old project state. Ditto for git bisect. This would limit the ability to work with the repository while offline.
• While the final bundle only ships php.wasm files, we also keep openssl and other libraries so we'd need to put them in the release as well.
• We'd need a bunch of code to orchestrate this commit <-> release synchronization, download the artifacts in the local dev setup, CI etc. It would require maintenance and likely break every now and then.