[Atom] symlinks stripped from app.asar, leading to larger installer and size on disk

[lptr]

I only checked this on macOS. The package contains the exact same 2.2 MB binary more than 120 times over. This wastes 270 MB disk space.

Dugite alone is responsible for 500 MB of the Mac installation of Atom for example.

Is there a way to reduce this overhead?

[shiftkey]

@lptr this is actually an issue with dugite-native which is the native toolchain. Gonna move this issue over there as that's where the Makefiles are located.

[shiftkey]

For context, the flag in Git's makefile that affects this behaviour is NO_INSTALL_HARDLINKS:

dugite-native/script/build-macos.sh

Lines 10 to 16 in 2968933

	DESTDIR="$DESTINATION" make install prefix=/ \
	NO_PERL=1 \
	NO_TCLTK=1 \
	NO_GETTEXT=1 \
	NO_DARWIN_PORTS=1 \
	NO_INSTALL_HARDLINKS=1 \
	MACOSX_DEPLOYMENT_TARGET=10.9

And the details from the Makefile source for it are:

# Define NO_INSTALL_HARDLINKS if you prefer to use either symbolic links or
# copies to install built-in git commands e.g. git-cat-file.

I'm trying to recall whether I had any issues with symlinks that I had to opt-in for that behaviour, but I can't recall.

[shiftkey]

The package contains the exact same 2.2 MB binary more than 120 times over. This wastes 270 MB disk space.

I'm curious about these details, as the dugite-native archive within dugite (this is me poking at the version we include in GitHub Desktop on my macOS environment) looks like this:

// just total up the size on disk
$ du -sh app/node_modules/dugite/git
 29M	app/node_modules/dugite/git

// find all the files that look like binaries
$ find app/node_modules/dugite/git -type f | perl -lne 'print if -B'
app/node_modules/dugite/git/bin/git
app/node_modules/dugite/git/libexec/git-core/git-lfs
app/node_modules/dugite/git/libexec/git-core/git-http-push
app/node_modules/dugite/git/libexec/git-core/git-imap-send
app/node_modules/dugite/git/libexec/git-core/git-sh-i18n--envsubst
app/node_modules/dugite/git/libexec/git-core/git-credential-store
app/node_modules/dugite/git/libexec/git-core/git-credential-cache
app/node_modules/dugite/git/libexec/git-core/git-daemon
app/node_modules/dugite/git/libexec/git-core/git-credential-cache--daemon
app/node_modules/dugite/git/libexec/git-core/git-remote-http
app/node_modules/dugite/git/libexec/git-core/git-fast-import
app/node_modules/dugite/git/libexec/git-core/git-shell
app/node_modules/dugite/git/libexec/git-core/git-http-fetch
app/node_modules/dugite/git/libexec/git-core/git-http-backend
app/node_modules/dugite/git/libexec/git-core/git

[lptr]

This is what I get with Atom (installed via brew cask install atom):

$ du -hs `pwd`
302M	/Applications/Atom.app/Contents/Resources/app.asar.unpacked/node_modules/dugite/git/libexec/git-core

$ find . -type f | perl -lne 'print if -B'
./git-mailsplit
./git-fetch
./git-lfs
./git-clean
./git-diff
./git-log
./git-http-push
./git-commit
./git-merge-index
./git-merge-recursive
./git-revert
./git-blame
# ... (many more)

[lptr]

And app.asar also includes these files redundantly, so it clocks in at 265MB.

[shiftkey]

cc @smashwilson for ideas on why this might be happening to Atom

[shiftkey]

Digging into it some more, this is related to how Electron and it's ASAR support works.

ASAR is a package support format intended to squash things like a complex node_modules folder into a single archive while still being usable into an app, rather than thousands of those loose files on disk. In this case the app.asar contains the dugite dependencies and other things that Atom needs at launch (Electron unpacks them into app.asar.unpacked because it can't entirely mimic the filesystem - I think):

$ cd /Applications/Atom.app/Contents/Resources
$ npx asar list app.asar | grep dugite
npx: installed 88 in 2.213s
/node_modules/dugite
/node_modules/dugite/build
/node_modules/dugite/build/lib
/node_modules/dugite/build/lib/errors.js
/node_modules/dugite/build/lib/git-environment.js
/node_modules/dugite/build/lib/git-process.js
/node_modules/dugite/build/lib/index.js
/node_modules/dugite/git
/node_modules/dugite/git/bin
/node_modules/dugite/git/bin/git
/node_modules/dugite/git/libexec
/node_modules/dugite/git/libexec/git-core
/node_modules/dugite/git/libexec/git-core/git
/node_modules/dugite/git/libexec/git-core/git-add
/node_modules/dugite/git/libexec/git-core/git-add--interactive
/node_modules/dugite/git/libexec/git-core/git-am
/node_modules/dugite/git/libexec/git-core/git-annotate
/node_modules/dugite/git/libexec/git-core/git-apply
/node_modules/dugite/git/libexec/git-core/git-archimport
/node_modules/dugite/git/libexec/git-core/git-archive
/node_modules/dugite/git/libexec/git-core/git-bisect
/node_modules/dugite/git/libexec/git-core/git-bisect--helper
/node_modules/dugite/git/libexec/git-core/git-blame
/node_modules/dugite/git/libexec/git-core/git-branch
/node_modules/dugite/git/libexec/git-core/git-bundle
/node_modules/dugite/git/libexec/git-core/git-cat-file
/node_modules/dugite/git/libexec/git-core/git-check-attr
/node_modules/dugite/git/libexec/git-core/git-check-ignore
/node_modules/dugite/git/libexec/git-core/git-check-mailmap
/node_modules/dugite/git/libexec/git-core/git-check-ref-format
/node_modules/dugite/git/libexec/git-core/git-checkout
/node_modules/dugite/git/libexec/git-core/git-checkout-index
/node_modules/dugite/git/libexec/git-core/git-cherry
...

You enable ASAR support in electron-packager and I believe the default behaviour is to ignore symlinks, which means it emulates the symlink by recreating the file each time.

For reference, GitHub Desktop doesn't have the same issue because we disabled ASAR and manage the embedded Git installation ourselves:

$ cd /Applications/GitHub Desktop.app/Contents/Resources/app
$ ls -l git/libexec/git-core
total 55224
-rwxr-xr-x   1 shiftkey  staff  2405568  9 Nov 14:35 git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-add -> git
-rwxr-xr-x   1 shiftkey  staff      100  5 Oct 16:08 git-add--interactive
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-am -> git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-annotate -> git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-apply -> git
-rwxr-xr-x   1 shiftkey  staff      100  5 Oct 16:08 git-archimport
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-archive -> git
-rwxr-xr-x   1 shiftkey  staff    14444  5 Oct 16:08 git-bisect
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-bisect--helper -> git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-blame -> git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-branch -> git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-bundle -> git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-cat-file -> git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-check-attr -> git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-check-ignore -> git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-check-mailmap -> git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-check-ref-format -> git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-checkout -> git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-checkout-index -> git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-cherry -> git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-cherry-pick -> git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-clean -> git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-clone -> git
lrwxr-xr-x   1 shiftkey  staff        3  9 Nov 15:42 git-column -> git
...

We tried ASAR ages ago and ended up just avoiding it (see desktop/desktop#476 and the linked PR for more info), and I see someone added support for being able to control the symlink behaviour (to opt-out of the default behaviour) in electron/packager#410.

Ultimately, this is up to the packaging apps rather than something I have control over. I don't see a derefSymlinks flag being set in the Atom packager config so maybe it's worth asking over in atom/atom if they've tried derefSymlinks when packaging.

[lptr]

Thanks for looking into this. Should I report this to Atom, to Electron, or are you going to take further steps here?

[shiftkey]

Should I report this to Atom, to Electron, or are you going to take further steps here?

Please have a read of the bug reporting guidelines for atom/atom.

[p-e-w]

So, has this been/should this be reported? From the exchange above, it's not 100% clear what the expected steps are here.

This issue is the root cause of atom/atom#18452.

[p-e-w]

Followup issue filed at atom/atom#18490.