feat: kubo 0.29 and native apple silicon

[lidel]

This is exploration into better support of Apple Silicon.

electron-builder supports producing either universal package or one for each architecture.

In case of Desktop, things are more complicated: we also bundle go-ipfs binary.
This means that unless the go-ipfs binary is also universal or per-arch, we can't improve much.

Due to the way we depends on go-ipfs via npm-go-ipfs universal binaries all the way sound like way easier route (downside being the disk size), however dist.ipfs.io already has amd64 and arm64 for darwin.

Universal binary test

Main cost of universal package is size, even with out go-ipfs (still being amd64) it adds ~75 MB:

• 195M IPFS Desktop-0.16.0-universal.dmg (electron amd64&arm64 + go-ipfs arm64)
• 119 MB IPFS-Desktop-0.16.0.dmg (amd64, M1 will use Roseta)

TODO

• decide if we want to provide single universal binary or one for amd64 (legacy intel) and one for arm64 (new apple silicon)
  • Data point from Brave:
    • atm IPFS integration is using amd64: go-ipfs binary is amd64, it runs on M1 via Rosetta. Brave plans on using arm64 in the near future, as they already provide arm64 version of Brave itself.. In that contenxt universal binary is wasteful (IPFS component is fetched on-demand, so small size matters).
• based on previous decision, decide how we handle go-ipfs dependency (right now it downloads kubo only for arch of CI worker, which is amd64).
• make Universal DMG and ZIP
• replace Kubo binary with universal one
  • postinstall script that detects darwin build on ARM, and fetches missing intel binary + merges both into universal and puts it at node_modules/kubo/bin/ipfs
• upstream kubo+0.28.0.patch to npm-kubo repo
  • opened feat: expose standalone download function npm-kubo#73
• test DMG from this PR on both
  • Intel
  • ARM: tested on M1 macMini with macOS Sonoma (14), works fine with universal Kubo binary
• update Electron (if needed) does not seem to be needed on ARM, tbd if Intel works
• wait for https://github.com/ipfs/npm-kubo to ship with exposed light download function
  • switch this PR to kubo dependency with feat: expose standalone download function npm-kubo#73 (0.29.x)
  • remove package-patch for kubo dependency

Closes #2813

[AlexxNica]

@lidel, do we know where the difference in size comes from, specifically? We could find where they are and try something to minimize the impact of shipping the universal package.

[unbeatable-101]

@lidel, do we know where the difference in size comes from, specifically? We could find where they are and try something to minimize the impact of shipping the universal package.

Not a dev, but I'm pretty sure the size increase is just the binaries about doubling in size due to basically containing two apps.

[jenkijo]

Hello, any update?
I've committed fix for go-ipfs ipfs/npm-kubo#44

[dvcrn]

Thanks for the PR! I wanted to build this locally but ran into issues with the build step trying to download a arm ipfs binary that obviously didn't exist. How did you build this?

[jenkijo]

It work fine with me, what error that you got?

[hacdias]

I think that Universal would be easier for the end users despite the storage size. In general, storage is cheap and 75 MB won't probably affect many people.

Iff storage size is very important, we could either build universal + download go-ipfs on first run, or ship different binaries.

[SgtPooki]

@lidel Have we looked into https://github.com/tauri-apps/tauri at all? It claims significant improvements over electron across all platforms, though I haven't investigated more deeply. I also haven't found any apps that have migrated from electron to tauri

[SgtPooki]

Regarding universal app vs rosetta on M1.. the speed increases you get by running M1 native is significant in every benchmark I've seen. However, it would be great to see a breakdown of the risks & benefits for our specific application.

I don't think we should decide to just force rosetta forevermore. If the decision is to wait for some period until we have more bandwidth, I think sticking with rosetta and the smaller bundle for now is fine. Ultimately, we should be moving to the type of app bundle the platform we're targeting recommends.

For Apple/macOS, that is the universal build.

[hacdias]

@SgtPooki an issue you may want to look at too is #867

[lidel]

Have we looked into https://github.com/tauri-apps/tauri

I think we looked at it briefly, but it was very alpha back then. IIRC the main downside was lack of tooling for end-to-end autoupdate solution, which is raison d'être of ipfs-desktop. It seems they work on it, but it is still undocumented? tauri-apps/tauri#2776

Long term, we want to move away from fetching updates from centralized HTTP server like github and leverage IPFS+IPNS instead (see old notes in #789), so if we have working setup for that, we could move away from Electron (literally the only value it brings is https://www.electron.build/auto-update working on all three platforms: macOS+Windows+Linux(AppImage))

75 MB won't probably affect many people.

Usually I would be against doubling the size of download, but we talk Apple here – it is fair to assume that if someone uses mac they most likely don't care about additional download cost over the mobile carrier (just being realistic here).

Ultimately, we should be moving to the type of app bundle the platform we're targeting recommends.
For Apple/macOS, that is the universal build.

Yeah, sounds like universal binary is the way to go. The caveat here is that we don't have "universal go-ipfs binary" – iiuc this PR (in current form) still bundles the intel version of go-ipfs.

Figuring out how to include both binaries (arm64 and amd64) for darwin is the key to unblocking this.
Someone familiar with Apple hardware and ecosystem needs to investigate if ipfs-desktop should fetch independent binaries during the build of DMG, or if dist.ipfs.io website should add universal binary upstream (if it is even possible for standalone CLI binaries).

[lidel]

I wrote a plan for unblocking this in ipfs/distributions#770 (comment).
Not a macOS user myself, but if someone wants to work on this, I'm around to advise.

[SgtPooki]

I wrote a plan for unblocking this in ipfs/distributions#770 (comment).

Not a macOS user myself, but if someone wants to work on this, I'm around to advise.

I read that comment and the approach seems reasonable and clear. Thanks for the backlink!

[whizzzkid]

@lidel should this be closed as ipfs/distributions#770 (comment) is closed.

I was also wondering if this could be solved as part of #2441 where instead of "shipping" a binary, we download and validate checksums and then use that?

[lidel]

There are two separate topics at play (mind I am not macOS user, below is my rough understanding on Apple Silicon situation)

• architecture of go binary (kubo daemon)
  • What we did on dist.ipfs.tech is we provide separate Kubo binaries for amd64 (intel) and arm64 (arm/apple silicon) (feat: kubo builds for arm64 distributions#838, fix(site): default apple downloads to arm64 distributions#841).
  • Brave downloads correct binary at runtime.
  • There is no universal binary for Kubo, but you could build one as noted in Add Universal Mac OS build distributions#770 (comment) – afaik nobody is doing that atm.
    • IPFS Desktop DMG bundles NPM version pulled in during the build – afaik this is still amd64, so it runs emulation.
    • IPFS Desktop needs to either stop embeedding and fetch correct binary at runtime, or include both and run correct one on macOS.
• architecture of electron app (ipfs desktop)
  • this feels unresolved, but maybe not a big deal, since the majority of work happens in kubo?
  • iiuc what happens on macOS is that we produce DMG that is amd64 and when run on Apple Silicon it runs emulation

I agree we should close this, as universal DMG does not solve it end-to-end.
If we want to use native arch on macOS, need to fill new issue and make a new plan for both topics above.

[SgtPooki]

please see #2681 for a continuation of this work

[lidel]

Reopening because of #2813

I will prototype Universal DMG that includes Universal Kubo binary as well.

[lidel]

Fetching both x86_64 arm64 and merging them into universal Kubo binary is working (build log):

→ lipo info after merging binaries Architectures in the fat file: /Users/runner/work/ipfs-desktop/ipfs-desktop/node_modules/kubo/kubo/ipfs are: x86_64 arm64

Remaining work is to upstream kubo+0.28.0.patch to npm-kubo repo and test DMG from this ZIP on both Intel and ARM macOS.

• 🟢 ARM: tested on M1 macMini with macOS Sonoma (14), works fine with universal Kubo binary
• ❓️ Intel: i dont have macOS machine with intel CPU, could someone help and test DMG from this ZIP? (IPFS mirror: ipfs-desktop-0.35.1-mac.dmg)
  • confirmation from community: v0.35.0 does not work on Intel Macs #2809 (comment)

[2color]

Also tested Tested https://dweb.link/ipfs/bafybeidw62sobej3qlfsmxxw5bbnczz6ithimvixf3xg7rwquur3iww66q/ipfs-desktop-0.35.1-mac.dmg on ARM and it worked. Can test on a MacOS Intel tonight.