[Desktop] arm64 builds

[chaserene]

Following up on our Slack convo, let's track here what's the status of building Standard Notes desktop for the arm64 architecture (also called arch64).

Currently it seems it's not possible. JohnnyA's experience has been that it's due to missing dependencies. Karol said doing it non-dockerized would be even harder. whichever the format, it would be greatly appreciated!

[maxdesalle]

As a Mac ARM user, I'm fully in favor of this. It should also make the app much smoother.

[JaspalSuri]

Hi @maxdesalle, we are still waiting for the dependencies that the desktop app relies on to support M1 Macs first.

[chaserene]

@JaspalSuri I just realized that my original request could have been interpreted in the context of the M1 Mac (which i didn't even have in mind). how about building for non-Apple arm64, like Cortex CPUs that are all over SoC's?

[JaspalSuri]

Hi @chaserene, no worries. We understand the difference between the requests. In both cases it's a matter of waiting for the maintainers to update the dependencies that desktop app relies on.

(For anyone else reading this thread, the web app will work on any unsupported system, like a Raspberry Pi, a BSD distro/variant, etc. 🙂)

[chaserene]

In both cases it's a matter of waiting for the maintainers to update the dependencies that desktop app relies on.

@JaspalSuri can you please list the dependencies that still block Linux arm64 builds from becoming part of the officially released builds? and can you describe how great of an effort would each be to update?

[chaserene]

I'd really like to see this happen, so I'm offering $500 to anyone (even SN team members) who succeeds in making arm64 Linux builds a sustainable part of the Standard Notes CI system.
requirements:

• you solve all blockers to making Linux arm64 (aarch64) desktop builds available in the Snap Store format and the AppImage format. you'll need to coordinate with the Standard Notes team about the specifics. as a result of your work, it becomes easy for them to officially support the above build types on the long-term.
• feature set and quality is on parity with the major desktop Standard Notes builds, especially when running on Manjaro XFCE and Armbian Buster XFCE (I can provide testing, you won't need to install these systems yourself).

the bounty will be paid out in your preferred major cryptocurrency, using rates from Coingecko at the time of sending.

if the outcome is a team effort, the involved will have to agree on a distribution before the bounty is paid out.

there's no bounty if the blockers are solved independently of your work.

if you're someone who is also excited about seeing this realized, add to the bounty in a comment below.

if there won't be serious takers, or those who take it don't progress, I maintain the right to take back this offer.

[JaspalSuri]

@JaspalSuri can you please list the dependencies that still block Linux arm64 build from becoming part of the officially released builds? and can you describe how great of an effort would each be to update?

Hi @chaserene, I'll get in touch with @johnny243 on this and get back to you.

[lsfxz]

Thanks for notifying me of this @chaserene – I've picked up working on it again and hope I'll get things somewhat sorted again (in such a way that the standard notes team can work with and easily maintain things in the future).
After all, I did manage to build it for several months until only a few weeks back, so it shouldn't be impossible :D

[lsfxz]

Potentially good news: dir and AppImage cross compilation works (even with electron 13 with a few smaller workarounds).

Snapcraft build is (seemingly) not possible to achieve on amd64, still working on that – looking promising, though.

Native compilation of dir targets works (and thus AUR/PKGBUILDs work again – I've just updated the AUR package on https://aur.archlinux.org/packages/standardnotes-desktop/ to get arm64 builds back ^^), as well as AppImage targets.

Most of the required changes are surprisingly small and only slightly more complex when building / running with a more up-to-date electron.

So once I've finished the snap process as well, I'll collect it all and present it / create PRs, if that's fine with the team.

It seems like the main leftover step would be building natively on arm64 (which seems to be required for snap builds): I don't really understand the CI/build system used (I'm usually working with gitlab and docker based runners on my own dedicated machines), so I don't know how much would complicate matters. Can you easily have parts of your release process run on arm64 machines?

Additionally: should a deb target also be striven for, or would that be covered by AppImage and snap releases?
(This should be not too much extra work, considering the main build process seems to work on a recent Ubuntu system at least; there are a few potential (known) issues with fpm when used for deb builds, which should be 'work-aroundable' by using a system fpm (via an env var), but hopefully nothing else insurmountable)

[lsfxz]

So that took a bit longer than expected (it seems like it's not possible to build a snap on Manjaro or in a (simple) docker environment, so I went with full-on qemu-system-aarch64 virtualization/emulation instead).

To sum it all up: Debian, AppImage and "unpacked" (dir) targets can be cross-built on amd64 with no changes in packages, dependencies etc. and only small changes in the build process (this was done directly on an Arch system).
This is also possible using electron 13 with only minor (and at least on the surface non-breaking) changes to dependencies.

Snaps cannot be produced on an amd64 system, but it is possible to create a snap release natively on arm64 (and with only minor changes to the build process as well). Arch / AUR packages are usually built "natively" on the target system, which is also possible.

Debian, AppImage and "unpacked" targets can also be built natively on arm64 systems with only minor changes as well.

For native arm64 builds, the same changes mentioned above allow building with electron 13 as well.

It might be possible to use a cross-built unpacked target output as a basis for packaging Snap and Arch builds. This could allow skipping the more extensive / time intensive build steps on arm64, so only the packaging steps would need to be run natively. This is just an assumption though and not tested – my assumption being that if we have to use an arm64 machine anyway, we could as well do everything there.

Furthermore, it might be possible to package an arm64 Arch package on amd64 with a few more or less ugly workarounds. Considering it's probably not a first level release target anyway (as in: it's on the AUR, that should be enough), that is probably not necessary anyway / out of scope.

I've uploaded the artifacts from my attempts at:

https://privacyshark.zero-credibility.net/privacyshark/standard-notes-3.8.18-linux-arm64.AppImage
https://privacyshark.zero-credibility.net/privacyshark/standard-notes-3.8.18-linux-arm64.deb
https://privacyshark.zero-credibility.net/privacyshark/standard-notes-3.8.18-linux-arm64.snap
https://privacyshark.zero-credibility.net/privacyshark/standardnotes-desktop-3.8.18-1-aarch64.pkg.tar.zst (this one is built using electron 13)

It would be great if someone would be willing to test those to ensure everything works properly – I'm currently not actively using standard-notes myself, so I don't really know what to look for specifically – I've only checked that things "seem to run" (and only throw updated related warnings in the terminal during launch).

Once this is done and it's ensured my attempts have lead to fully working builds, I'll happily share my process / create PRs where appropriate and try to lend a hand integrating them into the release process :)

[chaserene]

@lsfxz I really appreciate you took up the offer, even more so because now we have someone on board who has experience with the task :)

regarding a deb target: that's not a priority now, so don't strive for it. I like debs, but my motivation for now is to enable build types that can go into the official SN CI. the team currently covers Linux builds with snaps and appimages, thus my choices. each new build type means additional complexity, and a main philosophy of SN is to avoid complexity.

and thank you so much for providing all those technical details -- let us know when there are any new developments.

I'm starting the testing of your builds and will report back once I have enough experience with them.

other testers are welcome to join in!

[evs-ch]

In the meantime, I've prepared the necessary changes.

Basically the main problem with arm64 builds is node-sass. When building on arm64, for some reason node-gyp seems to pick up it's on an arm64 system, but still passes parameters to the compiler (-m64) which it cannot parse, causing it to fail.
To get around this, npm_config_target_arch=arm64 and npm_config_arch=arm64 need to be set.
This is potentially caused by the hard values in .npmrc.

That way, it compiles fine, only to fail during the first webpack invocation, which will complain of incompatible NODE_MODULE_VERSIONs. I've worked around that in the AUR package before by just forcing an npm rebuild outside of the main build scripts. The culprit here seems to be that the runtime option specified in .npmrc seems to be picked up when building the node-sass native extension by node-gyp, but it will not always be picked up by other steps / the webpack invocation, which will assume the node runtime that's used to build / run things is to be used.

To get around this, one could either set npm_config_target=$(node --version | sed -e 's/v\(\s*\)/\1/') and npm_config_runtime=node before running yarn so everything will work on the same assumptions. This might(!) cause issues elsewhere, so I've avoided that for now; even though it should only matter for native extensions (mainly node-sass, keytar, for example, is only used during packaging and will be handled by electron-builder directly).
Instead, we explicitely call npm_config_target=14.17.3 npm_config_runtime=node yarn --cwd ./web setup to limit it to ./web and then force an npm rebuild, just in case. This all seems not to be necessary on x64 because there's no native building involved and the downloaded binaries seem to be the right ones for the current node runtime.

With that done, the main stumbling block should be out of the way.

On that topic: there does seem to be a disconnect between the runtime / electron version specified in the .npmrc: it's electron 5 in there and 9 in the package.json – that should probably be reviewed?

In the current proposal I've prepared, I've defined separate steps/scripts in the package.json for arm64 builds (so a yarn bundle:arm64 gets the basics done to then allow building the individual targets (or all of them) with yarn build:dir:arm64, yarn build:appimage:arm64 etc.) – but remember that npm_config_target_arch=arm64 and npm_config_arch=arm64 need to be set.

Issues during packaging where: arm64 snap releases, as mentioned, cannot be cross compiled / packaged on an amd64 system. Also, snapcraft by default uses multipass to virtualize the build environment – which is not supported on arm64.
The solution here is to pass SNAPCRAFT_BUILD_ENVIRONMENT=host when running electron-builder. I've modified scripts/build.mjs to allow passing specific environment variables to runCommand/child_process.spawn so this is more or less automated.

Debian releases require usage of a native fpm installation, as otherwise an x64 binary will be downloaded – with no real way around it. USE_SYSTEM_FPM=true needs to be set, and an fpm binary needs to be in $PATH. On most systems / build environments, simply installing ruby and then running gem install fpm --no-document should be sufficient; although on some systems it might be necessary to add $GEM_HOME/bin manually to $PATH.

That's pretty much all there is to it, after a lot of trial and error and digging through docs ;)

As mentioned, cross compiling on an amd64 machine is possible, but only for appimage and dir targets. To get this done, the only requirement seems to be to set npm_config_target_arch=arm64. As node-sass is not used by the resulting packages, but only needed for preparation, the x64 binaries will be used – so there are no issues there.

Additionally, it's relatively simple to get things working with electron 11/12/13 as well:

There are two issues; one is that electron-builder will have issues with fetching (or (re)building) the arm64 keytar binary during packaging. This can be handled by using keytar version 7.7.0, for which electron-builder also needs to be updated to 22.11.1 (see atom/node-keytar#380 for the reasons behind that).

The other issue is that transpiling will fail. My knowledge is to limited to fully understand why, but it can be worked around by setting onlyTranspileTypescript and experimentalFeatures to true in webpack.prod.js.

The resulting build seems to run fine, but I'm not the best judge of that.

---

I've implemented those modifications at https://github.com/lsfxz/desktop/tree/arm64 and https://github.com/lsfxz/desktop/tree/arm64-electron (the latter is pretty much the same as the arm64 branch + the mentioned changes for electron 13).

As soon as it's verified that my testing builds from my previous post actually work as they should, I'll open PRs for those branches (or at least the arm64 one), so the standard notes team can see if those changes can be used in their build process – assuming running on arm64 is a possibility, as otherwise only a limited set of packages could be provided.

/edit

Maybe a tiny disclaimer: My understanding of the issues might be limited – I originally work in IT security and additionally code mainly in Ruby – so my node/electron/js knowledge is potentially lacking and my assumptions/understanding might be off ^^

[JaspalSuri]

On that topic: there does seem to be a disconnect between the runtime / electron version specified in the .npmrc: it's electron 5 in there and 9 in the package.json – that should probably be reviewed?

Thanks for bringing that to our attention! I've forwarded this over to the dev team for review. 🙂

[badrow]

@lsfxz I successfully tested your builds, it's working well ! Thank you !

I hope there will be no major changes and this version will stay compatible until an official version for arm64 will be released.

[lsfxz]

I successfully tested your builds, it's working well ! Thank you !
I hope there will be no major changes and this version will stay compatible until an official version for arm64 will be released.

That's great to hear!

I'm still ready and waiting to help getting things official, as soon as the standard notes team is ready :)

(btw: sorry for using the wrong github account in one of my comments here; sometimes I get them mixed up…)

[JaspalSuri]

I'm still ready and waiting to help getting things official, as soon as the standard notes team is ready :)

For what it's worth, the potential adoption would (currently) largely depend upon how involved the process of packaging each update would be.

[lsfxz]

Sure thing.

To sum up what my impression of that would be, as detailed a bit more elaborately in previous comments:

With the changes I've added to the fork, at least for AppImage, Debian and Dir targets it should be "not very involved", as things could even be cross-built on an amd64 system – so you'd only need to set an environment variable and run a different set of yarn build: commands.

Arch builds would probably be kept to the AUR anyway, I guess?

Snap builds would seem to require a dedicated arm64 build system (or at least an "emulated" one, like with qemu), but on such a build system, things would still be comparably simple to run.

From having maintained the AUR package for a while now, I'd also assume that additional workload for updates would be relatively low: except for some edge cases (like with the keytar version specific issue, where the proper binary cannot be downloaded with an older electron-builder), it's mostly "just working" – I've not had to change more substantial things (like required npm packages or something along that line), so there should be little potential for issues.

So I'd go out on a limb here and say that things should be very doable, and mostly depend on the availability of an arm64 build environment (for the snap releases, at least) and modifying your CI scripts, which I know too little about.

I'll update my fork to the current stable as soon as I find the time – maybe that might give a bit more insight into whether or not updates (smaller ones, at least) are a lot of trouble ^^

[lsfxz]

I've updated the fork (for 3.8.21) and tested the various build variations with it (except for the snap builds, as those were a bit more complicated, as I don't have an arm64-debian system lying around): Things work, build and launch fine with minimal changes: only @types/node needed to be updated, as otherwise tsc complained.

Only for the electron13 build another change was necessary (to avoid tsc assuming that something might be null in trayManager.ts) – which should probably be worked around cleaner, like with a proper check for null/undefined etc. at some point, I guess.

[chaserene]

@lsfxz, again, thank you a lot for these efforts.

I did some testing too and I'm happy to report that I found no big issues! so far I ran the Snap package you provided (I installed it with --dangerous as I saw no other way to do it, let me know if that was the right decision). right now I can't look into the AppImage, hopefully I'll get to it at a later time.

all functions and extensions (themes, editors, etc.) I can think of I am using worked as they should, all super stable. the only problem I experienced was that when I invoked the file picker (e.g. to redefine the automatic backup location), the file picker window displayed all characters as blank rectangles. this was on Manjaro. other than this, everything was good.

[evs-ch]

Admittedly, I don't know much about actually using (and installing) snap packages :D
If there's something that could be optimized when packaging it, let me know (having to use --dangerous sounds, well, dangerous ^^), as for now I've just used the default electron-builder way to create the snap package without further modification (besides getting it to package things for arm64), so there might be things that could be done about that.

Was the issue with the file picker on Manjaro with the package from the AUR (or the prebuilt .pkg.tar.zst from above), or was it with the snap release? Having Manjaro running on my Pinebook Pro as well, that would be a thing I could easily try to reproduce :)

Btw: I've uploaded the most recent test build artifacts as well:

https://privacyshark.zero-credibility.net/privacyshark/standard-notes-3.8.21-linux-arm64.AppImage (built on amd64)
https://privacyshark.zero-credibility.net/privacyshark/standard-notes-3.8.21-linux-arm64.deb (built on amd64 as well)
https://privacyshark.zero-credibility.net/privacyshark/standardnotes-desktop-3.8.21-2-aarch64.pkg.tar.zst (built on arm64, with electron13)

(no snap release for this testing run for now, as explained earlier – although I'll get one ready, should the need arise)

[chaserene]

as far as I remember, when I first tried to install your snap, snapd tried to retrieve a corresponding signature file from the Snap Store server, and complained because this snap (obviously) doesn't have one there. IIRC --dangerous is how you can sidestep that requirement. it may be possible to provide the signature as an auxiliary file, but maybe that was just a fever dream of mine.

the file picker character encoding issue was in the snap release.

[lsfxz]

Seems like it's indeed only possible to have a signed snap if it's in the/a snap store – so from my understanding at least, that's a problem that will go away once a snap release is made official and released to the snap store by the devs (assuming that's still planned, besides the other possible release types).

I'll take a look at the file picker issue soon :)

Oh, almost forgot: once I've looked at that and maybe cleaned up my forks a bit, I'd open a PR/PRs – would that make sense for the team, or would that be to early /untested for the standardnotes team?

/edit

Why, oh why do I always mix up my two github accounts? Again, sorry about that – I'll try not to do that anymore :D

[lsfxz]

So I did some digging into the snap issue. I could reproduce it on Manjaro with the arm64 snap – but I could also reproduce it on a regular (amd64) Arch installation with the official snap release from the snap store. It should be noted that both systems were using Wayland, so that might play a part in this.

It seems like it is a well known issue with (some) snaps on (some) different distros/systems for some users, without a clearly defined cause / way to address it; it might be solved by using newer versions of things (electron, etc.) and/or packaging an app with some default snap extensions (gnome etc.) and/or packaging it differently (ie. not with electron-builder) and/or manually providing the snapcraft.yml for packaging with some (not clearly explained anywhere) modifications. It's a bit murky.

There are quite a few workarounds mentioned all around the issue reports, but most of them actually didn't really work when I tested them. The only one that did was to rm -rf ~/snap/standard-notes/common/.cache/fontconfig and to then touch ~/snap/standard-notes/common/.cache/fontconfig – which, to my understanding, prevents snap / the app in the snap environment from using font caches. This made things look right, but the first time opening such a dialog after each launch of the app will take quite a bit longer than it should. Subsequent opening will be quick, though.

So that's probably not something that should be addressed here, as it seems to be more of a global-ish problem – I'll see if I can shed some more light on it and maybe find a way to address it during packaging, but I would not put too much energy into it, as it seems to be a bit out of scope of the arm64 issue in general. If I find something though, I'll certainly report it somewhere as well :)

Some links for reference:

https://wiki.archlinux.org/title/Snap#Text_unreadable
https://askubuntu.com/questions/1224125/font-characters-displayed-as-squares-in-ubuntu-18-04/1228809
https://forum.snapcraft.io/t/chromium-file-chooser-is-not-displaying-font/17021/5
https://bugs.launchpad.net/ubuntu/+source/chromium-browser/+bug/1864365
minbrowser/min#1251
https://discourse.ubuntu.com/t/snap-package-cannot-read-fonts-conf/15292/6
https://forum.inkdrop.app/t/snap-with-no-visible-letters-in-system-dialogs-on-fedora/2077/11
https://forum.snapcraft.io/t/snapped-app-not-loading-fonts-on-fedora-and-arch/12484/96
johannesjo/super-productivity#663
https://forum.snapcraft.io/t/missing-fonts-after-installing-signal-desktop-to-opensuse-tumbleweed/22102/2
VSCodium/vscodium#426
snapcrafters/codium#6
VSCodium/vscodium#767
fcole90/fah-snap#12
An especially old one: electron/electron#9616

[chaserene]

amazing digging @lsfxz, thank you! I agree that this is out-of-scope for this issue/bounty, so don't worry about it. if you have the time and feel like it, open an issue for this in the /desktop repo (I don't mean to push it on you, I just feel you're more competent to describe the problem than I am).

[moughxyz]

Merged: standardnotes/desktop#766

Thanks for the great work on this @lsfxz! And thanks for offering the bounty @chaserene. Did this ever get rewarded? I'm also not sure if @evs-ch is the same person as @lsfxz since they mentioned in their comment they made some changes on lsfxz's repo but I don't see any other commiters there.

[lsfxz]

@moughxyz Great, thanks for merging it!

Yep, @evs-ch is also me – I just accidentally made some comments while logged in with a work account instead of my separate / private one. I try to keep those apart, but every now and then I mess things up ;)

@chaserene absolutely did keep up their side of the offer and already rewarded the bounty quite a while ago. Thanks again for that – it was a great motivation to get things going on my end ;)

One last thing: If arm64 things cause issues in the future, feel free to get me involved again – I'd like to see this working "long-term", so I'd be happy to help.
Same goes for adding arm64 snap or macOS releases – as described in the PR, I did already work on that a bit, but I'd wait with putting in more time until I know it would be something that would actually be wanted by the team :)

/edit

He, scratch the macOS part – just saw the commit. Nice! :D

[chaserene]

thank you, both @lsfxz and @moughxyz, for making this possible, and I'm looking forward to using SN on a new platform 🙏

I felt obliged to reward @lsfxz early not only because of their swift reaction and past work on the build, but also because they are putting some amazing work into the Privacyshark repo, a privacy-centric app repo for Manjaro on ARM (the website seems to be dysfunctional now, but believe me, the repo is amazing).

[lsfxz]

(the website seems to be dysfunctional now, but believe me, the repo is amazing).

Thanks for making me aware of this! Looks like updating things to PHP 8 broke it. It's about time to switch to a fully static page with something like hugo instead! :D

In the meantime, it should be back up working again, for now!

[Lewiscowles1986]

So just a note that this does not build on all arm64. I'm using asahi linux for apple M1 and the failure I repeatedly get is

➤ YN0000: The command failed for workspaces that are depended upon by other workspaces; can't satisfy the dependency graph
➤ YN0000: Failed with errors in 16m 44s

Useless, not very clear. So I took a look at the github actions commands.

yarn run webpack --config desktop.webpack.prod.js
electron-builder --linux --arm64 -c.linux.target=dir --publish=never --c.extraMetadata.version=build

the electron-builder command is the one that fails.

⨯ Application entry file "dist/index.js" in the "/home/alarm/Projects/standardnotes/app/packages/desktop/dist/linux-arm64-unpacked/resources/app.asar" does not exist. Seems like a wrong configuration. failedTask=build stackTrace=Error: Application entry file "dist/index.js" in the "/home/alarm/Projects/standardnotes/app/packages/desktop/dist/linux-arm64-unpacked/resources/app.asar" does not exist. Seems like a wrong configuration.
at error (/home/alarm/Projects/standardnotes/app/packages/desktop/node_modules/app-builder-lib/src/asar/asarFileChecker.ts:7:12)
at Object.checkFileInArchive (/home/alarm/Projects/standardnotes/app/packages/desktop/node_modules/app-builder-lib/src/asar/asarFileChecker.ts:31:11)
at LinuxPackager.checkFileInPackage (/home/alarm/Projects/standardnotes/app/packages/desktop/node_modules/app-builder-lib/src/platformPackager.ts:503:7)
at LinuxPackager.sanityCheckPackage (/home/alarm/Projects/standardnotes/app/packages/desktop/node_modules/app-builder-lib/src/platformPackager.ts:551:5)
at LinuxPackager.doPack (/home/alarm/Projects/standardnotes/app/packages/desktop/node_modules/app-builder-lib/src/platformPackager.ts:312:5)
at LinuxPackager.pack (/home/alarm/Projects/standardnotes/app/packages/desktop/node_modules/app-builder-lib/src/platformPackager.ts:136:5)
at Packager.doBuild (/home/alarm/Projects/standardnotes/app/packages/desktop/node_modules/app-builder-lib/src/packager.ts:441:9)
at Object.executeFinally (/home/alarm/Projects/standardnotes/app/packages/desktop/node_modules/builder-util/src/promise.ts:12:14)
at Packager._build (/home/alarm/Projects/standardnotes/app/packages/desktop/node_modules/app-builder-lib/src/packager.ts:376:31)
at Packager.build (/home/alarm/Projects/standardnotes/app/packages/desktop/node_modules/app-builder-lib/src/packager.ts:337:12)
at Object.executeFinally (/home/alarm/Projects/standardnotes/app/packages/desktop/node_modules/builder-util/src/promise.ts:12:14)

I checked and the app.asar does exist. It's most frustrating that -v and --verbose both fail to get the command output of the manually run commands, or show any useful output at all.

[myreli]

It seems like arm64 builds have been consistently built as part of the CD pipeline and released.

I'm triaging older issues so I'll close this one, but please feel free to start a new discussion specific to the platform if you still need more help from the community. Thank you!