Adds a Linux "desktop app"

[mz2]

Adds a Linux "desktop app" for AMD64 and ARM64 platforms that builds into it Livebook and all its dependencies, with similar Just Works Without Knowing How To Install All The Dependencies kind of intent to the Mac and Windows apps. Namely, it includes:

• The Livebook server
• A GTK / Rust based app indicator app made for the purpose (https://github.com/mz2/livebook-indicator) that should work in various popular Linux desktop environments that support libappindicator.
• Elixir 1.14 RC 1 (since I noticed 1.14 is a requirement on the main branch), built from source.
• Erlang OTP 25.0.4, from packages.erlang-solutions.com.
• git, wget, cmake that were noted in the Dockerfile as useful dependencies to have around depending on packages users would use (I have not really tested any of this yet).

The build configuration is right now for Livebook 0.6.3, but I have also tested building the current main branch head, and it is simple enough to make the built Livebook version configurable (it is also not necessary at all to keep this source in the Livebook repo btw, the source code is pulled by snapcraft at build time completely independently).

Known issues

• The indicator app's livebook server process management is, shall we say, quick and dirty. It really is intended to demonstrate the concept more so than to be production ready, so if you otherwise like this sort a solution for Linux, please advise how the OS processes for the app indicator and the server should be arranged (how should they communicate? should the server be a systemd managed service perhaps instead?).
• I don't know what I'm doing with the COOKIE file. I wrote it in the released build, but I'm not sure that's the way to do it?
• The Livebook icon is duplicated inside the repo, whereas it probably can just by symlinked in place.

Testing it out

The app is available as a test as an unlisted package on the Snap Store. It can be installed as follows on any Linux system with snapd installed:

sudo snap install livebook --edge

After installing, you can run the app either from the command line with the command livebook which should launch the browser window and insert a desktop launcher in your libappindicator enabled task bar of choice (GNOME, KDE, MATE, ...) or using the desktop launcher that the package installs.

You can also build the package locally using snapcraft (optionally using a LXD managed system container with the --use-lxd flag):

snapcraft # --use-lxd
sudo snap install ./livebook_0.6.3_amd64.snap --devmode # ... or arm64.snap if you built it on ARM64

If this package is something you'd be interested, I'm naturally happy to hand over the livebook app name on Snap Store for you to manage publishing the application and managing its metadata. I'm also happy to help automate the Snap publishing with some GitHub actions.

[CLAassistant]

All committers have signed the CLA.

[wojtekmach]

Thank you for starting the discussion!

[wojtekmach]

speaking of, we were thinking of including nmake.exe in Livebook for Windows exactly to be able to build certain packages but have concluded that nmake is not enough, it is really compiler toolchain and 3rd party libraries that are the sources of most friction so by shipping nmake we'd be fixing just a small part of the problem.

double checking, is cmake (and build-essential which we have above) enough to build torchx? Don't we need any extra CUDA stuff?

I didn't yet do comprehensive research on app bundle landscape but in the one article I just checked [1] they say:

Disadvantages
The main drawback of Snap apps is their size and slower startup compared to Flatpak or AppImage packages. Additionally, Snaps can only use the libraries included in the package.

(emphasis mine)

Does it mean that unless we add some libX ourselves to the Snap, users won't be able to use it in their notebooks?

[1] https://phoenixnap.com/kb/flatpak-vs-snap-vs-appimage

[mz2]

speaking of, we were thinking of including nmake.exe in Livebook for Windows exactly to be able to build certain packages but have concluded that nmake is not enough, it is really compiler toolchain and 3rd party libraries that are the sources of most friction so by shipping nmake we'd be fixing just a small part of the problem.

double checking, is cmake (and build-essential which we have above) enough to build torchx? Don't we need any extra CUDA stuff?

Hmm, needs testing, I must confess I did not test this and really just based those packages on the Dockerfile and a cursory attempt at loading something that was fetched using git. Is there perhaps a notebook around somewhere, or an Elixir fragment, that would include all kinds of imaginable tricky package fetching scenarios that should be handled?

Does it mean that unless we add some libX ourselves to the Snap, users won't be able to use it in their notebooks?

In short, not quite true.

Whilst I packaged this package "strictly confined", i.e. the applications inside it are executed in a containerised environment ("sandboxed") with their access to the host system limited to whatever interfaces are provided to them, and indeed filesystem access (including for system libraries) limited. The package is built with its own bunch of libraries and tools, this is true and it contributes to it being portable across Linux distributions and secure, but it also has locations under $SNAP_DATA and $SNAP_USER_DATA which are writable to it (and I certainly made an attempt to map for example $LIVEBOOK_DATA_PATH to one of these writable locations, hopefully an appropriate one for the purpose) and the network interface is also made available to the livebook server, so fetching data over the network is available to it. An analog I'd give is a Docker container with readwrite mounted volumes available to it (an application running in the container could indeed dynamically link libraries from volumes available to it).

So, the contained application could write for example dynamic libraries or executables to these locations it has access to (as a result of fetching some packages with mix for example) and then execute them, so it's in this sense not similar to the iOS and macOS sandbox where in addition to the host filesystem there's additional limitations to dynamically linking libraries at runtime and executable memory pages (~ JIT compilers).

(As far as snap startup speed, there's a lot of ongoing work on this, with this piece a good summary from earlier https://www.omgubuntu.co.uk/2022/05/ubuntu-making-firefox-snap-app-faster, and there's since been more improvements delivered still, some CPU microarchitecture agnostic, some for example ARM specific).

[wojtekmach]

I assume this is going to install whatever the current version of this package is at the time of building the release. Do you know if there's a way to explicitly specify a particular version? We'd ideally pin to a particular version that we use on the other OSes.

[mz2]

I assume this is going to install whatever the current version of this package is at the time of building the release.

Yes, correct. For all the rest of the packages this makes sense (since all the rest of the packages used for build & stage steps are long-term supported in Ubuntu 20.04, i.e. security fixes and critical bug fixes are implicitly picked up whenever the snap is built, and package versions are otherwise unmodified), but you've got a point ofc raising this for the esl-erlang which is picked up from the Erlang Solutions APT repository that receives new major versions.

Do you know if there's a way to explicitly specify a particular version? We'd ideally pin to a particular version that we use on the other OSes.

Yep, I think this is possible, by overriding the stage step and issuing an apt install or dpkg -i command there by hand. Is this the version that should be bundled?

[wojtekmach]

https://packages.erlang-solutions.com/erlang/debian/pool/esl-erlang_25.0.4-1~ubuntu~focal_amd64.deb

yes, pinning to a very particular version will I think ease the maintenance and avoid surprises. Let's go with 25.0.4.

[wojtekmach]

We're currently using wxwidgets for GUI things [1] and we'd be keen on keeping the same stack across all OSes. Do you think that is viable?

Besides the GUI part, the system tray, we need two other things from the OS, handling the "open file" and "open url" events:

• Mac: Info.plist.eex, Launcher.swift.eex
• Windows: Installer.vbs.eex, Launcher.vbs.eex

Can we support these features on Linux?

Speaking of the files above, the way it works right now is app_bundler takes this configuration and then we have per-OS implementation, using the files above. Do you think a similar model could work for Linux where we configure things in mix.exs and then app bundler does Linux-specific things?

Though per:

(it is also not necessary at all to keep this source in the Livebook repo btw, the source code is pulled by snapcraft at build time completely independently)

it sounds like all we need to expose is a public git repo which snapcraft will pull from. So it seems it is fundamentally different model than we have with app bundler?

[1] https://github.com/livebook-dev/livebook/blob/v0.6.3/lib/livebook_app.ex

(all of the links above are for v0.6.3 tag so slightly outdated by now but there were no fundamental changes since.)

[mz2]

Speaking of the files above, the way it works right now is app_bundler takes this configuration and then we have per-OS implementation, using the files above. Do you think a similar model could work for Linux where we configure things in mix.exs and then app bundler does Linux-specific things?

I think this part is indeed pretty doable. I think the part where I'm less clear on is whether the wxwidgets based underlying implementation would work so well on Linux -- if you prefer to share that implementation, I might need some hand holding. 😄

Besides the GUI part, the system tray, we need two other things from the OS, handling the "open file" and "open url" events

Yep, I think this should be pretty simple. I can investigate this on top of the current indicator solution if the Rust / GTK route seems sane to you.

it sounds like all we need to expose is a public git repo which snapcraft will pull from. So it seems it is fundamentally different model than we have with app bundler?

To be precise, snapcraft allows for a fundamentally different model but does not require it. I suppose this source-type: git mode of fetching the source is motivated by the common need to support the publisher / packager and upstream being different entities (makes it possible, but not necessary, to keep the snap packaging artefacts in a separate repo from the upstream source being packaged). It's also possible to refer to the source locally from the same repository instead of using git (by changing source-type: local in snapcraft.yaml in the livebook part), and then snapcraft would at build time instead use the directory containing the snap directory (i.e. the root of the repo in this case) as the root source directory that is made available to the build process.

I made a separate indicator app instead of investigating Linux enabling the app bundler route mostly since a) on a cursory look the wxwidgets system tray support is for the legacy system tray protocol that libappindicator supersedes and b) I'm no expert on the subject and the libappindicator / gtk + Rust route involved less abstractions between me and making an indicator dingus show up 😄

[wojtekmach]

I can investigate this on top of the current indicator solution if the Rust / GTK route seems sane to you.

We'd prefer not to take the path of separate Linux-specific codebase for parts of the functionality. All the caveats aside, wx is still appealing to us exactly because we have a single tech stack that abstracts OS-specific details. Something we might consider in the future is going all in on something like Tauri (or putting together some of its underlying components or implementing more things ourselves in Rust or others)

Personally I think https://github.com/mz2/livebook-indicator is really cool and I think there could be many more apps using it - all they need is a system tray indicator that controls a server that runs in the background. For Livebook specifically I'd personally consider dropping our wx based solution in favour of this if it would support Mac & Windows too.

All that being said just letting you know that we're also investigating WebView-based solutions (so that we could have an iPad app) so we might be taking a very different approach in the future after all.

[mz2]

We'd prefer not to take the path of separate Linux-specific codebase for parts of the functionality.

I understand that, and I think for this kind of pretty spartan desktop integration purpose that makes perfect sense, a separate codebase is not appealing to maintain. To recap I pretty much built that indicator part indeed more or less as a proof of concept, and I knew wx specifically on Linux has the limitation I've raised so it didn't seem like the PoC route at least.

Another technology that might fit your needs, with some arguable benefits as far as crossplatform story at least, is Flutter. For example, there exists a sys tray package for it (https://pub.dev/packages/system_tray) and the file and URL opening bits are also very straightforward with it... and there are webview packages also available for it across the platforms (including indeed iPad). It also has a pretty good model for mixing and matching native componentry (platform views), which comes very handy with mobile.

For Livebook specifically I'd personally consider dropping our wx based solution in favour of this if it would support Mac & Windows too.

Challenge accepted 😆 I'll take a look. I know it'd be super easy to make it work on macOS as well. I've built a lot of stuff on macOS in the past and know that the necessary platform integration is available in a Rust bound way -- I'm less experienced on Windows but I do have it available somewhere (🕸️).

I've done some further investigating for the URL and file opening part of this as well -- that's another area, much like the app indicator being presented, where on Linux (as on macOS and Windows) you need some platform integration that I think wx doesn't give itself (and you've scoped the launcher basically in part to do that, right?): DBus activation of the application is basically the way to launch URLs to it (and to guarantee uniqueness of the launched application). I think this bit might indeed be simplest at least to be done in gtk-rs as a tiny launcher app because it's doable with very little code (basically similar level of complexity to how macOS does it).

[wojtekmach]

is it possible to use the ./rel/app/icon.png path instead? Or we'd need to symlink it?

[mz2]

In short yes, it's totally possible to make the build here use as the source resource ./rel/app/icon.png instead of the assets/livebook-icon.png that I copied in to the indicator app's separate source code repository. I can investigate this if it seems like a good idea to keep the Rust / GTK based indicator at all.

As an aside, I stuck the indicator app in a separate repo for now indeed because I meant to communicate that how it's tackled is kind of a separate sub problem from packaging some sort of an indicator and Livebook + its dependencies together into a package, and indeed assumed you might want to share more of the tech choices for Linux that you used for macOS and Windows. Some challenges with that are discussed above (I'm no expert on the topic, I may have got my facts about wxwidgets also wrong).

[josevalim]

It pains to say this but it is impractical for a small team like ours to develop and maintain Livebook Desktop for multiple platforms. Therefore, the simplest way is for us to try to use a cross-platform toolkit.

This was our first attempt but WxWidgets fell short of this goal. So we wrote a small kit around Swift and DotNet. Therefore, the best option to get Linux support is by running either the Swift or dotnet toolkits on Linux (and it seems the dotnet one is the most accessible thanks to mono).

So if anyeone wants to give us a hand, you can get started by trying to run the dotnet demo app on Linux. It may currently use some Windows APIs but our hope is that they can be replaced by https://www.mono-project.com/docs/gui/gtksharp/.

Thanks for the PR, the insights, and we will be glad to answer further questions.

[mz2]

That makes a lot of sense (and DotNet sounds more practical).