Arm Architecture (Raspberry Pi) - Would it be possible ?

[memento]

Hi everyone !

A lot of people are adopting the Raspberry Pi as a solution for self hosting.
I understand (if I get things right) that you guys had to choose an architecture once in for all for compiling the sandstorm application server and make it easy for users to download and install already compiled apps. I understand enabling a new architecture means compiling every version of sandstorm and apps that goes available (just like we do for linux distros and packages).
However, taking into account the growing "market share" (if you allow me) of arm platforms, I think it would be a good idea.
Of course, I'm willing to gather informations about your process/guidelines when it comes to compilation and release and to do this task, if you accept.

It would be great to have a sandstorm instance running on a Raspberry Pi 3 (for instance).
Here are the specs from www.raspberrypi.org

Specifications
SoC: Broadcom BCM2837
CPU: 4× ARM Cortex-A53, 1.2GHz
GPU: Broadcom VideoCore IV
RAM: 1GB LPDDR2 (900 MHz)
Networking: 10/100 Ethernet, 2.4GHz 802.11n wireless
Bluetooth: Bluetooth 4.1 Classic, Bluetooth Low Energy
Storage: microSD
GPIO: 40-pin header, populated
Ports: HDMI, 3.5mm analogue audio-video jack, 4× USB 2.0, Ethernet, Camera Serial Interface (CSI), Display Serial Interface (DSI)

[ocdtrekkie]

I am still kinda surprised that this request comes up so much, given that 1 GB RAM spec. I don't know how much the OS and Sandstorm itself would use, but running Sandstorm grains average about 100 MB of RAM each. So, I suppose it'd theoretically work, but resources would be incredibly constrained.

I also think earlier on MongoDB didn't work on ARM, which Sandstorm requires. I heard they got it working, allegedly, since.

Given the other complications you mentioned above, note that Sandstorm is still very early in development, and it probably makes sense to get it stable on one architecture before adding more.

@paulproteus: IMHO, there should be an FAQ question somewhere for "why doesn't Sandstorm currently support ARM?". It gets asked a lot. By definition, a frequently asked question.

[ocdtrekkie]

@kentonv from about a month ago: " Sandstorm will support ARM someday but it's going to require a large investment in tooling in order to be painless for developers."

[zarvox]

@ocdtrekkie did a pretty good job of explaining it already, but to summarize the technical reasons we don't plan to support ARM in the near future:

• MongoDB (which Meteor and Sandstorm rely on) doesn't officially support ARM yet: https://jira.mongodb.org/browse/SERVER-1811
• Since Sandstorm packages contain native code, either app developers would need to build packages for every platform supported, or we'd need to have some way to automate building these packages, lest we wind up with significant app ecosystem fragmentation. Building that tooling would take time and attention from other things which we think will better serve a greater number of users right now.

As a result of these factors, supporting Sandstorm on ARM is not a priority for us right now, though I agree it's a thing we want to work in the longer future.

[paulproteus]

Hi @memento ,

I wrote up some docs in #2432 about the status quo.

Are you still possibly interested in working on this? If so, then I'm happy to discuss details.

Let me know what you think!

[zenhack]

Just some observations:

• 1 GiB of memory in an arm system is actually pretty easy to come by these days.
• Memory usage would probably be a bit lower anyway on a 32-bit platform.

[zenhack]

Err.. I may have misunderstood what was being said about the ram; sorry for the noise :/

[3mbr4c3]

For some environments 1gb of RAM (on RPIs) could be enough. Everyone should agree that there is no such thing as running massive databases doing big data analysis or comparable operations on RPIs, but personally I see some usage in having container orchestration on these small ARM boards; e.g. I run container for all my little nifty home automatization scripts / tools on my RPIs. In fact, it shouldn't be that amount of work to optimize or strip down sandstorm for the arm64 architecture to keep it lightweight.

@memento : let's fork sandstorm to arm64 and just leave the building of compatible arm64 sandstorm apps to the current app developers.

[zenhack]

A fork seems premature; the developers haven't said they're closed to the idea, just that it isn't a priority for them. @paulproteus expressed openness to it.

It would be good to enumerate what actually needs to be done to get sandstorm to work on non x86_64 besides re-compiling. The mongodb issue is apparently solved. Here's what else I can think of, not being terribly familiar with the code:

1. I suspect the seccomp-bpf rules would have to be extended for each arch, since rules match per-arch. I imagine this is a fairly mechanical change though.
2. Per @zarvox's second point, we'll need to figure out how to deal with the possibility of native code for different CPUs. I'm sure this is soluable, but is going to require a bit of design work. I suspect the most challenging part of this is to find a way to make it easy for developers to build apps for different architectures.

Any other things people know of that would have to change?

//cc @kentonv

[zenhack]

Re (2), perhaps worth looking at how various linux distros deal with this; I know debian has a fair amount of machinery, though I'm not intimately familiar with them.

[memento]

@paulproteus @3mbr4c3

Hey guys.
Still interrested in that plaform (a lot).
I'm currently deep into freelancing right now. As soon as I earn enough to finish the year (the sinews of war) I'ld like to participate on this task (a few months I guess).

Chears,

[kentonv]

FWIW, people often find that 1GB of RAM is not enough for Sandstorm. It's pretty RAM-hungry since for some reason everyone wants to write apps in bulky garbage-collected dynamic languages like Ruby or Java. :)

But yeah, I'm happy to merge patches that make Sandstorm build on ARM if you want to play around with it.

I think it's pretty clear how we'd extend the low-level package format to support multi-arch. Take a look at the Archive type at the bottom of package.capnp -- that's the actual package format. We could add a way for a file to have multiple versions targeting different archs, and then the unpacker would only extract the one for the target arch.

The hard part, though, is providing tools to the developers such that they're able to construct these packages. The current design of the tooling isn't exactly well-designed for this, since it sort of slurps up binaries off the host system. If we designed tools where the user specifies a more precise package list (e.g. from Debian or maybe Nix) then it would be a lot easier to grab the packages for multiple archs. I think work on such tooling could be a pretty interesting project that someone could easily own and work on without requiring lots of coordination with other parts of the project, if someone is interested!

[3mbr4c3]

@memento: just get in contact, I'll open my SandstormArm64 repo this weekend.

I currently ported just 20% of core sandstorm to arm64, it will take some more time before I / we can take a look into the toolchain. As stated before, I will open my repo which I just forgot right now, but I think refactorization of the toolchain is no dependency so I welcome everyone to commit in this part (propably as dedicated project), too!

[Michael-S]

This is probably obvious to everyone in the discussion, but I suspect the Raspberry Pi 4 or 5 will have 2GB of RAM or more. And there are other single board computers only marginally more expensive than the Pi that have 2GB already. Any guesses as to whether that's sufficient for a Sandstorm installation? Or even then would users need to restrict the amount of concurrent applications they use?

[zenhack]

I've got sandstorm running on an x86_64 box with 2GiB of ram, so that's good enough for my own use. I'm pretty much the only user though.