-
-
Notifications
You must be signed in to change notification settings - Fork 707
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Arm Architecture (Raspberry Pi) - Would it be possible ? #2083
Comments
I am still kinda surprised that this request comes up so much, given that 1 GB RAM spec. I don't know how much the OS and Sandstorm itself would use, but running Sandstorm grains average about 100 MB of RAM each. So, I suppose it'd theoretically work, but resources would be incredibly constrained. I also think earlier on MongoDB didn't work on ARM, which Sandstorm requires. I heard they got it working, allegedly, since. Given the other complications you mentioned above, note that Sandstorm is still very early in development, and it probably makes sense to get it stable on one architecture before adding more. @paulproteus: IMHO, there should be an FAQ question somewhere for "why doesn't Sandstorm currently support ARM?". It gets asked a lot. By definition, a frequently asked question. |
@kentonv from about a month ago: " Sandstorm will support ARM someday but it's going to require a large investment in tooling in order to be painless for developers." |
@ocdtrekkie did a pretty good job of explaining it already, but to summarize the technical reasons we don't plan to support ARM in the near future:
As a result of these factors, supporting Sandstorm on ARM is not a priority for us right now, though I agree it's a thing we want to work in the longer future. |
Just some observations:
|
Err.. I may have misunderstood what was being said about the ram; sorry for the noise :/ |
For some environments 1gb of RAM (on RPIs) could be enough. Everyone should agree that there is no such thing as running massive databases doing big data analysis or comparable operations on RPIs, but personally I see some usage in having container orchestration on these small ARM boards; e.g. I run container for all my little nifty home automatization scripts / tools on my RPIs. In fact, it shouldn't be that amount of work to optimize or strip down sandstorm for the arm64 architecture to keep it lightweight. @memento : let's fork sandstorm to arm64 and just leave the building of compatible arm64 sandstorm apps to the current app developers. |
A fork seems premature; the developers haven't said they're closed to the idea, just that it isn't a priority for them. @paulproteus expressed openness to it. It would be good to enumerate what actually needs to be done to get sandstorm to work on non x86_64 besides re-compiling. The mongodb issue is apparently solved. Here's what else I can think of, not being terribly familiar with the code:
Any other things people know of that would have to change? //cc @kentonv |
Re (2), perhaps worth looking at how various linux distros deal with this; I know debian has a fair amount of machinery, though I'm not intimately familiar with them. |
Hey guys. Chears, |
FWIW, people often find that 1GB of RAM is not enough for Sandstorm. It's pretty RAM-hungry since for some reason everyone wants to write apps in bulky garbage-collected dynamic languages like Ruby or Java. :) But yeah, I'm happy to merge patches that make Sandstorm build on ARM if you want to play around with it. I think it's pretty clear how we'd extend the low-level package format to support multi-arch. Take a look at the The hard part, though, is providing tools to the developers such that they're able to construct these packages. The current design of the tooling isn't exactly well-designed for this, since it sort of slurps up binaries off the host system. If we designed tools where the user specifies a more precise package list (e.g. from Debian or maybe Nix) then it would be a lot easier to grab the packages for multiple archs. I think work on such tooling could be a pretty interesting project that someone could easily own and work on without requiring lots of coordination with other parts of the project, if someone is interested! |
@memento: just get in contact, I'll open my SandstormArm64 repo this weekend. I currently ported just 20% of core sandstorm to arm64, it will take some more time before I / we can take a look into the toolchain. As stated before, I will open my repo which I just forgot right now, but I think refactorization of the toolchain is no dependency so I welcome everyone to commit in this part (propably as dedicated project), too! |
This is probably obvious to everyone in the discussion, but I suspect the Raspberry Pi 4 or 5 will have 2GB of RAM or more. And there are other single board computers only marginally more expensive than the Pi that have 2GB already. Any guesses as to whether that's sufficient for a Sandstorm installation? Or even then would users need to restrict the amount of concurrent applications they use? |
I've got sandstorm running on an x86_64 box with 2GiB of ram, so that's good enough for my own use. I'm pretty much the only user though. |
Yes, people have reported success with 2GB. Note that the amount of RAM you need is proportional to the number of active users -- or, really, the number of active grains. (We have plans to improve the situation someday using https://github.com/sandstorm-io/snappy-start but not sure when that will happen.) |
Quoting Kenton Varda (2017-01-27 02:35:06)
Take a look at the Archive type at the bottom of
package.capnp -- that's the actual package format.
Curious, what's the difference between that and the actual .spk file?
capnp decode tells me it's not a capnp message, and It's apparent from
the file sizes I've seen that there's some compression going on as well.
I think work on such tooling could be a pretty interesting project that
someone could easily own and work on without requiring lots of
coordination with other parts of the project, if someone is interested!
Might be up for it if I can find some time, but I can't really promise
much at the moment. I have a separate interest in reproducible builds as
well, so all of the magic in the current tooling is something that is
very much the opposite of what I want :/.
|
See the comments on stuff defined right above "Archive" -- the full file is actually two capnp messages ( |
i've meddled with ekam and building on rpi3 arm64 , tends to eat disk.... |
I have a khadas vim2 with 3gb of ram, and (if I can find the time) I'm going to try to get sandstorm running on it. |
BTW, Wekan is made with Meteor, and Wekan already runs on RasPi3 and RasPi4 on arm64 and any other Node.js/MongoDB supported CPU architecture: |
I think the largest problem is that even though a lot of dependencies for Sandstorm probably support ARM now, that we have a hard enough time getting apps packaged and updated for Sandstorm right now. Adding a CPU architecture would require all the apps get packaged for it as well. |
Yes, Sandstorm build pipeline work for current x64 architecture should be continued. |
And only after that think of other architectures. |
I think it's worth putting some thought into what implications multiarch has for the build pipeline, rather than get something working for x86_64 and hope it still makes sense elsewhere. I haven't investigated what multiarch looks like in the docker world, I should find some time to read up on where things are there and what that means for |
Yes, I really would like to figure out multiarch Wekan Docker and Snap. I have tried but I will sometime try to research more, I don't have it working yet. I don't know would is be possible to run Sandstorm in Docker or Snap. |
ekam will build on arm64 , which is needed for some of the building.. some time ago for captnproto https://github.com/capnproto/ekam |
I was thinking about this again recently. For dev tooling, we could likely make use of the kernel's binfmt_misc with qemu user mode to create a vagrant vm that can be used to develop arm packages on an x86_64 host. This way devs could use their own machines (even if they don't have access to an arm box) and much of the vagrant-spk workflow would be the same. This still leaves devs needing to test each arch manually though. |
Does something need building on arm64? I have access to bare metal arm64 sever that has 125 GB RAM and CPU that has 96 cores. |
@zenhack I think the biggest issue is with already having difficulty getting people to update their packages, asking them to make two packages and update both is pretty painful. My trauma from Windows on ARM-related travesties comes to mind. I'd want our vagrant-spk scripts to have well-documented methods for splitting out "on arm do this, on x64 do this", so one doesn't have to maintain fully separate scripts for each architecture, despite the fact that a lot of the build instructions are probably architecture-specific. Presumably we'd want vagrant-spk devs to pass an |
By default, meteor only supports x64 (and maybe x32). For Wekan, I first create x64 bundle .zip file, and then make changes to it with maintainer bash script: For arm64, MongoDB 3.x is installed from Ubuntu repos. For IBM Z Series mainframes, I only add fibers, and use it with Node and MongoDB for s390x: |
That arm64 server is running Ubuntu 18.04. There I created lxc container that has Ubuntu 20.04 and I'm currently trying to build Sandstorm in that container. |
|
Looks like it's downloading the x86 clang binary. Is there a corresponding arm binary to download, and can the make system autodetect which one to download? |
We should be able to just use the system clang. I think that got added
at some point because we were using some C++ features that were very new
at the time, and a lot of folks couldn't build sandstorm with their
system compilers.
We could probably rip that out and use the system compiler
unconditionally now without much trouble.
Quoting Adam Bliss (2020-04-27 17:32:51)
… Looks like it's downloading the x86 clang binary. Is there a
corresponding arm binary to download, and can the make system
autodetect which one to download?
|
Docker is making this quite simple these days: https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images |
FWIW the Clang binary that we download from the Chromium project is a multi-target compiler. It can cross-compile to arm. Ekam is capable of cross-compiling and at Cloudflare we regularly use it to cross-compile to arm64, at least. But if you want to run the build on arm (not cross-compile)... that's a different story, yeah. |
@xet7, I think it should work to just override |
So how can I either: For me ARM has been common for a long time. I have:
I would also like to cross compile to s390x, etc |
Getting Sandstorm working on arm will almost certainly require some code changes, e.g. in the seccomp sandbox. It's not going to be trivial. I don't know what will be involved, though, since I haven't tried it... |
What kind of changes you image there could be in the seccomp sandbox? How can I try it? |
seccomp filters are architecture-specific, because many syscalls differ across architectures. So it will probably need to be tweaked for the architecture. |
Where in Sandstorm source code are those architecture-specific syscalls? What syscalls are used? |
No, that's not what I mean. Many common syscalls, like The seccomp code is here: sandstorm/src/sandstorm/supervisor.c++ Lines 1085 to 1240 in 69336bf
|
I hate to be discouraging here, but I don't think an arm port is likely to pan out without a lot of work, and I don't think I have time to provide the assistance that would be needed from me to get through it. So I'd recommend leaving this on the shelf for now. |
Is there some existing lightweight sandboxing library or software that already works on multiple CPUs? That could be used instead, for example sandboxing Wekan and other apps? Or is Docker only option? Or alternatively, is sandboxing easier in some other programming language? |
Well, having Wekan run currently on arm64 and s390x is already complicated, because when adding some new dependencies those in some cases do not work on other than x64 without modifications. Currently I'm figuring out can ostrio-files run on those architectures, and reverting changes while figuring it out is in progress. |
It shouldn't be too hard to tweak the seccomp filter for any particular CPU if you know what you're doing -- maybe half an hour of work. Trying to integrate some other sandbox would be much, much harder. This is just meant to be one example of a reason why porting Sandstorm to arm is not trivial. There will be many, many other issues like this. |
Quoting Lauri Ojansivu (2020-05-25 06:38:40)
b) compile Sandstorm on ARM ?
Doing something like:
```
CC=clang CXX=clang++ make
```
Should get it to use the system's clang instead of the downloaded one,
which will at least get you past the failures from before.
As @kentonv mentions, Sandstorm will probably need code changes to work
correctly, but it will be interesting to see how far we can get.
Note that even if it appears to work correctly it may not be blocking
everything it should, so is not necessarily safe. We'll have to audit
and test.
…-Ian
|
Thanks! I will test it a little. |
I attempted the compilation process following @zenhack 's suggestion and this is the compilation log : This was attempted on a pinebook pro with an aarch64 distribution. Edit : I thought uploading a file would show it's content, here's the important part :
|
So I actually see two separate errors in that snippet. One has to do with the seccomp filter, which we shouldn't expect to work yet since the code isn't portable in the first place, but it also seems to be having problems finding the node JS headers. Ekam has the annoying property that it tends to cascade errors, so when you see "file not found" at some point in the log, it's likely that the real error is further up, where ekam failed to generate that file (though node.h should be pre-existing). I haven't had a chance to comb through the logs to figure out exaclty what went wrong. |
Hi everyone !
A lot of people are adopting the Raspberry Pi as a solution for self hosting.
I understand (if I get things right) that you guys had to choose an architecture once in for all for compiling the sandstorm application server and make it easy for users to download and install already compiled apps. I understand enabling a new architecture means compiling every version of sandstorm and apps that goes available (just like we do for linux distros and packages).
However, taking into account the growing "market share" (if you allow me) of arm platforms, I think it would be a good idea.
Of course, I'm willing to gather informations about your process/guidelines when it comes to compilation and release and to do this task, if you accept.
It would be great to have a sandstorm instance running on a Raspberry Pi 3 (for instance).
Here are the specs from www.raspberrypi.org
The text was updated successfully, but these errors were encountered: