Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Arm Architecture (Raspberry Pi) - Would it be possible ? #2083

Open
memento opened this issue Jun 12, 2016 · 51 comments
Open

Arm Architecture (Raspberry Pi) - Would it be possible ? #2083

memento opened this issue Jun 12, 2016 · 51 comments

Comments

@memento
Copy link

memento commented Jun 12, 2016

Hi everyone !

A lot of people are adopting the Raspberry Pi as a solution for self hosting.
I understand (if I get things right) that you guys had to choose an architecture once in for all for compiling the sandstorm application server and make it easy for users to download and install already compiled apps. I understand enabling a new architecture means compiling every version of sandstorm and apps that goes available (just like we do for linux distros and packages).
However, taking into account the growing "market share" (if you allow me) of arm platforms, I think it would be a good idea.
Of course, I'm willing to gather informations about your process/guidelines when it comes to compilation and release and to do this task, if you accept.

It would be great to have a sandstorm instance running on a Raspberry Pi 3 (for instance).
Here are the specs from www.raspberrypi.org

Specifications
SoC: Broadcom BCM2837
CPU: 4× ARM Cortex-A53, 1.2GHz
GPU: Broadcom VideoCore IV
RAM: 1GB LPDDR2 (900 MHz)
Networking: 10/100 Ethernet, 2.4GHz 802.11n wireless
Bluetooth: Bluetooth 4.1 Classic, Bluetooth Low Energy
Storage: microSD
GPIO: 40-pin header, populated
Ports: HDMI, 3.5mm analogue audio-video jack, 4× USB 2.0, Ethernet, Camera Serial Interface (CSI), Display Serial Interface (DSI)

@ocdtrekkie
Copy link
Collaborator

I am still kinda surprised that this request comes up so much, given that 1 GB RAM spec. I don't know how much the OS and Sandstorm itself would use, but running Sandstorm grains average about 100 MB of RAM each. So, I suppose it'd theoretically work, but resources would be incredibly constrained.

I also think earlier on MongoDB didn't work on ARM, which Sandstorm requires. I heard they got it working, allegedly, since.

Given the other complications you mentioned above, note that Sandstorm is still very early in development, and it probably makes sense to get it stable on one architecture before adding more.

@paulproteus: IMHO, there should be an FAQ question somewhere for "why doesn't Sandstorm currently support ARM?". It gets asked a lot. By definition, a frequently asked question.

@ocdtrekkie
Copy link
Collaborator

@kentonv from about a month ago: " Sandstorm will support ARM someday but it's going to require a large investment in tooling in order to be painless for developers."

@zarvox
Copy link
Collaborator

zarvox commented Jun 13, 2016

@ocdtrekkie did a pretty good job of explaining it already, but to summarize the technical reasons we don't plan to support ARM in the near future:

  • MongoDB (which Meteor and Sandstorm rely on) doesn't officially support ARM yet: https://jira.mongodb.org/browse/SERVER-1811
  • Since Sandstorm packages contain native code, either app developers would need to build packages for every platform supported, or we'd need to have some way to automate building these packages, lest we wind up with significant app ecosystem fragmentation. Building that tooling would take time and attention from other things which we think will better serve a greater number of users right now.

As a result of these factors, supporting Sandstorm on ARM is not a priority for us right now, though I agree it's a thing we want to work in the longer future.

@paulproteus
Copy link
Collaborator

Hi @memento ,

I wrote up some docs in #2432 about the status quo.

Are you still possibly interested in working on this? If so, then I'm happy to discuss details.

Let me know what you think!

@zenhack
Copy link
Collaborator

zenhack commented Sep 18, 2016

Just some observations:

  • 1 GiB of memory in an arm system is actually pretty easy to come by these days.
  • Memory usage would probably be a bit lower anyway on a 32-bit platform.

@zenhack
Copy link
Collaborator

zenhack commented Sep 18, 2016

Err.. I may have misunderstood what was being said about the ram; sorry for the noise :/

@3mbr4c3
Copy link

3mbr4c3 commented Jan 21, 2017

For some environments 1gb of RAM (on RPIs) could be enough. Everyone should agree that there is no such thing as running massive databases doing big data analysis or comparable operations on RPIs, but personally I see some usage in having container orchestration on these small ARM boards; e.g. I run container for all my little nifty home automatization scripts / tools on my RPIs. In fact, it shouldn't be that amount of work to optimize or strip down sandstorm for the arm64 architecture to keep it lightweight.

@memento : let's fork sandstorm to arm64 and just leave the building of compatible arm64 sandstorm apps to the current app developers.

@zenhack
Copy link
Collaborator

zenhack commented Jan 21, 2017

A fork seems premature; the developers haven't said they're closed to the idea, just that it isn't a priority for them. @paulproteus expressed openness to it.

It would be good to enumerate what actually needs to be done to get sandstorm to work on non x86_64 besides re-compiling. The mongodb issue is apparently solved. Here's what else I can think of, not being terribly familiar with the code:

  1. I suspect the seccomp-bpf rules would have to be extended for each arch, since rules match per-arch. I imagine this is a fairly mechanical change though.
  2. Per @zarvox's second point, we'll need to figure out how to deal with the possibility of native code for different CPUs. I'm sure this is soluable, but is going to require a bit of design work. I suspect the most challenging part of this is to find a way to make it easy for developers to build apps for different architectures.

Any other things people know of that would have to change?

//cc @kentonv

@zenhack
Copy link
Collaborator

zenhack commented Jan 21, 2017

Re (2), perhaps worth looking at how various linux distros deal with this; I know debian has a fair amount of machinery, though I'm not intimately familiar with them.

@memento
Copy link
Author

memento commented Jan 25, 2017

@paulproteus @3mbr4c3

Hey guys.
Still interrested in that plaform (a lot).
I'm currently deep into freelancing right now. As soon as I earn enough to finish the year (the sinews of war) I'ld like to participate on this task (a few months I guess).

Chears,

@kentonv
Copy link
Member

kentonv commented Jan 27, 2017

FWIW, people often find that 1GB of RAM is not enough for Sandstorm. It's pretty RAM-hungry since for some reason everyone wants to write apps in bulky garbage-collected dynamic languages like Ruby or Java. :)

But yeah, I'm happy to merge patches that make Sandstorm build on ARM if you want to play around with it.

I think it's pretty clear how we'd extend the low-level package format to support multi-arch. Take a look at the Archive type at the bottom of package.capnp -- that's the actual package format. We could add a way for a file to have multiple versions targeting different archs, and then the unpacker would only extract the one for the target arch.

The hard part, though, is providing tools to the developers such that they're able to construct these packages. The current design of the tooling isn't exactly well-designed for this, since it sort of slurps up binaries off the host system. If we designed tools where the user specifies a more precise package list (e.g. from Debian or maybe Nix) then it would be a lot easier to grab the packages for multiple archs. I think work on such tooling could be a pretty interesting project that someone could easily own and work on without requiring lots of coordination with other parts of the project, if someone is interested!

@3mbr4c3
Copy link

3mbr4c3 commented Jan 27, 2017

@memento: just get in contact, I'll open my SandstormArm64 repo this weekend.

I currently ported just 20% of core sandstorm to arm64, it will take some more time before I / we can take a look into the toolchain. As stated before, I will open my repo which I just forgot right now, but I think refactorization of the toolchain is no dependency so I welcome everyone to commit in this part (propably as dedicated project), too!

@Michael-S
Copy link
Contributor

This is probably obvious to everyone in the discussion, but I suspect the Raspberry Pi 4 or 5 will have 2GB of RAM or more. And there are other single board computers only marginally more expensive than the Pi that have 2GB already. Any guesses as to whether that's sufficient for a Sandstorm installation? Or even then would users need to restrict the amount of concurrent applications they use?

@zenhack
Copy link
Collaborator

zenhack commented Jan 27, 2017

I've got sandstorm running on an x86_64 box with 2GiB of ram, so that's good enough for my own use. I'm pretty much the only user though.

@kentonv
Copy link
Member

kentonv commented Jan 27, 2017

Yes, people have reported success with 2GB. Note that the amount of RAM you need is proportional to the number of active users -- or, really, the number of active grains. (We have plans to improve the situation someday using https://github.com/sandstorm-io/snappy-start but not sure when that will happen.)

@zenhack
Copy link
Collaborator

zenhack commented Jan 27, 2017 via email

@kentonv
Copy link
Member

kentonv commented Jan 28, 2017

Curious, what's the difference between that and the actual .spk file?

See the comments on stuff defined right above "Archive" -- the full file is actually two capnp messages (Signature and Archive), xz-compressed, with an 8-byte "magic number" prefix.

@necrose99
Copy link

necrose99 commented Jun 9, 2018

ekam_bin_arm64.zip

i've meddled with ekam and building on rpi3 arm64 , tends to eat disk....
however looking to get a rockpro64 for home soon , could try to build on cloud.scaleaway. com .. as i have 8 gigs... of ram etc...

@abliss
Copy link
Contributor

abliss commented Jun 25, 2018

I have a khadas vim2 with 3gb of ram, and (if I can find the time) I'm going to try to get sandstorm running on it.

@xet7
Copy link
Contributor

xet7 commented Dec 2, 2019

BTW, Wekan is made with Meteor, and Wekan already runs on RasPi3 and RasPi4 on arm64 and any other Node.js/MongoDB supported CPU architecture:

@ocdtrekkie
Copy link
Collaborator

I think the largest problem is that even though a lot of dependencies for Sandstorm probably support ARM now, that we have a hard enough time getting apps packaged and updated for Sandstorm right now. Adding a CPU architecture would require all the apps get packaged for it as well.

@xet7
Copy link
Contributor

xet7 commented Dec 2, 2019

@ocdtrekkie

Yes, Sandstorm build pipeline work for current x64 architecture should be continued.

@xet7
Copy link
Contributor

xet7 commented Dec 2, 2019

And only after that think of other architectures.

@zenhack
Copy link
Collaborator

zenhack commented Dec 2, 2019

I think it's worth putting some thought into what implications multiarch has for the build pipeline, rather than get something working for x86_64 and hope it still makes sense elsewhere. I haven't investigated what multiarch looks like in the docker world, I should find some time to read up on where things are there and what that means for docker-spk's prospects of being able to handle this.

@xet7
Copy link
Contributor

xet7 commented Dec 2, 2019

@zenhack

Yes, I really would like to figure out multiarch Wekan Docker and Snap. I have tried but I will sometime try to research more, I don't have it working yet.

I don't know would is be possible to run Sandstorm in Docker or Snap.

@necrose99
Copy link

necrose99 commented Dec 5, 2019

ekam will build on arm64 , which is needed for some of the building.. some time ago for captnproto
however for rpi3 or less , you may need to make -j2 1 etc lowering memory foot print.
{however (https://www.scaleway.com/en/virtual-instances/arm-instances/)
debian native , 8-16 gigs of ram should be easy to build and exportbuilt artifacts ... to RPI etc... }

https://github.com/capnproto/ekam
"Ekam's primary user and maintainer right now (Sandstorm.io) is itself highly Linux-specific, so there is not much pressure. (Let us know if you want to help.)"

@zenhack
Copy link
Collaborator

zenhack commented Apr 27, 2020

I was thinking about this again recently. For dev tooling, we could likely make use of the kernel's binfmt_misc with qemu user mode to create a vagrant vm that can be used to develop arm packages on an x86_64 host. This way devs could use their own machines (even if they don't have access to an arm box) and much of the vagrant-spk workflow would be the same. This still leaves devs needing to test each arch manually though.

@xet7
Copy link
Contributor

xet7 commented Apr 27, 2020

Does something need building on arm64? I have access to bare metal arm64 sever that has 125 GB RAM and CPU that has 96 cores.

@ocdtrekkie
Copy link
Collaborator

@zenhack I think the biggest issue is with already having difficulty getting people to update their packages, asking them to make two packages and update both is pretty painful. My trauma from Windows on ARM-related travesties comes to mind.

I'd want our vagrant-spk scripts to have well-documented methods for splitting out "on arm do this, on x64 do this", so one doesn't have to maintain fully separate scripts for each architecture, despite the fact that a lot of the build instructions are probably architecture-specific. Presumably we'd want vagrant-spk devs to pass an arm argument somewhere in their vagrant-spk commands to trigger testing and packing the ARM version of a given package.

@xet7
Copy link
Contributor

xet7 commented Apr 27, 2020

By default, meteor only supports x64 (and maybe x32). For Wekan, I first create x64 bundle .zip file, and then make changes to it with maintainer bash script:
https://releases.wekan.team/raspi3/

For arm64, MongoDB 3.x is installed from Ubuntu repos.

For IBM Z Series mainframes, I only add fibers, and use it with Node and MongoDB for s390x:
https://github.com/wekan/wekan/wiki/s390x#install

@xet7
Copy link
Contributor

xet7 commented Apr 27, 2020

That arm64 server is running Ubuntu 18.04. There I created lxc container that has Ubuntu 20.04 and I'm currently trying to build Sandstorm in that container.

@xet7
Copy link
Contributor

xet7 commented Apr 27, 2020

==== compiling bootstrap ekam ====
g++ -Isrc -std=c++14 -pthread tmp/base/Promise.o tmp/base/sha256.o tmp/base/Debug.o tmp/base/OwnedPtr.o tmp/base/Hash.o tmp/os/ByteStream.o tmp/os/EventManager.o tmp/os/DiskFile.o tmp/os/Socket.o tmp/os/EpollEventManager.o tmp/os/Subprocess.o tmp/os/OsHandle.o tmp/os/File.o tmp/os/EventGroup.o tmp/ekam/initNetworkDashboardStub.o tmp/ekam/CppActionFactory.o tmp/ekam/SimpleDashboard.o tmp/ekam/Tag.o tmp/ekam/ekam.o tmp/ekam/ExecPluginActionFactory.o tmp/ekam/MuxDashboard.o tmp/ekam/Driver.o tmp/ekam/ConsoleDashboard.o tmp/ekam/ActionUtil.o tmp/ekam/Action.o tmp/ekam/Dashboard.o -o bin/ekam-bootstrap
make[1]: Leaving directory '/home/wekan/repos/sandstorm/deps/ekam'
==== downloading Clang binaries from Chromium project ====
Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Linux_x64/clang-n346557-4e0d9925-2.tgz .......... Done.
==== configuring BoringSSL ====
cd deps/boringssl/build && cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="/home/wekan/repos/sandstorm/deps/llvm-build/Release+Asserts/bin/clang" -DCMAKE_CXX_COMPILER="/home/wekan/repos/sandstorm/deps/llvm-build/Release+Asserts/bin/clang++" -DCMAKE_C_FLAGS="-fPIE" -DCMAKE_CXX_FLAGS="-fPIE" ..
-- The C compiler identification is unknown
-- Check for working C compiler: /home/wekan/repos/sandstorm/deps/llvm-build/Release+Asserts/bin/clang
-- Check for working C compiler: /home/wekan/repos/sandstorm/deps/llvm-build/Release+Asserts/bin/clang -- broken
CMake Error at /usr/share/cmake-3.16/Modules/CMakeTestCCompiler.cmake:60 (message):
  The C compiler

    "/home/wekan/repos/sandstorm/deps/llvm-build/Release+Asserts/bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /home/wekan/repos/sandstorm/deps/boringssl/build/CMakeFiles/CMakeTmp
    
    Run Build Command(s):/usr/bin/make cmTC_89ea3/fast && make[1]: Entering directory '/home/wekan/repos/sandstorm/deps/boringssl/build/CMakeFiles/CMakeTmp'
    /usr/bin/make -f CMakeFiles/cmTC_89ea3.dir/build.make CMakeFiles/cmTC_89ea3.dir/build
    make[2]: Entering directory '/home/wekan/repos/sandstorm/deps/boringssl/build/CMakeFiles/CMakeTmp'
    Building C object CMakeFiles/cmTC_89ea3.dir/testCCompiler.c.o
    /home/wekan/repos/sandstorm/deps/llvm-build/Release+Asserts/bin/clang   -fPIE    -o CMakeFiles/cmTC_89ea3.dir/testCCompiler.c.o   -c /home/wekan/repos/sandstorm/deps/boringssl/build/CMakeFiles/CMakeTmp/testCCompiler.c
    make[2]: /home/wekan/repos/sandstorm/deps/llvm-build/Release+Asserts/bin/clang: Command not found
    make[2]: *** [CMakeFiles/cmTC_89ea3.dir/build.make:66: CMakeFiles/cmTC_89ea3.dir/testCCompiler.c.o] Error 127
    make[2]: Leaving directory '/home/wekan/repos/sandstorm/deps/boringssl/build/CMakeFiles/CMakeTmp'
    make[1]: *** [Makefile:121: cmTC_89ea3/fast] Error 2
    make[1]: Leaving directory '/home/wekan/repos/sandstorm/deps/boringssl/build/CMakeFiles/CMakeTmp'
    
 
  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:14 (enable_language)


-- Configuring incomplete, errors occurred!
See also "/home/wekan/repos/sandstorm/deps/boringssl/build/CMakeFiles/CMakeOutput.log".
See also "/home/wekan/repos/sandstorm/deps/boringssl/build/CMakeFiles/CMakeError.log".
make: *** [Makefile:236: deps/boringssl/build/Makefile] Error 1
wekan@sand:~/repos/sandstorm$ uname -a
Linux sand 4.15.0-36-generic #39-Ubuntu SMP Wed Sep 26 06:05:44 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux
wekan@sand:~/repos/sandstorm$ cat /etc/issue
Ubuntu 20.04 LTS \n \l

@abliss
Copy link
Contributor

abliss commented Apr 27, 2020

Looks like it's downloading the x86 clang binary. Is there a corresponding arm binary to download, and can the make system autodetect which one to download?

@zenhack
Copy link
Collaborator

zenhack commented Apr 27, 2020 via email

@yajo
Copy link

yajo commented May 8, 2020

I haven't investigated what multiarch looks like in the docker world, I should find some time to read up on where things are there and what that means for docker-spk's prospects of being able to handle this.

Docker is making this quite simple these days: https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images

@kentonv
Copy link
Member

kentonv commented May 25, 2020

FWIW the Clang binary that we download from the Chromium project is a multi-target compiler. It can cross-compile to arm. Ekam is capable of cross-compiling and at Cloudflare we regularly use it to cross-compile to arm64, at least.

But if you want to run the build on arm (not cross-compile)... that's a different story, yeah.

@zenhack
Copy link
Collaborator

zenhack commented May 25, 2020

@xet7, I think it should work to just override CC and CXX with the systems' clang and clang++ respectively (assuming they're new enough, which they probably are). I raised the question of un-bundling clang in #3331, but we decided against it. If building on Arm becomes common we can revisit how the build system should work.

@xet7
Copy link
Contributor

xet7 commented May 25, 2020

@kentonv @zenhack

So how can I either:
a) cross compile Sandstorm to ARM ?
b) compile Sandstorm on ARM ?

For me ARM has been common for a long time. I have:

  • RasPi4 with 4 GB RAM running Wekan on Ubuntu arm64 and Raspbian armhf
  • RasPi3 with 1 GB RAM
  • RasPi1
  • bare metal arm64 server that has 128 GB RAM

I would also like to cross compile to s390x, etc

@kentonv
Copy link
Member

kentonv commented May 25, 2020

Getting Sandstorm working on arm will almost certainly require some code changes, e.g. in the seccomp sandbox. It's not going to be trivial. I don't know what will be involved, though, since I haven't tried it...

@xet7
Copy link
Contributor

xet7 commented May 25, 2020

@kentonv

What kind of changes you image there could be in the seccomp sandbox? How can I try it?

@kentonv
Copy link
Member

kentonv commented May 25, 2020

seccomp filters are architecture-specific, because many syscalls differ across architectures. So it will probably need to be tweaked for the architecture.

@xet7
Copy link
Contributor

xet7 commented May 25, 2020

@kentonv

Where in Sandstorm source code are those architecture-specific syscalls? What syscalls are used?

@kentonv
Copy link
Member

kentonv commented May 25, 2020

No, that's not what I mean. Many common syscalls, like clone(), have different signatures on different architectures. This is just normal. glibc typically wraps the details. But seccomp filters have to operate on the raw syscalls. Sandstorm employs a seccomp filter as part of its sandbox. This filter will likely need to be tweaked to work correctly on other architectures.

The seccomp code is here:

void SupervisorMain::setupSeccomp() {
// Install a rudimentary seccomp blacklist.
// TODO(security): Change this to a whitelist.
scmp_filter_ctx ctx = seccomp_init(SCMP_ACT_ALLOW);
if (ctx == nullptr)
KJ_FAIL_SYSCALL("seccomp_init", 0); // No real error code
KJ_DEFER(seccomp_release(ctx));
#define CHECK_SECCOMP(call) \
do { \
if (auto result = (call)) { \
KJ_FAIL_SYSCALL(#call, -result); \
} \
} while (0)
// Native code only for now, so there are no seccomp_arch_add calls.
// Redundant, but this is standard and harmless.
CHECK_SECCOMP(seccomp_attr_set(ctx, SCMP_FLTATR_CTL_NNP, 1));
// It's easy to inadvertently issue an x32 syscall (e.g. syscall(-1)). Such syscalls
// should fail, but there's no need to kill the issuer.
CHECK_SECCOMP(seccomp_attr_set(ctx, SCMP_FLTATR_ACT_BADARCH, SCMP_ACT_ERRNO(ENOSYS)));
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wmissing-field-initializers" // SCMP_* macros produce these
// Disable some things that seem scary.
if (!devmode) {
// ptrace is scary
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(ptrace), 0));
} else {
// Try to be somewhat safe with ptrace in dev mode. Note that the ability to modify
// orig_ax using ptrace allows a complete seccomp bypass.
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(ptrace), 1,
SCMP_A0(SCMP_CMP_EQ, PTRACE_POKEUSER)));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(ptrace), 1,
SCMP_A0(SCMP_CMP_EQ, PTRACE_SETREGS)));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(ptrace), 1,
SCMP_A0(SCMP_CMP_EQ, PTRACE_SETFPREGS)));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(ptrace), 1,
SCMP_A0(SCMP_CMP_EQ, PTRACE_SETREGSET)));
}
// Restrict the set of allowable network protocol families
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EAFNOSUPPORT), SCMP_SYS(socket), 1,
SCMP_A0(SCMP_CMP_GE, AF_NETLINK + 1)));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EAFNOSUPPORT), SCMP_SYS(socket), 1,
SCMP_A0(SCMP_CMP_EQ, AF_AX25)));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EAFNOSUPPORT), SCMP_SYS(socket), 1,
SCMP_A0(SCMP_CMP_EQ, AF_IPX)));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EAFNOSUPPORT), SCMP_SYS(socket), 1,
SCMP_A0(SCMP_CMP_EQ, AF_APPLETALK)));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EAFNOSUPPORT), SCMP_SYS(socket), 1,
SCMP_A0(SCMP_CMP_EQ, AF_NETROM)));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EAFNOSUPPORT), SCMP_SYS(socket), 1,
SCMP_A0(SCMP_CMP_EQ, AF_BRIDGE)));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EAFNOSUPPORT), SCMP_SYS(socket), 1,
SCMP_A0(SCMP_CMP_EQ, AF_ATMPVC)));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EAFNOSUPPORT), SCMP_SYS(socket), 1,
SCMP_A0(SCMP_CMP_EQ, AF_X25)));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EAFNOSUPPORT), SCMP_SYS(socket), 1,
SCMP_A0(SCMP_CMP_EQ, AF_ROSE)));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EAFNOSUPPORT), SCMP_SYS(socket), 1,
SCMP_A0(SCMP_CMP_EQ, AF_DECnet)));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EAFNOSUPPORT), SCMP_SYS(socket), 1,
SCMP_A0(SCMP_CMP_EQ, AF_NETBEUI)));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EAFNOSUPPORT), SCMP_SYS(socket), 1,
SCMP_A0(SCMP_CMP_EQ, AF_SECURITY)));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EAFNOSUPPORT), SCMP_SYS(socket), 1,
SCMP_A0(SCMP_CMP_EQ, AF_KEY)));
// Disallow DCCP sockets due to Linux CVE-2017-6074.
//
// The `type` parameter to `socket()` can have SOCK_NONBLOCK and SOCK_CLOEXEC bitwise-or'd in,
// so we need to mask those out for our check. The kernel defines a constant SOCK_TYPE_MASK
// as 0x0f, but this constant doesn't appear to be in the headers, so we specify by hand.
//
// TODO(security): We should probably disallow everything except SOCK_STREAM and SOCK_DGRAM but
// I don't totally get how to write such conditionals with libseccomp. We should really dump
// libseccomp and write in BPF assembly, which is frankly much easier to understand.
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPROTONOSUPPORT), SCMP_SYS(socket), 1,
SCMP_A1(SCMP_CMP_MASKED_EQ, 0x0f, SOCK_DCCP)));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(add_key), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(request_key), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(keyctl), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(syslog), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(uselib), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(personality), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(acct), 0));
// 16-bit code is unnecessary in the sandbox, and modify_ldt is a historic source
// of interesting information leaks.
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(modify_ldt), 0));
// Despite existing at a 64-bit syscall, set_thread_area is only useful
// for 32-bit programs. 64-bit programs use arch_prctl instead.
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(set_thread_area), 0));
// Disable namespaces. Nested sandboxing could be useful but the attack surface is large.
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(unshare), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(mount), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(pivot_root), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(quotactl), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(clone), 1,
SCMP_A0(SCMP_CMP_MASKED_EQ, CLONE_NEWUSER, CLONE_NEWUSER)));
// AIO is scary.
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(io_setup), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(io_destroy), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(io_getevents), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(io_submit), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(io_cancel), 0));
// Scary vm syscalls
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(remap_file_pages), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(mbind), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(get_mempolicy), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(set_mempolicy), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(migrate_pages), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(move_pages), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(vmsplice), 0));
// Scary futex operations
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(set_robust_list), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(get_robust_list), 0));
// Utterly terrifying profiling operations
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(perf_event_open), 0));
// Don't let apps specify their own seccomp filters, since seccomp filters are literally programs
// that run in-kernel (albeit with a very limited instruction set).
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EINVAL), SCMP_SYS(prctl), 1,
SCMP_A0(SCMP_CMP_EQ, PR_SET_SECCOMP)));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(seccomp), 0));
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(bpf), 0));
// New syscalls that don't seem useful to Sandstorm apps therefore we will disallow them.
// TODO(cleanup): Can we somehow specify "disallow all calls greater than N" to preemptively
// disable things until we've reviewed them?
CHECK_SECCOMP(seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOSYS), SCMP_SYS(userfaultfd), 0));
// TOOD(someday): See if we can get away with turning off mincore, madvise, sysinfo etc.
// TODO(someday): Turn off POSIX message queues and other such esoteric features.
if (seccompDumpPfc) {
seccomp_export_pfc(ctx, 1);
}
CHECK_SECCOMP(seccomp_load(ctx));
#pragma GCC diagnostic pop
#undef CHECK_SECCOMP
}

@kentonv
Copy link
Member

kentonv commented May 25, 2020

I hate to be discouraging here, but I don't think an arm port is likely to pan out without a lot of work, and I don't think I have time to provide the assistance that would be needed from me to get through it. So I'd recommend leaving this on the shelf for now.

@xet7
Copy link
Contributor

xet7 commented May 25, 2020

@kentonv

Is there some existing lightweight sandboxing library or software that already works on multiple CPUs? That could be used instead, for example sandboxing Wekan and other apps? Or is Docker only option?

Or alternatively, is sandboxing easier in some other programming language?

@xet7
Copy link
Contributor

xet7 commented May 25, 2020

Well, having Wekan run currently on arm64 and s390x is already complicated, because when adding some new dependencies those in some cases do not work on other than x64 without modifications. Currently I'm figuring out can ostrio-files run on those architectures, and reverting changes while figuring it out is in progress.

@kentonv
Copy link
Member

kentonv commented May 25, 2020

It shouldn't be too hard to tweak the seccomp filter for any particular CPU if you know what you're doing -- maybe half an hour of work. Trying to integrate some other sandbox would be much, much harder.

This is just meant to be one example of a reason why porting Sandstorm to arm is not trivial. There will be many, many other issues like this.

@zenhack
Copy link
Collaborator

zenhack commented May 25, 2020 via email

@xet7
Copy link
Contributor

xet7 commented May 25, 2020

@zenhack

Thanks! I will test it a little.

@nniro
Copy link

nniro commented May 22, 2021

I attempted the compilation process following @zenhack 's suggestion and this is the compilation log :

compile.log

This was attempted on a pinebook pro with an aarch64 distribution.

Edit : I thought uploading a file would show it's content, here's the important part :

✔ test: capnp/serialize-async-test
✔ test: capnp/fuzz-test
✘ bpf_asm: sandstorm/seccomp-bpf/filter.s
    error: syntax error at line 810
✘ compile: node-capnp/capnp.cc
    /ekam-provider/canonical/node-capnp/capnp.cc:29:10: fatal error: 'node.h' file not found
    #include <node.h>
             ^~~~~~~~
    1 error generated.
✘ jstest: node-capnp/capnp-test.js
    full log: tmp/node-capnp/capnp-test.js.log
✘ install: sandstorm/sandstorm.ekam-manifest
    sandstorm/run-bundle: not found
✘ compile: sandstorm/supervisor.c++
    /ekam-provider/canonical/sandstorm/supervisor.c++:91:10: fatal error: 'sandstorm/seccomp-bpf/filter.h' file not found
    #include <sandstorm/seccomp-bpf/filter.h>
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    1 error generated.
✘ install: node-capnp/node-capnp.ekam-manifest
    node-capnp/capnp.node: not found
✘ link: sandstorm/run-bundle.o
    /usr/bin/ld: tmp/kj/async-io-unix.o: in function `kj::(anonymous namespace)::SocketAddress::lookupHost(kj::LowLevelAsyncIoProvider&, kj::String, kj::String, unsigned int, kj::_::
      NetworkFilter&)::$_3::operator()(kj::(anonymous namespace)::SocketAddress::LookupParams&&) const':
    /ekam-provider/canonical/kj/async-io-unix.c++:1007: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for
      linking
    /usr/bin/ld: tmp/sandstorm/run-bundle.o: in function `kj::Own<sandstorm::SupervisorMain> kj::heap<sandstorm::SupervisorMain, kj::ProcessContext&>(kj::ProcessContext&)':
    /ekam-provider/c++header/kj/memory.h:450: undefined reference to `sandstorm::SupervisorMain::SupervisorMain(kj::ProcessContext&)'
    /usr/bin/ld: tmp/sandstorm/run-bundle.o: in function `~SupervisorMain':
    /ekam-provider/canonical/sandstorm/supervisor.h:30: undefined reference to `vtable for sandstorm::SupervisorMain'
    /usr/bin/ld: /ekam-provider/canonical/sandstorm/supervisor.h:30: undefined reference to `vtable for sandstorm::SupervisorMain'
    /usr/bin/ld: tmp/kj/async.o(.debug_info+0x1569c): R_AARCH64_ABS64 used with TLS symbol _ZN2kj12_GLOBAL__N_120threadLocalEventLoopE
    /usr/bin/ld: tmp/kj/exception.o(.debug_info+0xaba1): R_AARCH64_ABS64 used with TLS symbol _ZN2kj12_GLOBAL__N_116currentExceptionE
    /usr/bin/ld: tmp/kj/exception.o(.debug_info+0xabbc): R_AARCH64_ABS64 used with TLS symbol _ZN2kj12_GLOBAL__N_119threadLocalCallbackE
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:268: tmp/.ekam-run] Error 1

@zenhack
Copy link
Collaborator

zenhack commented May 22, 2021

So I actually see two separate errors in that snippet. One has to do with the seccomp filter, which we shouldn't expect to work yet since the code isn't portable in the first place, but it also seems to be having problems finding the node JS headers.

Ekam has the annoying property that it tends to cascade errors, so when you see "file not found" at some point in the log, it's likely that the real error is further up, where ekam failed to generate that file (though node.h should be pre-existing). I haven't had a chance to comb through the logs to figure out exaclty what went wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests