ARM architecture support

[reqresnext]

I have Raspberry pi 3 b+ board and I am trying to use caxa on it. I've compiled examples and it shows an error:
bash: ./echo-command-line-parameters: cannot execute binary file: Exec format error.
file ./echo-command-line-parameters gives an output:
ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=ALTTohZDVVCYbbGb20Qi/X-X6HXeYwbH84AVQlSaQ/0RzAyRSxhi0ygj4oAFbv/ywg9-ioLknP7q7VwDKv0, not stripped
Is it possible to create execurable for ARM instead of x86-64?

[maxb2]

+1, ARM support is the only thing stopping me from switching to caxa.

[leafac]

Sure. It’s a matter of compiling the stub to ARM. I’ll get to it soon…

[reqresnext]

Sure. It’s a matter of compiling the stub to ARM. I’ll get to it soon…

Actually I've tried to add flags like GOARCH=arm and GOARM=7 to caxa package.json stubs section but executable is still x86. Looks like it is wrong place

[maxb2]

Sure. It’s a matter of compiling the stub to ARM. I’ll get to it soon…

Actually I've tried to add flags like GOARCH=arm and GOARM=7 to caxa package.json stubs section but executable is still x86. Looks like it is wrong place

This works for me:

GOOS=linux GOARCH=arm GOARM=7 go build -o linux-armv7 stub.go
GOOS=linux GOARCH=arm64 go build -o linux-arm64 stub.go

[robertsLando]

Caxa doesn't hide source code

[leafac]

@maxb2: Thanks for finding the configurations that work. Do you know what we need to do from Node’s side to detect that we should use these stubs (that is, to detect that we’re in a Raspberry Pi)?

@robertsLando: Yes, you’re right. I talked about this in the README briefly. If hiding the source code is a concern for you, then you may obfuscate the code as part of the preparation for packaging. It probably isn’t as good as the V8 snapshot approach used by pkg, but it may be good enough. And if you have suggestions on how to improve on this regard, I’m all ears.

[n1ru4l]

@leafac this pr includes code for detecting the arch: https://github.com/vercel/pkg-fetch/pull/106/files

[reqresnext]

@leafac this pr includes code for detecting the arch: https://github.com/vercel/pkg-fetch/pull/106/files

Is it possible, please, to update caxa source code for ARM support, since I changed the caxa`s package.json file with GOOS=linux GOARCH=arm GOARM=7 go build -o linux-armv7 stub.go, but i still get X86 executable?

[maxb2]

@maxb2: Thanks for finding the configurations that work. Do you know what we need to do from Node’s side to detect that we should use these stubs (that is, to detect that we’re in a Raspberry Pi)?

I'd use the uname -a solution from @n1ru4l:

@leafac this pr includes code for detecting the arch: https://github.com/vercel/pkg-fetch/pull/106/files

(We're working on the same project)

[leafac]

Hi all,

I’m checking in to let you know that I’m still working on this.

I took a brief detour to introduce caxa into some of my other projects: https://github.com/leafac/kill-the-newsletter / https://github.com/courselore/courselore. The intent here was to confirm that everything was working and that caxa was production-ready. Everything went very smoothly 🙌

Now I’m back to working in caxa proper, and ARM support is at the top of my priority list.

I’ll get back to you when this is ready for testing. I’ll also get a Raspberry Pi myself to test.

[robertsLando]

@leafac I have find some time to read all the README. My compliments for all the research behind this project, I tried almost all the solution you listed and I get the same conclusions on all projects. I think that the best one out there ATM is pkg, the only problems are nodejs patches.

I love your idea and I will keep an eye on this project, as I mentioned above the main problem in my case is to hide the soruce code, obfuscation could be a way for sure

[n1ru4l]

@robertsLando could sth like http://peterforgacs.github.io/2018/09/12/How-to-create-a-V8-snapshot-of-your-javascript-file/ work with caxa?

[robertsLando]

That would be interesting 🤔 @leafac thoughts?

[leafac]

I looked into the idea of using V8 snapshots and I think that the way to go would be leverage something like this when it lands. Everything else that I found was either tied to Electron or sounded like too much magic, so things could break too easily. The principle of least surprise is in the DNA of caxa: If it works in development then it should work after packaging.

Am I missing something? Is there a simpler way of working with snapshots?

[fcastilloec]

Just found this wonderful app, the only thing preventing me from using it is armv7 support. Looking forward to when this is done!

[fcastilloec]

I've been using pkg until now because it was the only one that works well on armv7 but their latest update removed support for this architecture, even compiling node for ourselves is broken.
Is there anything I could do to help move this along? I'm happy to test anything and provide logs, I'm just not familiar enough with the code to provide a PR myself

[leafac]

@fcastilloec: Thanks for the nice words. When I have something for you to test I’ll get back in touch. I’ll get to it this week…

[pdcastro]

Is it possible, please, to update caxa source code for ARM support, since I changed the caxa`s package.json file with GOOS=linux GOARCH=arm GOARM=7 go build -o linux-armv7 stub.go, but i still get X86 executable?

@reqresnext, I made that same change and it worked for me. Maybe you didn't run npm run stubs after you had changed the package.json file?

Before: cross-env GOOS=linux go build -o linux stub.go
After: cross-env GOOS=linux GOARCH=arm GOARM=7 go build -o linux-armv7 stub.go

$ npm run stubs
$ file stubs/linux-armv7
stubs/linux-armv7: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, Go BuildID=6FImErS1M7MallmrRuW_/u1ULaUtXVJeNiLV9KHZS/kkyZwTs9VvVhRe5XZsuY/J6tj0CCqVr-bFJd2Et3u, not stripped

[pdcastro]

Do you know what we need to do from Node’s side to detect that we should use these stubs (that is, to detect that we’re in a Raspberry Pi)?

this pr includes code for detecting the arch: https://github.com/vercel/pkg-fetch/pull/106/files

Here's the output of uname -a for some Raspberry Pi versions:

# Raspberry Pi Zero and Pi 1, 32 bits
$ docker run balenalib/raspberry-pi uname -a
Linux 25c422059b44 5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021 armv6l GNU/Linux

# Raspberry Pi 2 and Pi 3, 32 bits
$ docker run balenalib/raspberrypi3 uname -a
Linux c04f8265a27a 5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021 armv7l GNU/Linux

# Raspberry Pi 3, 64 bits
$ docker run balenalib/raspberrypi3-64 uname -a
Linux 59f165e6b265 5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021 aarch64 GNU/Linux

# Raspberry Pi 4, 64 bits
$ docker run balenalib/raspberrypi4-64 uname -a
Linux 757ae5f95a63 5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021 aarch64 GNU/Linux

Balenalib image list: https://www.balena.io/docs/reference/base-images/base-images-ref/
Balena has images for a lot of IoT devices.

By the way @leafac, while at it, please add ARM v6 as well. These commands were successful for me:

BIN=linux-armv6; GOOS=linux GOARCH=arm GOARM=6 go build -o $BIN stub.go && echo >> $BIN && echo "### CAXA ###" >> $BIN
BIN=linux-armv7; GOOS=linux GOARCH=arm GOARM=7 go build -o $BIN stub.go && echo >> $BIN && echo "### CAXA ###" >> $BIN
BIN=linux-arm64; GOOS=linux GOARCH=arm64       go build -o $BIN stub.go && echo >> $BIN && echo "### CAXA ###" >> $BIN

I don't have a Pi Zero or Pi 1 on my desk to test the v6 stub, but I tested the v7 stub on a Pi 3. The reason why I ask is that the Pi Zero is fairly popular in a category of Pi projects: different form factor (photos), price point, power consumption. I am currently considering using caxa for the balena CLI, and if caxa doesn't include ARM v6, I would have to build it separately. :-)

[maxb2]

@pdcastro

I don't have a Pi Zero or Pi 1 on my desk to test the v6 stub

I did some investigating with an old Pi 1 and the arm situation is weird.

Binary	Compilation Method	Raspi1 Test (Raspberry Pi OS armv6l)
linux-armv6	Cross-compile	success
linux-armv6	Emulate	caxa stub: Failed to find archive (did you append the separator when building the stub?): stat /tmp/caxa/linux-armv6/temp: no such file or directory success
linux-armv7	Cross-compile	Illegal Instruction
linux-armv7	Emulate	success

It seems that cross-compilation produces the correct behavior while emulation does not. (Related #11)

I made the test the following way:

# Emulated armv6
docker run --rm -it -v $PWD:/usr/src/myapp -w /usr/src/myapp balenalib/raspberry-pi:build /bin/bash -c 'wget -cv https://golang.org/dl/go1.16.4.linux-armv6l.tar.gz && rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.4.linux-armv6l.tar.gz && export PATH=$PATH:/usr/local/go/bin && CGO_ENABLED=0 go build -o linux-armv6 stub.go && echo "" >> linux-armv6 && echo "### CAXA ###" >> linux-armv6'

# Emulated armv7
docker run --rm -it --platform linux/arm/v7 -v $PWD:/usr/src/myapp -w /usr/src/myapp arm32v7/golang:1.16 /bin/bash -c 'CGO_ENABLED=0 go build -o linux-armv7 stub.go && echo "" >> linux-armv7 && echo "### CAXA ###" >> linux-armv7'


# Cross-compiled
BIN=linux-cross-armv6; GOOS=linux GOARCH=arm GOARM=6 go build -o $BIN stub.go && echo >> $BIN && echo "### CAXA ###" >> $BIN
BIN=linux-cross-armv7; GOOS=linux GOARCH=arm GOARM=7 go build -o $BIN stub.go && echo >> $BIN && echo "### CAXA ###" >> $BIN

echo 'success' > test.txt
tar -czf - test.txt >> linux-armv6
printf "\n{ \"identifier\": \"linux-armv6/temp\", \"command\": [\"cat\", \"{{caxa}}/test.txt\"] }" >> linux-armv6
# And similarly for the rest of the binaries

[pdcastro]

@maxb2, the error message (caxa stub: Failed to find archive) comes from here:
https://github.com/leafac/caxa/blob/v1.0.0/stubs/stub.go#L62

This means that the stub was somewhat successfully executed (good news!), perhaps just the test went wrong. The error message goes on to report: stat /tmp/caxa/linux-armv6/temp: no such file or directory

I suspect that that directory would normally be created on these lines:
https://github.com/leafac/caxa/blob/v1.0.0/src/index.ts#L31-L32

But I gather that your test did not include invoking caxa in the normal way, so it would be up to you to manually create /tmp/caxa/linux-armv6/temp. Was it created?
 

linux-armv7 | Cross-compile | Illegal Instruction

How did that Illegal Instruction happen? The stub generated and tested as follows worked for me on an RPi 3:

$ cd stubs
$ mv linux linux-amd64
$ BIN=linux-armv7; GOOS=linux GOARCH=arm GOARM=7 go build -o $BIN stub.go && echo >> $BIN && echo "### CAXA ###" >> $BIN
$ ln -s linux-armv7 linux
$ cd ..
$ mkdir t && echo 'success' > t/test.txt
$ node lib/index.js -d t -c cat "{{caxa}}/test.txt" -o test.bin
$ scp test.bin pi3:/tmp/
$ ssh pi3 chmod +x /tmp/test.bin \&\& /tmp/test.bin
success

[maxb2]

But I gather that your test did not include invoking caxa in the normal way,

I'm following the instructions from the readme for manually creating a self-extracting executable.

How did that Illegal Instruction happen? The stub generated and tested as follows worked for me on an RPi 3

This was all run on a first generation RPi with an armv6l kernel. If you have other ideas you want to test on a RPi1, let me know.

This means that the stub was somewhat successfully executed (good news!), perhaps just the test went wrong. The error message goes on to report: stat /tmp/caxa/linux-armv6/temp: no such file or directory

It was a problem with the compilation command. I wasn't paying enough attention to the quotes 🤦 It should be:

docker run --rm -it -v $PWD:/usr/src/myapp -w /usr/src/myapp balenalib/raspberry-pi:build /bin/bash -c 'wget -cv https://golang.org/dl/go1.16.4.linux-armv6l.tar.gz && rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.4.linux-armv6l.tar.gz && export PATH=$PATH:/usr/local/go/bin && CGO_ENABLED=0 go build -o linux-armv6 stub.go && echo "" >> linux-armv6 && echo "### CAXA ###" >> linux-armv6'

The emulated linux-armv6 binary works as expected now.

[leafac]

Thank you all for the excellent investigative work. You’re making my life a lot easier 🙌

ARM support will be out this week. I’m excited to have belena use caxa. I’ve used balenaEtcher before and I think you’re doing great work.

I’m live-streaming as I work on this, and I’ll be delighted to have you jump in and follow along:

https://youtu.be/4YIiNACGTzI

[fcastilloec]

@leafac this pr includes code for detecting the arch: https://github.com/vercel/pkg-fetch/pull/106/files

Why not use process.arch from node itself to get the architecture? It will show arm and arm64.
In order to get the version of arm (either 6 or 7), you can look at process.config.variables.arm_version. For non-arm architecture, it will just show as undefined.

Also, I think it'll be better to specify the architecture we want to package for on the command line. In other words, I can package for arm on an x64 machine, or vice versa.

[maxb2]

caxa doesn't do "cross-packaging". Caxa just copies the host's node binary and the locally built project (this could include native modules such as sqlite). Supporting arm packaging on an x64 host would lead to some of the same complexities of other packaging solutions.

However, it's pretty easy to set up an emulated arm environment on an x64 host using docker.

# Setup qemu
docker run --rm --privileged tonistiigi/binfmt:latest --install all

cd <project directory>

# Build and package project
docker run --rm --platform linux/arm/v7 -v $(pwd):/usr/src/myapp -w /usr/src/myapp arm32v7/node sh -c '<project build script>'

Where arm32v7/node can be changed to whatever docker image you wish to use. See here for more info.

[fcastilloec]

@maxb2 I'm was trying to avoid using docker. Docker will always work for any cross-packaging but for less advanced users, docker is a roadblock.
Because I don't see caxa fetching binaries for other architectures directly from the nodejs site (that would create more problems than anything else), maybe let the user specify the node file to be packaged. That way I can manually download the node binary for arm and let caxa know to use that one. This is related to #9, where we could tell caxa to not package any node binary, or let the user specify one.

[maxb2]

Any native modules will be architecture specific as well. See this section of the readme.

The reason why caxa is so simple is that it relies on the host system to build the project. caxa simply bundles the built app in such a way that it can be easily copied to and run on another compatible host.

[fcastilloec]

@maxb2 You are right! I'm currently not using any native modules, I totally forgot about them. I do have a couple of docker images set up for building on other platforms, I was hoping not to have to use them.

[leafac]

@fcastilloec

Why not use process.arch from node itself to get the architecture? It will show arm and arm64.
In order to get the version of arm (either 6 or 7), you can look at process.config.variables.arm_version. For non-arm architecture, it will just show as undefined.

Awesome. I was going to look for something like this before I adopted the uname -a approach, and you made my life easier 😃

---

Regarding specifying a different node executable to package: I’ll get to that later this week. It’ll be an advanced option for when you know that you don’t have native dependencies and you know what you’re doing.

[leafac]

ARM support is ready for testing. Please grab caxa@2.0.0-beta.5 and give a try. The proper 2.0.0 will wait until I fix a couple more bugs; it’ll probably be out by the end of next week.

Let me know how it goes.

[fcastilloec]

@leafac just tested the beta version on my Raspberry Pi 4 (armv7) and everything is working great!
For anybody testing, the command line arguments have changed and the README for that tag hasn't been updated yet. So make sure to run the help to check what to do.
The only thing missing to actually use caxa in production is fixing #12, probably by merging #8

[leafac]

Thanks for testing!

Yep, a fix to #12 will be in 2.0.0 as well…

[leafac]

Hi all,

I’m happy to announce that v2.0.0 is out of beta, including ARM support!

(Also, #12 has been fixed.)

Enjoy!