Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support for rpi0 #59

Merged
merged 3 commits into from Dec 21, 2018

Conversation

nandra
Copy link
Contributor

@nandra nandra commented Dec 10, 2018

Added few tweaks to have support for rpi0. With actual mender-convert master I anyway get crash when staring mender like:

Nov 13 14:05:20 raspberrypi systemd[1]: mender.service: Main process exited, code=killed, status=11/SEGV
Nov 13 14:05:20 raspberrypi systemd[1]: mender.service: Unit entered failed state.
Nov 13 14:05:20 raspberrypi systemd[1]: mender.service: Failed with result 'signal'.
Nov 13 14:05:20 raspberrypi systemd[1]: mender.service: Service hold-off time over, scheduling restart.
Nov 13 14:05:20 raspberrypi systemd[1]: Stopped Mender OTA update service.
Nov 13 14:05:20 raspberrypi systemd[1]: Started Mender OTA update service.
Nov 13 14:05:20 raspberrypi systemd[1]: mender.service: Main process exited, code=killed, status=11/SEGV
Nov 13 14:05:20 raspberrypi systemd[1]: mender.service: Unit entered failed state.
Nov 13 14:05:20 raspberrypi systemd[1]: mender.service: Failed with result 'signal'.
Nov 13 14:05:21 raspberrypi systemd[1]: mender.service: Service hold-off time over, scheduling restart.
Nov 13 14:05:21 raspberrypi systemd[1]: Stopped Mender OTA update service.
Nov 13 14:05:21 raspberrypi systemd[1]: Started Mender OTA update service.
Nov 13 14:05:21 raspberrypi systemd[1]: mender.service: Main process exited, code=killed, status=11/SEGV
Nov 13 14:05:21 raspberrypi systemd[1]: mender.service: Unit entered failed state.
Nov 13 14:05:21 raspberrypi systemd[1]: mender.service: Failed with result 'signal'.
Nov 13 14:05:21 raspberrypi systemd[1]: mender.service: Service hold-off time over, scheduling restart.
Nov 13 14:05:21 raspberrypi systemd[1]: Stopped Mender OTA update service.
Nov 13 14:05:21 raspberrypi systemd[1]: Started Mender OTA update service.
Nov 13 14:05:21 raspberrypi systemd[1]: mender.service: Main process exited, code=killed, status=11/SEGV
Nov 13 14:05:21 raspberrypi systemd[1]: mender.service: Unit entered failed state.
Nov 13 14:05:21 raspberrypi systemd[1]: mender.service: Failed with result 'signal'.
Nov 13 14:05:22 raspberrypi systemd[1]: mender.service: Service hold-off time over, scheduling restart.
Nov 13 14:05:22 raspberrypi systemd[1]: Stopped Mender OTA update service.
Nov 13 14:05:22 raspberrypi systemd[1]: Started Mender OTA update service.
Nov 13 14:05:22 raspberrypi systemd[1]: mender.service: Main process exited, code=killed, status=11/SEGV
Nov 13 14:05:22 raspberrypi systemd[1]: mender.service: Unit entered failed state.
Nov 13 14:05:22 raspberrypi systemd[1]: mender.service: Failed with result 'signal'.
Nov 13 14:05:22 raspberrypi systemd[1]: mender.service: Service hold-off time over, scheduling restart.
Nov 13 14:05:22 raspberrypi systemd[1]: Stopped Mender OTA update service.
Nov 13 14:05:22 raspberrypi systemd[1]: mender.service: Start request repeated too quickly.
Nov 13 14:05:22 raspberrypi systemd[1]: Failed to start Mender OTA update service.
Nov 13 14:05:22 raspberrypi systemd[1]: mender.service: Unit entered failed state.
Nov 13 14:05:22 raspberrypi systemd[1]: mender.service: Failed with result 'signal'.

even reading of version failed:

pi@raspberrypi:~$ mender -version
Segmentation fault

Any ideas? I'm using raspbian lite (so maybe something is missing there)? Thanks.

@estenberg
Copy link
Contributor

Nice!
I did use the desktop lite version for RPi3 and it worked well.

How do you compile Mender? Is it the default one in the mender-convert container? Have you produced a build of Mender that runs on this device in the past? How did you do that?

@nandra
Copy link
Contributor Author

nandra commented Dec 10, 2018

It was precompiled from docker container

@mirzak
Copy link
Contributor

mirzak commented Dec 10, 2018

Check output of file mender and compare it to some other binary that is installed on the device.

@nandra
Copy link
Contributor Author

nandra commented Dec 10, 2018

I just mounted rootfs and file mender produce:

file /mnt/usr/bin/mender 
/mnt/usr/bin/mender: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=0f2b87874c832929167c9b67eda37cd4611277c9, not stripped

compared to:

file /mnt/usr/bin/hcitool 
/mnt/usr/bin/hcitool: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=970be41d1977fbd616d516c2d26b93dd10d44201, stripped

which looks same to me (except armhf version)

@estenberg
Copy link
Contributor

Which Linux kernel version is on your Raspbian?

Mender seems to be for 2.6, the other tool is 3.2.

@nandra
Copy link
Contributor Author

nandra commented Dec 10, 2018

kernel is 4.14

@mirzak
Copy link
Contributor

mirzak commented Dec 10, 2018

Yeah the Linux kernel version mentioned in the file output is simply the version of the Linux header files that are part of the tool-chain. Should not really matter as it is a newer Linux kernel anyway. Linux is not allowed to break ABI to user-space.

Probably un-related but the file we create is not stipped. We should probably do that as part of the build to reduce the file size.

@mirzak
Copy link
Contributor

mirzak commented Dec 11, 2018

Could you try re-building the container (mender-client) with the addition GOARM=6 here https://github.com/mendersoftware/mender-convert/blob/master/Dockerfile#L68-L71

@nandra
Copy link
Contributor Author

nandra commented Dec 11, 2018

Tried with that :

Step 16/21 : RUN git checkout $MENDER_CLIENT_VERSION
 ---> Using cache
 ---> 029c945b4350
Step 17/21 : RUN env CGO_ENABLED=1     CC=arm-linux-gnueabihf-gcc     GOOS=linux     GOARM=6     GOARCH=arm make build
 ---> Running in d764bddf2702
go build -ldflags "-X main.Version=1.6.0"  
 ---> 35d86350abe4

but symptoms are same. IIRC when we developing with version where we need to put explicitly mender binary it was working fine. Need to check what is difference.

Copy link
Contributor

@apodogrocki apodogrocki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rebase PR. Few commits have been merged in the meantime. It will affect e.g. renaming rpi3-convert-stage-3.sh to simply convert-stage-3.sh

@@ -57,7 +57,12 @@ build_uboot_files() {
git checkout $commit >> "$build_log" 2>&1

make --quiet distclean >> "$build_log"
make --quiet rpi_3_32b_defconfig >> "$build_log" 2>&1
if [ "$2" = "raspberrypi3" ]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add $2 to the list of arguments the function takes. Add separate variable for 'defconfig' (or name it as you wish) so that you can call make --quiet $defconfig only once.

Copy link
Contributor Author

@nandra nandra Dec 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@apodogrocki can be something like that:

 local defconfig="rpi_3_32b_defconfig"

  if [ "$2" = "raspberrypi0" ]; then
    defconfig="rpi_0_w_defconfig"
  fi  

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, exactly. Only remember about == for comparison.

@@ -118,12 +123,21 @@ install_files() {
sudo ln -sf /dev/null ${rootfs_dir}/etc/systemd/system/udisks2.service

# Extract Linux kernel and install to /boot directory on rootfs
sudo cp ${boot_dir}/kernel7.img ${rootfs_dir}/boot/zImage
if [ "${device_type}" = "raspberrypi3" ]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same here, use a variable for kernel7.img / kernel.img file and call sudo cp ... only once. Having this you will also call a single command in line 135, instead of checking again device_type.

@tuupola
Copy link
Contributor

tuupola commented Dec 14, 2018

Could you try re-building the container (mender-client) with the addition GOARM=6 here https://github.com/mendersoftware/mender-convert/blob/master/Dockerfile#L68-L71

I remember having to use GOARM=5 with old Raspberries when compiling Go binaries. Otherwise got unknown instruction or something similar errors. It's been a while though so the memory is not the clearest.

@nandra
Copy link
Contributor Author

nandra commented Dec 14, 2018

ok let me try to build with GOARM=5 and verify

@nandra
Copy link
Contributor Author

nandra commented Dec 14, 2018

@tuupola Same issue -> very strange

@melias122
Copy link

@tuupola, cat /proc/cpuinfo gives model name : ARMv6-compatible processor rev 7 (v6l), so at least GOARCH=6 should be used.

@mirzak
Copy link
Contributor

mirzak commented Dec 14, 2018

Yeah, I do not believe that it is a compile time arch problem. As the error is a segfault.

pi@raspberrypi:~$ mender -version
Segmentation fault

Could you try running "strace mender", to try to pin-point where the segfault is.

@nandra
Copy link
Contributor Author

nandra commented Dec 14, 2018

strace mender
execve("/usr/bin/mender", ["mender"], [/* 16 vars */]) = 0
brk(NULL)                               = 0xced000
uname({sysname="Linux", nodename="raspberrypi", ...}) = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap2(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6fa5000
access("/etc/ld.so.preload", R_OK)      = 0
open("/etc/ld.so.preload", O_RDONLY|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=42, ...}) = 0
mmap2(NULL, 42, PROT_READ|PROT_WRITE, MAP_PRIVATE, 3, 0) = 0xb6fa4000
close(3)                                = 0
open("/usr/lib/arm-linux-gnueabihf/libarmmem.so", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\210\5\0\0004\0\0\0"..., 512) = 512
lseek(3, 20868, SEEK_SET)               = 20868
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1000) = 1000
lseek(3, 20540, SEEK_SET)               = 20540
read(3, "A,\0\0\0aeabi\0\1\"\0\0\0\0056\0\6\6\10\1\t\1\n\3\f\1\22\4\24"..., 45) = 45
fstat64(3, {st_mode=S_IFREG|0644, st_size=21868, ...}) = 0
mmap2(NULL, 86080, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6f62000
mprotect(0xb6f67000, 61440, PROT_NONE)  = 0
mmap2(0xb6f76000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0xb6f76000
close(3)                                = 0
munmap(0xb6fa4000, 42)                  = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=27256, ...}) = 0
mmap2(NULL, 27256, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb6f9e000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/arm-linux-gnueabihf/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\3\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0dI\0\0004\0\0\0"..., 512) = 512
lseek(3, 125740, SEEK_SET)              = 125740
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1560) = 1560
lseek(3, 90536, SEEK_SET)               = 90536
read(3, "A.\0\0\0aeabi\0\1$\0\0\0\0056\0\6\6\10\1\t\1\n\2\22\4\23\1\24"..., 47) = 47
fstat64(3, {st_mode=S_IFREG|0755, st_size=127300, ...}) = 0
mmap2(NULL, 164432, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6f39000
mprotect(0xb6f4f000, 61440, PROT_NONE)  = 0
mmap2(0xb6f5e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0xb6f5e000
mmap2(0xb6f60000, 4688, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb6f60000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/arm-linux-gnueabihf/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0@h\1\0004\0\0\0"..., 512) = 512
lseek(3, 1231820, SEEK_SET)             = 1231820
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2880) = 2880
lseek(3, 1228284, SEEK_SET)             = 1228284
read(3, "A.\0\0\0aeabi\0\1$\0\0\0\0056\0\6\6\10\1\t\1\n\2\22\4\23\1\24"..., 47) = 47
fstat64(3, {st_mode=S_IFREG|0755, st_size=1234700, ...}) = 0
mmap2(NULL, 1303872, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6dfa000
mprotect(0xb6f23000, 65536, PROT_NONE)  = 0
mmap2(0xb6f33000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x129000) = 0xb6f33000
mmap2(0xb6f36000, 9536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb6f36000
close(3)                                = 0
set_tls(0xb6fa73b0, 0xb6fa7a98, 0xb6fa9050, 0xb6fa73b0, 0xb6fa9050) = 0
mprotect(0xb6f33000, 8192, PROT_READ)   = 0
mprotect(0xb6f5e000, 4096, PROT_READ)   = 0
mprotect(0xb6f62000, 20480, PROT_READ|PROT_WRITE) = 0
mprotect(0xb6f62000, 20480, PROT_READ|PROT_EXEC) = 0
cacheflush(0xb6f62000, 0xb6f67000, 0, 0x15, 0) = 0
mprotect(0xb6f76000, 4096, PROT_READ)   = 0
mprotect(0xb6fa8000, 4096, PROT_READ)   = 0
munmap(0xb6f9e000, 27256)               = 0
set_tid_address(0xb6fa6f58)             = 526
set_robust_list(0xb6fa6f60, 12)         = 0
rt_sigaction(SIGRTMIN, {sa_handler=0xb6f3d2b0, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0xb6e266c0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0xb6f3d390, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0xb6e266c0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
ugetrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x1} ---
+++ killed by SIGSEGV +++
Segmentation fault

@melias122
Copy link

access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) this looks strange

@mirzak
Copy link
Contributor

mirzak commented Dec 14, 2018

access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) this looks strange

This seems to be fine, Debian patch to glibc after a quick search and should fail in most cases.

But the fact that it segfaults just after:

ugetrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0

Stack-overflow?

   RLIMIT_STACK
         This is the maximum size of the process stack, in bytes.  Upon
         reaching this limit, a SIGSEGV signal is generated.  To handle
         this signal, a process must employ an alternate signal stack
         (sigaltstack(2)).

Curious, can you run sudo mender --version?

@nandra
Copy link
Contributor Author

nandra commented Dec 14, 2018

sudo mender -version
Segmentation fault

@nandra
Copy link
Contributor Author

nandra commented Dec 20, 2018

@apodogrocki pushed fixes + rebased on top of actual master


# Replace kernel with U-boot and add boot script
sudo mkdir -p ${rootfs_dir}/uboot

sudo cp ${bin_dir_pi}/u-boot.bin ${boot_dir}/kernel7.img
if [ "${device_type}" = "raspberrypi3" ]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

== instead of = right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@apodogrocki yes sure sorry overlooked that -> pushed new version

@apodogrocki
Copy link
Contributor

Final changes are fine. Could you please do one more thing and squash two first commits, so that there is not separate Comments fixes commit?

@nandra
Copy link
Contributor Author

nandra commented Dec 21, 2018

@apodogrocki sure I'll do. Also I plan to add note to README that autobuild mender client is not working on rpi0

- bump mender-uboot (as patches for rpi0w are present in later revisions)
- reuse as much as possible rpi3 convert files

Signed-off-by: Marek Belisko <marek.belisko@open-nandra.com>
As we now support also rpi0 drop 3 from script name

Signed-off-by: Marek Belisko <marek.belisko@open-nandra.com>
Signed-off-by: Marek Belisko <marek.belisko@open-nandra.com>
@nandra
Copy link
Contributor Author

nandra commented Dec 21, 2018

@apodogrocki done

@apodogrocki apodogrocki merged commit e83ee15 into mendersoftware:master Dec 21, 2018
@mirzak
Copy link
Contributor

mirzak commented Dec 21, 2018

@nandra, You mention that the pre-built mender binary does not work, as we have discussed here.

Have you built one that does work? And how if that is the case?

@nandra
Copy link
Contributor Author

nandra commented Dec 21, 2018

@mirzak well it was build by customer and I'll try to get info how it was build and update. Thanks.

@rnehrboss
Copy link

@nandra , @mirzak did you get the pi0 working with a custom built?
Can you help by elaborating how to get it working?
Thanks

@nandra
Copy link
Contributor Author

nandra commented Feb 5, 2019

I have some good news. I was able to fix mender client segfault on rpi0w by using raspberrypi toolchain (taken from https://github.com/raspberrypi/tools).

There are 2 commits in
https://github.com/nandra/mender-conversion-tools/commits/rpi0w-toolchain

This is more less hack and clean solution would be to IMO add parameter to docker-build to specify for which device we want to build and then use proper toolchain to build mender client + also u-boot. Or use precompiled versions of mender + u-boot and pass it as parameters. Ideas? Thanks.

@estenberg
Copy link
Contributor

Nice!

I would really like to avoid different toolchains for different boards, this is getting more and more custom.. As an alternative, perhaps upgrading U-Boot for every board would help? See this ticket: https://tracker.mender.io/browse/MEN-2198

The precompiled option would be my dream. :) But would this work on most boards, or would we have to provide a precompiled uboot for each and every board?

@nandra
Copy link
Contributor Author

nandra commented Feb 5, 2019

Well currently we need to have separate u-boot for every board (basically every board have different config) and there doesn't exists something like multiarch u-boot. Also rpi0w is bit exception as architecture is armv6 while current linaro + ubuntu toolchain have support only armv7+ So for that reason we need to have custom toolchain for that purpose. Other commonly used arm board should work fine with linaro toolchains though. u-boot update is doable we just need to add few mender specific patches + 2018.x (not sure exactly which version) require gcc >= 6.x so we will have then problem for rpi0w (or we compile by hand new toolchain for this board).

@mirzak
Copy link
Contributor

mirzak commented Feb 5, 2019

Note that we only rely on compiling U-boot for Raspberry Pi boards, e.g BBB we use GRUB and this is not a problem.

I would suggest that we keep the raspberry0w as experimental and try to document the limitations (maybe a seperate Mender Hub post for it with link t your fork with the fixes) until we have explored more boards and confirmed that the raspberrypi boards are the exception.

We probably need to rework this anyway if we are to generalise the tool.

@nandra
Copy link
Contributor Author

nandra commented Feb 6, 2019

@mirzak ok clear. I'll add post to Mender Hub (debian family). But from testing it looks like also sunxi boards doesn't support grub yet so if anyone would like to have e.g. armbian we need to anyway compile u-boot to support that boards (but we can stick with current setup and don't need extra toolchain)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
7 participants