Docker not starting with v1.18.1 on arm64

[iexos]

I am trying to run ntfy on docker on a Raspberry 4. Executing

docker run -p 127.0.0.1:4043:80 -it binwiederhier/ntfy serve

stops after a few seconds without any output, and return code 132. (same if explicitly using the arm64-tagged image)

Running on Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0-1056-raspi aarch64)

[binwiederhier]

There have been issues with arm docker images in the past (search the GitHub issues). I don't have a raspberry pi so it's hard for me to test.

Can you try older image versions?

[binwiederhier]

I did a quick check on AWS, with the EC2 Ubuntu 20.04 image on amd64/aarch64 and it worked fine....

root@ip-172-31-13-139:/home/ubuntu# sudo docker run -it binwiederhier/ntfy
Unable to find image 'binwiederhier/ntfy:latest' locally
latest: Pulling from binwiederhier/ntfy
148d739a8e6b: Pull complete 
e72dc2f90671: Pull complete 
Digest: sha256:06e44c33adec7bfb87235ba0f9af2b0172c2f65c14b3742c9101ffda3622ef72
Status: Downloaded newer image for binwiederhier/ntfy:latest
NAME:
   ntfy - Simple pub-sub notification service

USAGE:
   ntfy [OPTION..]

COMMANDS:
   help, h  Shows a list of commands or help for one command
   Client commands:
     publish, pub, send, trigger  Send message via a ntfy server
     subscribe, sub               Subscribe to one or more topics on a ntfy server
   Server commands:
     serve   Run the ntfy server
     user    Manage/show users
     access  Grant/revoke access to a topic, or show access

GLOBAL OPTIONS:
   --help, -h  show help (default: false)

Try 'ntfy COMMAND --help' or https://ntfy.sh/docs/ for more information.

To report a bug, open an issue on GitHub: https://github.com/binwiederhier/ntfy/issues.
If you want to chat, simply join the Discord server (https://discord.gg/cT7ECsZj9w), or
the Matrix room (https://matrix.to/#/#ntfy:matrix.org).

ntfy 1.18.1 (a3e28e7), runtime go1.17, built at 2022-03-22T03:27:29Z
Copyright (C) 2022 Philipp C. Heckel, licensed under Apache License 2.0 & GPLv2

root@ip-172-31-13-139:/home/ubuntu# sudo docker run -it binwiederhier/ntfy serve
2022/03/29 23:21:10 Listening on :80[http]

root@ip-172-31-13-139:/home/ubuntu# uname -a
Linux ip-172-31-13-139 5.11.0-1022-aws #23~20.04.1-Ubuntu SMP Mon Nov 15 14:04:48 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

[iexos]

1.18.0 works, seems to be a recent regression

[binwiederhier]

I'm sorry but I can't help unless I can reproduce it or at least have some error message I can Google.

You could try the binary outside of docker and see if that makes a difference, and/or use strace to show all the system calls it's making, and where it is dying.

Those are my only ideas right now.

[iexos]

I made straces of both version to compare
strace-ntfy-v1.18.0.log
strace-ntfy-v1.18.1.log

[binwiederhier]

I think you probably need to strace the binary and not the docker run command. The 132 is definitely from the Docker binary, but my guess is that that's just a generic exit code when the binary dies. Either download the tarball with the correct binary and run that, or extract it from the docker (see #113 (comment)), or exec into the docker and strace in there.

[binwiederhier]

You may also want to join the Discord (https://discord.com/invite/cT7ECsZj9w), maybe there are others with a Pi4 that can troubleshoot with you

[iexos]

Actually, it is the ntfy binary returning 132:

# docker run -it --rm --entrypoint sh binwiederhier/ntfy:v1.18.1-arm64v8
/ # ntfy
Illegal instruction (core dumped)
/ # echo $?
132

(there is no strace inside this image)
I can take a look later, if I find out more. Thank you for your quick responses :)

[binwiederhier]

This typically only happens if you pick the wrong architecture. You're sure that this is arm64 and not armv7?

I am sorry I can't help more. I gotta get a Pi ;-)

[iexos]

This is getting more and more confusing... I extracted both binaries from the docker image, and suddenly v1.18.1 worked and v1.18.0 had illegal instructions. But only once, retrying restored the normal order... I attached the log of what I did as I would not believe myself without it.

# uname -a
Linux xxx 5.4.0-1056-raspi #63-Ubuntu SMP PREEMPT Mon Mar 14 07:48:06 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
# image="binwiederhier/ntfy:v1.18.1-arm64v8" && container_id=$(docker create "$image") && docker cp "$container_id:/usr/bin/ntfy" ntfy-1.18.1 && docker rm "$container_id"
fae72c230be5b877bbf23c02f1ec9cedbdf8a8abbb78e920305e1b8a20804784
# image="binwiederhier/ntfy:v1.18.0-arm64v8" && container_id=$(docker create "$image") && docker cp "$container_id:/usr/bin/ntfy" ntfy-1.18.0 && docker rm "$container_id"
0d3d92769c5eaeab76bc9cfbbbe293aba21640160fdfca0a65cbffcc20ba422f
# file ntfy-1.18.*
ntfy-1.18.0: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, no section header
ntfy-1.18.1: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, no section header
# ./ntfy-1.18.1
NAME:
   ntfy - Simple pub-sub notification service

USAGE:
   ntfy [OPTION..]

COMMANDS:
   help, h  Shows a list of commands or help for one command
   Client commands:
     publish, pub, send, trigger  Send message via a ntfy server
     subscribe, sub               Subscribe to one or more topics on a ntfy server
   Server commands:
     serve   Run the ntfy server
     user    Manage/show users
     access  Grant/revoke access to a topic, or show access

GLOBAL OPTIONS:
   --help, -h  show help (default: false)

Try 'ntfy COMMAND --help' or https://ntfy.sh/docs/ for more information.

To report a bug, open an issue on GitHub: https://github.com/binwiederhier/ntfy/issues.
If you want to chat, simply join the Discord server (https://discord.gg/cT7ECsZj9w), or
the Matrix room (https://matrix.to/#/#ntfy:matrix.org).

ntfy 1.18.1 (a3e28e7), runtime go1.17, built at 2022-03-22T03:27:29Z
Copyright (C) 2022 Philipp C. Heckel, licensed under Apache License 2.0 & GPLv2
# ./ntfy-1.18.0
zsh: illegal hardware instruction (core dumped)  ./ntfy-1.18.0
# ./ntfy-1.18.0
NAME:
   ntfy - Simple pub-sub notification service

USAGE:
   ntfy [OPTION..]

COMMANDS:
   help, h  Shows a list of commands or help for one command
   Client commands:
     publish, pub, send, trigger  Send message via a ntfy server
     subscribe, sub               Subscribe to one or more topics on a ntfy server
   Server commands:
     serve   Run the ntfy server
     user    Manage/show users
     access  Grant/revoke access to a topic, or show access

GLOBAL OPTIONS:
   --help, -h  show help (default: false)

Try 'ntfy COMMAND --help' or https://ntfy.sh/docs/ for more information.

To report a bug, open an issue on GitHub: https://github.com/binwiederhier/ntfy/issues.
If you want to chat, simply join the Discord server (https://discord.gg/cT7ECsZj9w), or
the Matrix room (https://matrix.to/#/#ntfy:matrix.org).

ntfy 1.18.0 (0a32925), runtime go1.17, built at 2022-03-16T23:39:33Z
Copyright (C) 2022 Philipp C. Heckel, licensed under Apache License 2.0 & GPLv2
# ./ntfy-1.18.1
zsh: illegal hardware instruction (core dumped)  ./ntfy-1.18.1
# ./ntfy-1.18.1
zsh: illegal hardware instruction (core dumped)  ./ntfy-1.18.1
# ./ntfy-1.18.1
zsh: illegal hardware instruction (core dumped)  ./ntfy-1.18.1
# ./ntfy-1.18.0
NAME:
   ntfy - Simple pub-sub notification service

USAGE:
   ntfy [OPTION..]

COMMANDS:
   help, h  Shows a list of commands or help for one command
   Client commands:
     publish, pub, send, trigger  Send message via a ntfy server
     subscribe, sub               Subscribe to one or more topics on a ntfy server
   Server commands:
     serve   Run the ntfy server
     user    Manage/show users
     access  Grant/revoke access to a topic, or show access

GLOBAL OPTIONS:
   --help, -h  show help (default: false)

Try 'ntfy COMMAND --help' or https://ntfy.sh/docs/ for more information.

To report a bug, open an issue on GitHub: https://github.com/binwiederhier/ntfy/issues.
If you want to chat, simply join the Discord server (https://discord.gg/cT7ECsZj9w), or
the Matrix room (https://matrix.to/#/#ntfy:matrix.org).

ntfy 1.18.0 (0a32925), runtime go1.17, built at 2022-03-16T23:39:33Z
Copyright (C) 2022 Philipp C. Heckel, licensed under Apache License 2.0 & GPLv2
# ./ntfy-1.18.0
NAME:
   ntfy - Simple pub-sub notification service

USAGE:
   ntfy [OPTION..]

COMMANDS:
   help, h  Shows a list of commands or help for one command
   Client commands:
     publish, pub, send, trigger  Send message via a ntfy server
     subscribe, sub               Subscribe to one or more topics on a ntfy server
   Server commands:
     serve   Run the ntfy server
     user    Manage/show users
     access  Grant/revoke access to a topic, or show access

GLOBAL OPTIONS:
   --help, -h  show help (default: false)

Try 'ntfy COMMAND --help' or https://ntfy.sh/docs/ for more information.

To report a bug, open an issue on GitHub: https://github.com/binwiederhier/ntfy/issues.
If you want to chat, simply join the Discord server (https://discord.gg/cT7ECsZj9w), or
the Matrix room (https://matrix.to/#/#ntfy:matrix.org).

ntfy 1.18.0 (0a32925), runtime go1.17, built at 2022-03-16T23:39:33Z
Copyright (C) 2022 Philipp C. Heckel, licensed under Apache License 2.0 & GPLv2
#

[iexos]

Also, strace:

# strace ./ntfy-1.18.1
execve("./ntfy-1.18.1", ["./ntfy-1.18.1"], 0xffffd285c5d0 /* 41 vars */) = 0
mmap(0x23f0000, 17999316, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x23f0000
--- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPC, si_addr=0x3518e44} ---
+++ killed by SIGILL (core dumped) +++
zsh: illegal hardware instruction (core dumped)  strace ./ntfy-1.18.1

[binwiederhier]

The is exactly the behavior I have been seeing in #113 (scroll through that ticket a little). It's random almost. I assumed it was because of the qemu-system-aarch64 emulator, but I'm beginning to think it's not.

There's one thing we can try: I can build you a binary that isn't compressed with upx. Let me do that real quick.

[binwiederhier]

Re-built v1.18.1 without any changes (17M)

curl "https://phil.nopaste.net/crurkdncaQ?a=8rNj0yc4yg" > ntfy-1.18.1-rebuilt
chmod +x ntfy-1.18.1-rebuilt
./ntfy-1.18.1-rebuilt

Built v1.18.1 WITHOUT upx packing (31M)

curl "https://phil.nopaste.net/gO6XY7J7wR?a=2i9mQy0B3I" > ntfy-1.18.1-no-upx
chmod +x ntfy-1.18.1-no-upx
./ntfy-1.18.1-no-upx

Please do try BOTH of them to see if this is an environmental issue.

[binwiederhier]

Also, run each of them many times. In my previous tests it worked sometimes, but not always.

[iexos]

# for i in `seq 1 10000`; do echo $i; ./ntfy-1.18.1-rebuilt >> rebuilt.out 2>&1; echo $? >> rebuilt.rc; ./ntfy-1.18.1-no-upx >> no-upx.out 2>&1; echo $? >> no-upx.rc; done
...
# for e in rebuilt.rc no-upx.rc; do echo $e; cat $e | sort | uniq -c; done
rebuilt.rc
   9871 0
    128 132
      1 2
no-upx.rc
  10000 0

No crashes without compression, ~1% crashes with. And a random different crash for good measure. I leave the interpretation to you :)

[binwiederhier]

That's amazing and sad at the same time. I'll remove the upx step from the ARM builds then.

[binwiederhier]

Try the latest image, v1.19.0.

[iexos]

Yepp, that seems to solve it. Im still a bit puzzled whats really happening here, but I guess it doesn't have too much to do with ntfy anyway. Thank you for your effort!

[binwiederhier]

a bit puzzled whats really happening here

The ntfy binary is quite large because it includes the videos and images from the docs, so I use upx, a binary packer, to compress the binary after it is built. And apparently upx has a bug that makes things crash. That's all I know. So now the binary or ARM is double the size, but it doesn't crash.