Container won't start on RPI Zero W running Raspbian Buster

[tms0]

What are the steps to reproduce?

• Install latest stable docker version (19.03.3, build a872fc2) with containerd version 1.2.10.
• Run the given command docker run -it -p 1880:1880 --name mynodered nodered/node-red

What happens?

• Container exits with status 139, with no logs.

What do you expect to happen?

A working container ;-)

Note : I tried the old image (nodered/node-red-docker:rpi) and it worked perfectly (but with an outdated node.js version).

Please tell us about your environment:

[ ] Platform/OS: RPI Zero W with a fresh install of the latest version of Raspbian Buster

[ ] Browser:

• If you are a command line user please execute the following command and attach the log file.

  docker inspect <container_id> > container.log

container.log

[RaymondMouthaan]

Hi @tms0,

Try one of the following command, where explicit tags are used for arm32v6.

docker run -it -p 1880:1880 --name mynodered nodered/node-red:1.0.1-12-minimal-arm32v6

or

docker run -it -p 1880:1880 --name mynodered nodered/node-red:1.0.1-12-arm32v6

Let me know if that works for you.

Ray

[tms0]

Hi @RaymondMouthaan,

Thanks, it works with the explicit tag !
That will do the trick for me now, but I guess it's not the expected behavior ?

tms

[RaymondMouthaan]

Hi @tms0,

Glad it worked out for you, using explicit tags for arm32v6 on Raspberry Zero. Expected behaviour would be that Docker pulls automatically the correct arm32v6 image for you, instead of pulling arm32v7, however it doesn't. It's a docker issue as described here moby#37647

[tms0]

Good to know, thank you !

As a workaround, do you think it's possible to add rpi tags like in the deprecated repository ? This way we can get the latest image for rpis easily.

Regards,
tms

[RaymondMouthaan]

In case of non rpi zero, docker pulls the right image. Tested on a rpi 3b with Raspbian (32bit) and Ubuntu Server (64bit), where Docker respectively pulls arm32v7 and arm64v8. So the issue only occurs on rpi zero.

Given the above, I'll consider to add a tags related to Raspberry Pi Zero.

This would mean 6 new manifest list with the following tags:

• rpiz (linked to 1.0.1-10-arm32v6 or newer)
• rpiz-10 (linked to 1.0.1-10-arm32v6 or newer)
• rpiz-12 (linked to 1.0.1-12-arm32v6 or newer)
• rpiz-minimal (linked to 1.0.1-10-minimal-arm32v6 or newer)
• rpiz-10-minimal (linked to 1.0.1-10-minimal-arm32v6 or newer)
• rpiz-12-minimal (linked to 1.0.1-12-minimal-arm32v6 or newer)

[dceejay]

Not sure we want to add specific labels like this when it's an upstream problem that should get fixed. Once we have them we can't then easily remove them later. (well we can but it would upset users).

I think maybe we just add extra docs for now - to both the Readme and the dockerhub page ?

[RaymondMouthaan]

@dceejay, I agree - some extra lines in the docs would help Raspberry zero users, without define specific tags in the Node-RED repo.

One remark to this: there is no 'latest' like tag for arm32v6 specific images, which means Raspberry PI Zero users always need to change the full image tag if they want to update to a newer version.

[dceejay]

ok - noted. Will have a go at it later.

[dceejay]

Ok have added some Note: to all the READMEs
will leave open so we remember to check upstream

[tms0]

Hi,

Did you see this comment moby/moby#34875 (comment) ? It seems like reordering the manifest can fix this issue.

[dceejay]

oh good find - worth a try ? @RaymondMouthaan ?

[dceejay]

hmm - ours look like this - so v6 is already ahead of v7

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2419,
         "digest": "sha256:1a60576578f2dab7e813c14be7448223ade6591c5d8d183c181b59aa1a10bb7b",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2419,
         "digest": "sha256:8175e15337195d2daa67623fe84b4c9c3d5ac9bedea8a0493fa85ef9694f35de",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v6"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2418,
         "digest": "sha256:bdb6962803a722c457e60b27f079468c59129f778333daf278c0819f7c0ec4d0",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2419,
         "digest": "sha256:6cdefce9f8f4f7ff5d1ae2b8df3fcd5bd64d8611bf8c6dc4d836c57016561851",
         "platform": {
            "architecture": "arm64",
            "os": "linux",
            "variant": "v8"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2419,
         "digest": "sha256:61c641969dc254e600336e575fffff293f2ae42ca21005d04428ee23db482a02",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

[tms0]

Hi,

@dceejay, there is a new proposed workaround : moby/moby#34875 (comment)

Can you have a look please ?

[dceejay]

if you have time to test it and provide a pull request that would be good. The comments following that still say it is not recommended.

[dceejay]

My testing today - via https://hub.docker.com/r/nodered/node-red-dev/tags - setting the variant to v6l - still loaded the v7 for pizero - so by all means prove me wrong and show me how, but I have now looked.
Still needs to be fixed upstream

[tms0]

Sorry, I didn't mean to waste your time, thanks for trying !

[dceejay]

Not a problem - if you do find a fix and manage to get it to work - please do let us know.

[RaymondMouthaan]

Just came across this (still) open issue and notice that moby/moby#34875 has been merged just 4 days ago, perhaps worth trying?

[dceejay]

I see several thing being merged in that thread. I thought I tried the proposed workaround a few comments above and it didn't work... what has been merged to now make it work ?
(was it to just change the arch to v6l again ?)

[tms0]

Hi @dceejay,

I can confirm now it works on my rpi zero (Docker version 20.10.6, build 370c289 / containerd containerd.io 1.4.4) :

$ docker run --rm nodered/node-red:latest

> node-red-docker@1.3.4 start /usr/src/node-red
> node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS "--userDir" "/data"

3 May 09:32:05 - [info]

Welcome to Node-RED
===================

....

I think the fix is this one : containerd/containerd#4530

[dceejay]

Sweet ! So Ok to close this now then :-)