Unable to start the build-nilrt container due to missing PYREX_UID

[Greg-Freeman]

Looks like my container doesn't have the PYREX_UID environment variable set. Should it? If so, how do I go about doing this and what should its value be? Right now my container won't start.

[amstewart]

Could you give a summary of what steps you have done so far? And of what task you're running when you get that PYREX_UID failure? And what branch refs are you using for nilrt.git and pyrex.git?

[Greg-Freeman]

To answer your first questions, I get the PYREX_UID error inside the docker container when I try to start it. It's in the docker log.
pyrex submodule: 00b8ee
meta-nilrt: d0e85
branch: nilrt/master/kirkstone

I have made it through step 3. I was trying to run on windows, but I have since abandoned that idea and moved to an Ubuntu VM, due to issues with line endings in the python scripts etc. I expect attempting this on windows was causing a slew of issues so I abandoned that idea.

So now I'm on my Ubuntu VM and made it through step 3. The docker images exist but when i try to run the ni-oe-init-build-env script I get the following error:

invalid mount config for type "bind": bind source path does not exist: /tmp/tmp96jztxsb

So a couple of questions.

1. does the init script set the PYREX_UID variable?
2. should the init script be run on the host machine or in the docker container?

If 1 is true, then the container is probably failing to start with the PYREX_UID error due to the init script failing. In that case I will need to figure that out. With regards to the init script failure, through some preliminary digging I can't find where the configuration is that is attempting to mount some tmp directory. So no idea what's going on there.

[amstewart]

Using Ubuntu is a better idea. I don't think anyone is testing either the NILRT build or pyrex on Windows, and I wouldn't expect it to work.

does the init script set the PYREX_UID variable?

Not directly. PYREX_UID is set within the pyrex tool's pyrex.py script, based on the UID of the running process. So it is eventually called by the initscript, but isn't something you're expected to set or toggle.

should the init script be run on the host machine or in the docker container?

The init script should be sourced from a shell running on the host machine. Once the init script has dropped you into the :build/ directory, all your bitbake calls from the host should be routed into the container context automatically.

With regards to the init script failure, through some preliminary digging I can't find where the configuration is that is attempting to mount some tmp directory. So no idea what's going on there.

When pyrex is setting up the container, it writes environmental information into a temporary directory, and then mounts it into the container. I presume that is what the failing step is talking about. Is there anything non-standard about how you have configured the Ubuntu VM's /tmp filesystem? Can you run mktemp -d?

Also, what version of Ubuntu are you using on the VM? Docker changes their API frequently enough that it might be incompatible with the latest pyrex tool.

[Greg-Freeman]

Ok, so far everything I've done seems to align with the steps here. I'm really unsure what the deal is with the PYREX_UID, but it seems like all the issues I'm having are related to the Pyrex container setup. Both the UID isn't being set and it seems the tmp directory isn't being created and/or mounted. I'll come back with answers to your additional questions, but I am currently running an Ubuntu 22.04 LTS version in Hyper-V on Windows.

edit:
mktemp -d works
docker version: 24.0.6, build ed223bc

[amstewart]

As a sanity check: can you start up a simple debian/ubuntu container in the Ubuntu host? Does that throw an error?

[Greg-Freeman]

I can but I was actually trying to do some additional stuff with docker on an Ubuntu VM running with Hyper-V and some of the linux-only networking options were not working how I'd expect. So I'm modifying my partitions a bit and setting my machine to dual boot Ubuntu and windows. So first I try this outside of a VM and directly on my Ubuntu OS. Then I'll come back and give feedback.

I can say I'm running other docker containers based off the latest Alpine image on my VM and they do start up fine and I run tshark in them. It seems like for some reason the PYREX_UID variable isn't being set or isn't persisting and maybe it's an issue running inside a VM?

[Greg-Freeman]

Edit: wait -- not resolved yet but maybe getting closer. Realized my container wasn't running and once starting it the error is back. But I'm going to blow away my whole cache and try again. I have a strange feeling that docker desktop on linux may still have been the issue, so I'm leaving the below for now.

The road to the solution is winding. But here is what happened.

TL;DR; I had Docker desktop installed on Linux which still runs a VM under the hood.

Longer version:
For an unrelated effort I was trying to read layer2 traffic within a docker container. I couldn't see it for the life of me despite many systems engineers telling me it should work if running my container in host mode. I even went so far as to making my machine dual boot and booting directly into Ubuntu. Finally I came across a thread buried in the docker forums about how host mode won't work even on a linux machine if you're using Docker Desktop because it still runs inside a VM to keep the architecture inline with how it works on windows. Along with this thread were a couple other enraged developers complaining this isn't noted in the Docker documentation anywhere.

After many curse words were said, I uninstalled docker desktop and made sure only the docker engine was installed. Then I started seeing network traffic in my container.

It got me thinking that maybe there was some sort of issue with the PYREX_UID variable not working because I was using Docker desktop, despite it being on Linux.

So I deleted my nilrt repo and started from square one. After doing that I got one error, but googling that error took me to this thread:
https://stackoverflow.com/questions/67642620/docker-credential-desktop-not-installed-or-not-available-in-path

So I took the suggestion from there and changed credsStore to credStore and that worked.

Check your ~/.docker/config.json and replace "credsStore" by "credStore"

{
"stackOrchestrator" : "swarm",
"experimental" : "disabled",
"credStore" : "desktop"
}

And finally I ended up having the scripts fail because I wasn't running things as sudo despite my user being the Docker group, which lead me to this:
https://github.com/sindresorhus/guides/blob/main/docker-without-sudo.md

And step 4 from that page is:
4. Change the context from Docker Desktop to Engine
If you installed Docker Desktop first, then removed it and installed the Docker Engine, you may need to switch the Docker context with this command:

$ docker context use default

[Greg-Freeman]

Man, I just had a realization. I was trying to start the docker container using docker run -it etc etc... My guess is I don't need to, nor am I supposed to do that? I must have just been on autopilot thinking I had to spin up the docker container. Alas, when I run bitbake nilrt-safemode-rootfs it works.

My bad...

What may have thrown me was this line: "Enter the NILRT build environment. Sourcing the init script the first time will automatically setup your pyrex container shim." I assumed I had to spin up a container from the image to work inside it. I somehow got wrapped around the axle and forgot you had mentioned this:

The init script should be sourced from a shell running on the host machine. Once the init script has dropped you into the :build/ directory, all your bitbake calls from the host should be routed into the container context automatically.

[amstewart]

Ah. Yeah, that all sounds like the kind of docker configuration hassle that I was expecting to come out of this. :) Well, I'm glad you were able to get it figured out.

I posted some updated language in the README doc, plus a note about avoiding Docker Desktop, in #250.