Podman failing to start on UDM SE v3.0.10

[mayankst]

Describe the bug
Podan is failing to start on UDM SE v3.0.10 with following error

Oct 18 12:28:33 UDM-SE podman[1465457]: Error: OCI runtime error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented

To Reproduce
Steps to reproduce the behavior:
Start the pi-hole container as service.

ExecStart=/usr/bin/podman run --cidfile=%t/%n.ctr-id --cgroups=no-conmon --pull=always --rm --sdnotify=conmon --replace -d --network dns --name pihole -e "TZ=America/Los Angeles" -v "/mnt/data/etc-pihole/:/etc/pihole/" -v "/mnt/data/pihole/etc-dnsmasq.d/:/etc/dnsmasq.d/" --dns=127.0.0.1 --dns=1.1.1.1 --dns=8.8.8.8 --hostname pi.hole -e VIRTUAL_HOST="pi.hole" -e PROXY_LOCATION="pi.hole" -e ServerIP="192.168.4.3" -e IPv6="False" pihole/pihole:latest

UDM Information

• UDM SE
• 3.0.10
• 7.3.73

[chrisdooks]

+1
I think it's because of the upgrade of Debian, it requires Podman v4 to work with the new kernel.

[mayankst]

@chrisdooks Do you know how to build new podman version for unifi? I tried following the instructions in the readme but doesn't seem to work. @boostchicken Can you please help?

If you can provide the instructions to build, then I can build and open a PR.

[chrisdooks]

I'm afraid not, it's above my level of knowledge.

[mayankst]

@boostchicken can you please help. I tried building podman on wsl following your instructions but I am getting compile errors due to missing libseccomp. I tried installing arm64 version of that lib but wasn't successful as dependencies won't install. Main base dependency is libc6:arm64, when I try to install that, apt wants to remove amd64 version of that. So, I am not sure what to do next. Can you please help as pi-hole is completely down due to this incompatibility.

[mayankst]

So, I am able to compile the podman but container still won't start. Keep getting Exit status 1.

[sanderdewit]

I'm interested in this as well.
@mayankst can you share the compiling instructions?

[T3LC0]

@mayankst how did you solve the multiarch problem with getting that libc6:arm64 package installed? Trying to compile this myself since support had me upgrade to EA to fix a problem that turned out to not be fixable and I want to run home bridge on my UDM SE with the EA software.

[mayankst]

@T3LC0 Yea, I got it to compile but unfortunately looks like Unifi have borked the any ability to run containers in v3 of the OS. There are some major dependencies missing from the OS which are causing runc to not work. At this point I have given up (spent over a month on this researching and debugging) and moved on to alternative solution.

[dquach93]

@mayankst whats the alternative solution you moved to?

[mayankst]

@mayankst whats the alternative solution you moved to?

Running Ad Guard home on home assistant. Could have done pi-hole on rPi as well, but I already have the home assistant so it was easier.

[dquach93]

@mayankst whats the alternative solution you moved to?

Running Ad Guard home on home assistant. Could have done pi-hole on rPi as well, but I already have the home assistant so it was easier.

Ah i see. Im using this for wpa_supplicant. Didnt see this issue till after updating to 3.0.12 today...

[emilvissing]

@T3LC0 Yea, I got it to compile but unfortunately looks like Unifi have borked the any ability to run containers in v3 of the OS. There are some major dependencies missing from the OS which are causing runc to not work. At this point I have given up (spent over a month on this researching and debugging) and moved on to alternative solution.

did you try setting up podman on 3.x using the latest podman/runc/conmon that @boostchicken pushed a couple of days ago?

[TotalGriffLock]

@T3LC0 Yea, I got it to compile but unfortunately looks like Unifi have borked the any ability to run containers in v3 of the OS. There are some major dependencies missing from the OS which are causing runc to not work. At this point I have given up (spent over a month on this researching and debugging) and moved on to alternative solution.

did you try setting up podman on 3.x using the latest podman/runc/conmon that @boostchicken pushed a couple of days ago?

Can confirm same behaviour here on a UDM-SE on v3.0.13 and that's with @boostchicken latest published podman. It would seem that the BPF_CGROUP_DEVICE call has been removed from the new kernel on v3 of the OS.

[kennylerma]

Just noting I have the same issue :( I grabbed the latest podman for UDM Pro SE and still the same start error.

[boostchicken]

https://github.com/fabianishere/udm-kernel-tools

You need that for the UDM-SE

[simkin]

https://github.com/fabianishere/udm-kernel-tools

You need that for the UDM-SE

Kindly elborate as I have some doubts this will work: fabianishere/udm-kernel-tools#82
If it does work, which custom kernel must be loaded ? Edge?

[mazzy89]

@boostchicken the project is not yet ready for UDM Pro SE.

[mazzy89]

Lookign at the kernel file what it has been stripped out by the stock kernel is the CONFIG_BPF_SYSCALL kernel parameter. it must be set to y because Podman same as other containers runtime use nowadays ebpf to hooks programs into the kernel. The only way here is to have a custom kernel into our UDM SE and to achieve this, we need to bring the support for UDM SE into here https://github.com/fabianishere/udm-kernel-tools and have installed into our UDM SE the edge version.
I will try to check what it is needed to be done there.

[andrewmiskell]

@mayankst whats the alternative solution you moved to?

Running Ad Guard home on home assistant. Could have done pi-hole on rPi as well, but I already have the home assistant so it was easier.

Ah i see. Im using this for wpa_supplicant. Didnt see this issue till after updating to 3.0.12 today...

On 2.x firmwares, you can just install the Debian package for wpa_supplicant instead of running it in a container so you can break the reliance on podman for that. (Assuming you're using it like I am with AT&T Fiber).

Haven't tried it on 3.x yet (I'm using a UDM and just got the 2.x firmware yesterday).

[SirUli]

Apparently rootless containers fail. e.g. the https://hub.docker.com/r/tusc/chrony-udm still works for me.

[foxcris]

I am on UDM SE v3.0.13. There you can install podman directly via apt from the debian repository. Have you tried to use that package? Installation runs without any errors but i haven't tried to run a container yet.
Here the output of podman info:

host:
  arch: arm64
  buildahVersion: 1.19.6
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 4
  distribution:
    distribution: debian
    version: "11"
  eventLogger: journald
  hostname: UDM-SE
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.19.152-ui-alpine
  linkmode: dynamic
  memFree: 161476608
  memTotal: 4137881600
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 7509372928
  swapTotal: 7516188672
  uptime: 191h 36m 17.01s (Approximately 7.96 days)
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus: {}
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.15
  OsArch: linux/arm64
  Version: 3.0.1
```

[mazzy89]

@foxcris it won't work. This is not a problem of podman. The problem is in the OS kernel which come with the newer firmware version.

[foxcris]

@mazzy89 You are right. Just tried the hello world container and i am getting:

podman run hello-world
Resolved "hello-world" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/hello-world:latest...
Getting image source signatures
Copying blob 7050e35b49f5 done
Copying config 46331d942d done
Writing manifest to image destination
Storing signatures
Error: open /proc/self/uid_map: no such file or directory

[bennettp123]

@foxcris I think the issue is probably crun—it requires the kernel to be compiles with CONFIG_USER_NS, but the UDM kernels aren't.

What happens if you switch to the older runc ociRuntime instead?

apt-get remove crun
apt-get install runc

[jimmy-ungerman]

What happens if you switch to the older runc ociRuntime instead?

Just tried this on my UDM SE v3.0.13 and still the same issue.

[foxcris]

@bennettp123
Same error when i use runc instaed of crun.

[foxcris]

Isn't it an option to run a standard debian linux kernel?

[frasderp]

@foxcris any progress on this?

[foxcris]

No. I didn't had time to try to use a standard Debian kernel. Don't want to brick my udm se. Till now I can install podman but it's not working due to the missing kernel features.

[cftechwiz]

FWIW - I did actually brick my UDM on one occasion when I accidentally executed an rm -rf at the wrong subfolder and removed a good chunk of the OS. Turns out there is a neat trick to bring it back to stock from an embedded image.
The instructions I got from Support were as follows:

1. Remove the unit from your network and disconnect the cables from the unit.
2. Press down the reset button for 40+ seconds without power and cables.
3. Release the reset button and power the unit.
4. Again press the reset button for 15+ seconds.

I did this and it worked like a charm.

[mazzy89]

It's quite hilarious how the lack of a container runtime in a linux box is send to panic a lot of ubiquiti users.
The industry has run without containers runtime for ages.
You can do everything that you can do with podman also without podman.
I run bgp having bird binary on the machine. You can persist packages as well.
It might be not handy as running a container runtime. Containers is well know thay have made deployment and lifecycle of the app easier but still you can do eveyrhing even without podman.

[wjbridge]

I was trying to get ADgaurdhome running without podman but I cannot attach it to a network interface (i.e. 192.168.1.0) on port 53. Port 53 was taken by another program. Does anybody have any guidelines on how to do this without podman?

[foxcris]

You should bind the already installed dns resolver (probably dnsmasq) to local host and configure adguardhome to bind to your specific local network network ips. An other option is to bind dnsmasq to a different port. 19.01.2023 01:23:18 wjbridge ***@***.***>:

…

I was trying to get ADgaurdhome running without podman but I cannot attach it to a network interface (i.e. 192.168.1.0) on port 53. Port 53 was taken by another program. Does anybody have any guidelines on how to do this without podman? — Reply to this email directly, view it on GitHub[#428 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AACYGU5FQXGAWF3QGV72SA3WTCCOXANCNFSM6AAAAAARINPA2Q]. You are receiving this because you were mentioned.[Verfolgungsbild][https://github.com/notifications/beacon/AACYGU7Y4K656B57WVMVUR3WTCCOXA5CNFSM6AAAAAARINPA2SWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSTHF5LO.gif]

[cftechwiz]

It's quite hilarious how the lack of a container runtime in a linux box is send to panic a lot of ubiquiti users. The industry has run without containers runtime for ages. You can do everything that you can do with podman also without podman. I run bgp having bird binary on the machine. You can persist packages as well. It might be not handy as running a container runtime. Containers is well know thay have made deployment and lifecycle of the app easier but still you can do eveyrhing even without podman.

Just an easy way to leverage MacVLAN, and eliminate port conflicts, maintain isolation.

[wjbridge]

You should bind the already installed dns resolver (probably dnsmasq) to local host and configure adguardhome to bind to your specific local network network ips. An other option is to bind dnsmasq to a different port. 19.01.2023 01:23:18 wjbridge @.***>:
…
I was trying to get ADgaurdhome running without podman but I cannot attach it to a network interface (i.e. 192.168.1.0) on port 53. Port 53 was taken by another program. Does anybody have any guidelines on how to do this without podman? — Reply to this email directly, view it on GitHub[#428 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AACYGU5FQXGAWF3QGV72SA3WTCCOXANCNFSM6AAAAAARINPA2Q]. You are receiving this because you were mentioned.[Verfolgungsbild][https://github.com/notifications/beacon/AACYGU7Y4K656B57WVMVUR3WTCCOXA5CNFSM6AAAAAARINPA2SWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSTHF5LO.gif]

@foxcris Do you know how to do this? I don't really know how to change dnsmasq to a different port within Unifi or change it to local host only for all networks defined. Any help for a noob would be appreciated.

[boostchicken]

It's quite hilarious how the lack of a container runtime in a linux box is send to panic a lot of ubiquiti users. The industry has run without containers runtime for ages. You can do everything that you can do with podman also without podman. I run bgp having bird binary on the machine. You can persist packages as well. It might be not handy as running a container runtime. Containers is well know thay have made deployment and lifecycle of the app easier but still you can do eveyrhing even without podman.

Let us not judge our container loving users! Alas, it does enable people to do things that escape their current knowledge / expertise.

I am sure everyone would be grateful if you wrote steps on how to achieve your setup without containers and contributed them to the repo @mazzy89

[boostchicken]

You should bind the already installed dns resolver (probably dnsmasq) to local host and configure adguardhome to bind to your specific local network network ips. An other option is to bind dnsmasq to a different port. 19.01.2023 01:23:18 wjbridge @.***>:
…
I was trying to get ADgaurdhome running without podman but I cannot attach it to a network interface (i.e. 192.168.1.0) on port 53. Port 53 was taken by another program. Does anybody have any guidelines on how to do this without podman? — Reply to this email directly, view it on GitHub[#428 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AACYGU5FQXGAWF3QGV72SA3WTCCOXANCNFSM6AAAAAARINPA2Q]. You are receiving this because you were mentioned.[Verfolgungsbild][https://github.com/notifications/beacon/AACYGU7Y4K656B57WVMVUR3WTCCOXA5CNFSM6AAAAAARINPA2SWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSTHF5LO.gif]

@foxcris Do you know how to do this? I don't really know how to change dnsmasq to a different port within Unifi or change it to local host only for all networks defined. Any help for a noob would be appreciated.

If you don't already know how to do this I suggest sticking with containers and letting the CNI handle it for you. You need to create a new netns and have your programs run in it. The CNI plugins just automate this for you.

[mazzy89]

I do love containers too and I'm a big supporter of kubernetes and anything related to containers technologies as you can see from my GH. But containers are a way among many ones to get to the final result. Let's not sediment us to just use container in a box where officially they are not allowed anymore from the vendor. Adaption is the way.

The way how I achieve the same thing is via saltstack. I'm not sure if users here they do want to learn saltstack and manage their router with that. If there is the interest I will definitely share it but I doubt it.

[boostchicken]

I do love containers too and I'm a big supporter of kubernetes and anything related to containers technologies as you can see from my GH. But containers are a way among many ones to get to the final result. Let's not sediment us to just use container in a box where officially they are not allowed anymore from the vendor. Adaption is the way.

The way how I achieve the same thing is via saltstack. I'm not sure if users here they do want to learn saltstack and manage their router with that. If there is the interest I will definitely share it but I doubt it.

Nowhere did it say containers are not allowed. There just isn't support for it out of the box anymore. I would argue containers are better because it keeps all customization and stuff isolated, leading to less issues.

[boostchicken]

The way how I achieve the same thing is via saltstack. I'm not sure if users here they do want to learn saltstack and manage their router with that. If there is the interest I will definitely share it but I doubt it.

I didn't think anyone would have interest in this repo, much less it growing to become the bible on UniFi OS Console customization. Put it out there! If it works well you may kill the need for containers, the community will decide!

[mazzy89]

Ok @boostchicken i will definitely.

[aescolastico]

is there an alternate way to run homebridge on udmse outside of podman?

[aescolastico]

this still seems above my head, but it seems from the convo, that it might be a while before the podman/udm kernel gets updated.
so for now ill just install homebridge directly on the udm via their debian 11 package https://github.com/homebridge/homebridge/wiki/Install-Homebridge-on-Debian-or-Ubuntu-Linux

[aescolastico]

it just occurred to me that homebridge probably cant be discovered by homekit if its advertising on the same ip as your networks default gateway. which is the case when you install it on the udmse.

[aescolastico]

is it possible to add a virtual interface to the debian install on the udmse? the purpose would be to assign an ip to homebridge that isnt the gateway address. perhaps that could be used as an alternative to container networking.

[jimmy-ungerman]

@aescolastico No, you would still be able to run it on the gateway. Think of it this way, the Unifi console is a web server that is running on the same IP as your network's default gateway. Even when you run it in docker, you still have to expose a port to access it on the host IP of your UDM.

I would start by installing the package, and trying to access the webserver using your UDM's IP and port that homebridge runs on. Or even trying to see if homebridge.local works for it.

[aescolastico]

i did install the package, but was unable to add the homebridge qr code to homekit. i thought it was related to it advertising on default gateway but maybe its something else.

[jimmy-ungerman]

How were you trying to add it? Were you able to go to the Homebridge UI and scan the QR code, or were you just trying to automatically add it from the Home app on your device?

[aescolastico]

scanning the qr code. The bridge is listening on 51206.

[aescolastico]

@boostchicken would you have any idea on how i could make the networking work out here?

[aescolastico]

@jimmy-ungerman you were right about it not being an issue advertising on gateway. I was able to pair the homebridge by using a different advertiser, specifically "ciao". not super relevant to podman issue but maybe this info will be useful for others. thanks.

[Stroh37]

it just occurred to me that homebridge probably cant be discovered by homekit if its advertising on the same ip as your networks default gateway. which is the case when you install it on the udmse.

You can definitely install it using the same ip address as the gateway without using the cni networking tool and HomeKit will still see it. The only issue I had was homebridge unifi protect can’t see the gateway to discover protect cameras. Same with scrypted. So in that case cni pluggin is necessary so homebridge or scrypted can use different ip than gateway.

[peacey]

For those who want to use containers, you should all consider using systemd-nspawn instead. It works well on 3.x and you can create and run any system container (debian, arch linux, fedora, etc) and install whatever you want within it (pihole, nginx, etc). This way you can run a separate system container and your changes within that container won't break the main Unifi system with conflicting dependencies, and Unifi can't break your container. You still can't run podman in the container since it's the same kernel, but you can natively install anything in the container with apt-get or old-school manual methods like on any linux system. See my other post briefly talking about nspawn.

I've been running a debian unstable container on UDMP 3.x with pihole, nginx, dnsdist, and some other programs, and it works really well. Even the starting at boot is managed natively by systemd so it's a natural solution if you're familiar with linux. Systemd-nspawn also natively supports creating macvlans and other private networks in case you want to isolate the container network.

[georgeboot]

@peacey would you be willing to create a step by step tutorial for this?

Ive come this far:

apt install systemd-container debootstrap

I want to create a container for AdGuard home uses my dns network. With podman, I created a custom macvlan bridge network following these steps. How would I do a similar thing using nspawn? Once I have a shell in the "container", I will be able to install my tools.

Thanks!

[andrewmiskell]

@peacey would you be willing to create a step by step tutorial for this?

Ive come this far:

apt install systemd-container debootstrap

I want to create a container for AdGuard home uses my dns network. With podman, I created a custom macvlan bridge network following these steps. How would I do a similar thing using nspawn? Once I have a shell in the "container", I will be able to install my tools.

Thanks!

ArchLinux Wiki to the rescue. :D https://wiki.archlinux.org/title/systemd-nspawn

[cwyse]

Have to comment on the container use first ( a little further back on Jan 24).

To me containers are the way to go, especially when they have relatively low overhead like docker or podman. Wasn't a huge fan of VM Ware and the other one... (can't even remember the name now) - they were too big and slow. As @mazzy89 stated, it can all be done without them or with other tools. Who wants to do that? The biggest problem for me is maintaining a standard configuration - if I install everything under the sun on my host, it's a nightmare to maintain. Much easier to modify a build script for a container, or even better, pull a prebuilt container that has the change I need. It doesn't matter much which system is used (although I don't really want to continually have to learn new systems that do the same thing), I just need my application to be consistently configured and encapsulated.

[cwyse]

Now regarding this issue - I woke up this morning and everything had stopped working. No updates done overnight, but the APs had stopped and I only had wired access to the router. I don't allow automatic updates, so I (maybe foolishly) updated all my firmwares, including moving to v3.x.x of the UDM. I have several containers that typically run on my system - pihole, mdns, unbound, etc. Anyway, tried to get things working again, and got stuck trying to get pihole running. Downloaded image again, created container, then got the user namespace error.

Looking at my kernel config, I see, as others stated, that there are some kernel configuration options that are problematic. In particular the user namespace option, which I believe is preventing podman from working.

My question is regarding using the --userns=host option of podman. I believe that this option would circumvent the user namespace issue by using the host namespace. I tried it but was unable to resolve the issue. Not sure if I did something wrong. Has tried using that option? Are there other problems that I'm not aware of? Is anyone bothering to look at this now?

@boostchicken Thanks for all your work. It must get annoying when Ubiquiti changes keep breaking the udm_boot...