Replies: 91 comments 63 replies
-
|
This is a limitation on the systemd side. They will only accept notifications, or PID files, that are created by or sent by root, for security reasons - even if the User and Group of the unit file are explicitly set to start the process as a non-root user. Their recommendation was to start the container as a user service of the user in question via |
Beta Was this translation helpful? Give feedback.
-
|
Previous discussion: #9642 |
Beta Was this translation helpful? Give feedback.
-
|
Thank you both. For now I've worked around it by managing the service under the user's systemd which is clunky to say the least. I don't understand systemd's security argument - if the process is run as a given user, why would systemd not allow that user's process to send sd_notify? Who else could? But I guess this is no flaw of podman. #9642 mentions some code changes that need to happen to podman for sd_notify, what are those? And have they progressed since March? I guess you could close this issue or use it to track progress. |
Beta Was this translation helpful? Give feedback.
-
Yes, there is some progress. The main PID is now communicated via sd notify but there are still some remaining issues. For instance, |
Beta Was this translation helpful? Give feedback.
-
|
I think the next big thing to tackle is finding a way how to lift the |
Beta Was this translation helpful? Give feedback.
-
But even that is rejected by systemd, as seen in the logs above. |
Beta Was this translation helpful? Give feedback.
-
|
I fear there's not much Podman can do at the moment. |
Beta Was this translation helpful? Give feedback.
-
|
Only after solving this problem can become truly rootless. So I have to keep using the root account for now. |
Beta Was this translation helpful? Give feedback.
-
|
Is there a quick overview what, at the moment, the best approach / workaround is for starting podman containers with systemd as a specific non-root user? Furthermore, if a container is run as root, is there a workaround how to change the ownership of files and directories created inside the container (in a bound volume) to a specific host user? |
Beta Was this translation helpful? Give feedback.
-
use use |
Beta Was this translation helpful? Give feedback.
-
The services need to be started and managed as the specific non-root user. Using the |
Beta Was this translation helpful? Give feedback.
-
For the moment my workaround is to run such containers in a systemd --user. This means that for every system service I want to run as a rootless container, I need to create a separate system user, enable linger, and run a separate systemd --user instance for that user. It works but it's clunky, e.g. restarting Nginx is Inside these rootless containers root is mapped to the system user, which is a different uid for each service. If something inside the containers runs as non-root, that gets mapped to a high-numbered host uid by default. However with some magic on the host you can map a specific non-root uid in the container to a host uid of your choice, which can then be mapped to a different non-root uid in a different container running under a different user. I should probably document my setup one of these days... |
Beta Was this translation helpful? Give feedback.
-
|
@Gchbg If you are running a recent systemd version (for instance by running Fedora 35), I think you could run No need to set |
Beta Was this translation helpful? Give feedback.
-
Is that
How does that relate to what @Gchbg and @eriksjolund wrote above? Do I have to run several instances of systemd or is there another way? For systemd beginners like me, it is quite difficult to understand the various layers of abstraction and user permission between systemd, host processes and containers. After all, I would assume that this is the use case for 80 % of the users: run some container service that gets restarted automatically when the machine boots and that is as restricted as possible (by means of user permissions). |
Beta Was this translation helpful? Give feedback.
-
I got it wrong, modifying UID and GID via env requires entrypoint.sh。 https://docs.docker.com/engine/security/userns-remap/ |
Beta Was this translation helpful? Give feedback.
-
|
I've been working on setting up a bare-metal environment that runs exclusively in ram on fcos and boots over pxe with an ignition file. My goal is to go from zero -> running github actions container / terraform cloud agent container securely without manual intervention. Admittedly I am pretty unfamiliar with linux internals and systemd, but I am trying to familiarize myself. Ignition only supports systemd for launching services, so understanding this is pretty crucial. At this point I've been able to figure out every step of my config, but getting rootless containers working has been a thorn in my side. Trying to set the I'm a bit stumped here and am reverting back to running as root for now, but I'm open to any pointers (including this ignition process as a whole):
|
Beta Was this translation helpful? Give feedback.
-
|
+1 As quadlets were created with the idea of "integrating better with systemd", not supporting the systemd way of managing rootless services does not make sense. |
Beta Was this translation helpful? Give feedback.
-
|
What's the next-best-thing approach for running a container as systemd service, but as a non-root user that's easy and repeatable?
I would happy if all uids in the container were collapsed to a single user and I could specific that user with |
Beta Was this translation helpful? Give feedback.
-
rootless podman is "running podman in a user namespace" + "running workload in a user namespace". If you use --userns with root, you only lose the first half of rootless. Once the container runs there are no differences between root with userns and rootless. The confusion here seems to be around User=. That doesn't set the environment for rootless to work correctly: it does not set the user session so there are no tmp dirs, as well as no journal for logs. |
Beta Was this translation helpful? Give feedback.
-
|
I will note that the issues @giuseppe raised are fundamentally systemd problems - if those were fixed, rootless Podman would work with User= without issue. If you'd like this to work, please talk to the systemd maintainers; there's nothing the Podman team can do about it. For reference, we have had this conversation with the systemd team in the past, and the recommended solution back then was to use |
Beta Was this translation helpful? Give feedback.
-
@rhatdan Is there a way to work around this by e.g. changing the overflowuid value for a user namespace? Or is that a system-wide setting? |
Beta Was this translation helpful? Give feedback.
-
|
No way that I know of . |
Beta Was this translation helpful? Give feedback.
-
Any link to this discussion? I've revisited this discussion several times and I always find this feature gap to be furstrating and the underlying reasons seem quite arcane. It /feels/ like something should /just work/ but it doesn't. Perhaps the context of these previous discussions could shed some further light. |
Beta Was this translation helpful? Give feedback.
-
|
I was able to run rootless podman containers using systemd units with "User=" without using I just use normal service commands like Podman was run with the following options, where 1111 is the serviceuser's uid: Systemd unit file has the following lines (I posted only the most important lines): Subuids was configured for the serviceuser: Also linger state was enabled for servicesuser account: The key option is
@eriksjolund posted this option in his message. |
Beta Was this translation helpful? Give feedback.
-
|
Interesting, thank you. We use something similar except without |
Beta Was this translation helpful? Give feedback.
-
|
This seems to be the magic combination that makes
|
Beta Was this translation helpful? Give feedback.
-
|
An attempt to provide clarity to this method, here are the docs for all the mentioned options. I have not yet tested this method myself. Let's start with with podman options: podman run options
systemd unit options.
Interpretation
SummaryIt's amazing that someone found a combination of flags and directives which seem to enable User=, because even from reading all their docs together, it still not clear how they could work together to accomplish that! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the overview! Shallow and pedantic, but:
conmon
conmon Edit: Some clarifications, as far as I know of them:
The CID file is used as a sentinel so the service doesn't leak resources while stopping. Cf. #13236
Yes.
It's not the conmon process that handles service readiness, but the podman process. Since systemd looks at the conmon process to determine service uptime,
No, delegation here refers to cgroup handling. Normally only systemd is allowed to create and remove cgroups, but with |
Beta Was this translation helpful? Give feedback.
-
|
Same issue applies to systemd units for creating/starting pods. |
Beta Was this translation helpful? Give feedback.
-
|
I wrote a workaround for this case, using |
Beta Was this translation helpful? Give feedback.
-
|
Hi, In case it’s useful to anyone, I wanted to write up a “report” of my experiences with this unsupported use-case and what currently works for me and what doesn’t on a reasonably up-to-date “normal” Debian system with fairly modern versions of all the relevant tools. Note tl;dr - Everything seems (?) to work fine except for Starting point service templateThe process I followed to get started was to create My thinking was that this would provide a good starting template that incorporates the “best practices” of how the upstream developers currently intend podman to be used under systemd. However, I converted the quadlets to regular services because I found I was modifying them so much that it obviated the benefits of the nice simplified quadlet syntax, and the “overhead” of keeping slightly different service definition files in a slightly different filesystem path from all other services made it less useful to me to use the quadlet format. While some of the changes I made to the service definitions were indeed to make them work as To keep things comprehensible, I’ll be showing a really minimal example that strips out as many “irrelevant” details as possible, and just runs the container in as direct of a way as possible. In reality, you’d want to configure some dependencies, perhaps set up a dedicated network, configure restart policies, perhaps reference images with their sha256 digest, etc. Note that any supporting services, like ones that pull images or set up networks, will also need the relevant My working minimal example looks like this (click to expand):
|
Beta Was this translation helpful? Give feedback.
-
|
I think I also encountered the issue of failing |
Beta Was this translation helpful? Give feedback.
-
|
The health check failure is because podman internally uses
|
Beta Was this translation helpful? Give feedback.
-
|
FWIW, I'm testing a solution for secure, scalable provisioning of rootless Podman environments for external (AD) users on RHEL: sempernow/podman |
Beta Was this translation helpful? Give feedback.
-
|
I don't get why it is required to get podman to fork to run in user mode. Shouldn't it be much easier, to just run podman in the foreground as the controlling process, and it forwards all the signals to the container it is controlling? It seems that this should make the implementation much simpler? |
Beta Was this translation helpful? Give feedback.
-
|
It is common practice for Podman services (rootless and rootful) to be |
Beta Was this translation helpful? Give feedback.
-
|
I mean, sure, but wouldn't it be way easier to fix those race conditions (as that is something we can control here) instead of waiting for systemd to eventually be able to run the detached containers? |
Beta Was this translation helpful? Give feedback.
-
|
Using |
Beta Was this translation helpful? Give feedback.
-
|
additionally, if you have multiple containers running in different user namespaces e.g. for security reasons, it is not possible to define any denpendency rules. But with USER option of systemd even this would possible. On top, it is getting kind of messy if those users are not allowed to login, but you need to interact with the containers e.g. restart, exec -it something etc. |
Beta Was this translation helpful? Give feedback.
-
|
This also makes doing rootless quadlets in BootC a lot trickier as you have to create user directories and symlink them in /var using tmpfiles.d. It would also be nice to be able to assign capabilities such as CAP_DAC_READ_SEARCH. |
Beta Was this translation helpful? Give feedback.
-
|
I've seen How are the uids and gids mapped on container restarts? Will they be different each time? How does this impact volume directories that are mounted? Does this complicate any back ups that are taken of my mounted volume directories? |
Beta Was this translation helpful? Give feedback.
-
|
Your experiment, while interesting perhaps, doesn't seem to advance the objective of being able to use the |
Beta Was this translation helpful? Give feedback.
-
|
It's related as we're being told we don't need that feature because we can do UserNs=auto. I think it will be useful to lots of people reading through this thread as lots of people here are basically wanting containers running in the system instance of systemd but not running in the root user namespace. This brings you closer to that requirement just in a different way. Having said that I'll stop responding as I don't want to bicker any further. |
Beta Was this translation helpful? Give feedback.
-
|
I also agree that However, manually configuring Running each service/pod in its own user as a user-level systemd service also has its own problems. For example, images can't be shared between users, so it may use more storage, and more importantly for me, networks can't be shared between users (#20408). This makes running a reverse proxy in container as one user, and services as other users seems impossible, at least without exposing each service to the host's localhost, which also isn't a good solution, as then all services are directly exposed to anything other on the host, and it also makes proxying them more annoying, as you also have to rely on numerical ports and make sure that two services don't bind to the same port. With proper network sharing and per-container hostnames, this would be easier, as the reverse proxy could just point to the specific hostname of the service. Also, user-level systemd services make central monitoring harder. However, they allow for easiet access control to allow specific sysadmins to control only specific services. So, having a way to run a service/container/pod as another user (specified with username), or to somehow connect services running as user-level systemd services under different users, would be quite useful. |
Beta Was this translation helpful? Give feedback.
-
|
It appears dealing with Networking issues as expressed in #20408 also appear with:
Why is this not something that we can control reliably from outside a user's namespace slice from within the slice of the |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the direction @connelhooley, IMO this is the kind of thing that Github Discussions are for so I for one am glad you took the time to write this out. I have a few seemingly simple requirements on podman:
Your suggestions are the closest I feel I can get to my desired config as long as I take the loss of point 1, so thanks for the guidance |
Beta Was this translation helpful? Give feedback.
-
|
Inspired by NixOS, an option I'm considering here is taking Podman out of the loop. Systemd natively supports many directives to help with process isolation, chroot'ing and security features to restrict what processes can do. In some cases, the underlying mechanisms used are exactly what container tech would be doing anyway. If you don't need the packaging format of container images, this may be an option for you. When running and securing processes directly via systemd, using User= in systemd is not a problem. For complex cases this would not be an option. |
Beta Was this translation helpful? Give feedback.
-
|
Any News on this? |
Beta Was this translation helpful? Give feedback.
-
|
Thank you all for the replies guys :) |
Beta Was this translation helpful? Give feedback.
-
Precisely my experience as well. Maybe I've been spoiled by systemd and its general approach of seamless integration but having to create an arbitrary non-root user on a container host to then put it in linger mode is just odd... It makes podman <-> systemd integration feel like an afterthought - something glued-on post-factum. |
Beta Was this translation helpful? Give feedback.
-
Is it not possible to for example make the network owned by root or some specific user, and then let all other users in that user's group join the network?
How do you manage networks? Dou you just export ports from all services on localhost and then have a separate reverse proxy forward requests to the correct port? Is there any other way than exposing all ports directly? Also, do you have any monitoring/observability system for your services? Podman exporter currently only supports monitoring a specific Podman instance, so having services in separate users with lingering means that each service also needs its own Podman exporter instance, which seems annoying to set up and waste of resources. |
Beta Was this translation helpful? Give feedback.
-
|
it seems to me that the confusion here is about having the expectation that setting What is the expectation when the user doesn't even exist? Where will be storage be stored? Again, the biggest limitation without an user session is that we cannot use cgroups, with all the limitations caused by that. What you could try is to add a dependency to the systemd user session ( |
Beta Was this translation helpful? Give feedback.
-
Could it be made to work if the user already exists and has lingering enabled? It seems that was the original issue. |
Beta Was this translation helpful? Give feedback.
-
|
I have been able to get Not the prettiest, but a decent workaround for now. I do have the problems that for some of my containers podman refuses to show logs on stdout, but I can live with that for the moment. |
Beta Was this translation helpful? Give feedback.
-
|
how does it solve any of the problems that were reported in this discussion? Is the user session running for the |
Beta Was this translation helpful? Give feedback.
-
|
Since these units are generated by Ansible, I am able to hardcode absolute paths and avoid using problematic specifiers like For the second part: this is a system unit (not |
Beta Was this translation helpful? Give feedback.
-
|
The point is that without a systemd user session for that user, you are not able to use some features like cgroups. What happens if you try to set a memory limit for example? |
Beta Was this translation helpful? Give feedback.
-
|
I can give that a try, but my deployments fortunately do not need that feature. |
Beta Was this translation helpful? Give feedback.
-
The original issue is that Ideally, you could use any user for |
Beta Was this translation helpful? Give feedback.
and others
Uh oh!
There was an error while loading. Please reload this page.
-
Is this a BUG REPORT or FEATURE REQUEST?
/kind bug
Description
I want to have a systemd system service that runs a rootless container under an isolated user, but systemd rejects the sd_notify call and terminates the service.
A similar problem was menitoned but not resolved in #5572, which seems to have been closed without a resolution.
Happy to help tracking this down.
Steps to reproduce the issue:
podman generate systemd --new:Describe the results you received:
Describe the results you expected:
Nginx runs until the end of time.
Output of
podman version:Output of
podman info --debug:Package info (e.g. output of
apt list podman):Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)
Yes and yes.
Additional environment details (AWS, VirtualBox, physical, etc.):
Machine is a VM.
Beta Was this translation helpful? Give feedback.
All reactions