Systemd unit without chroot #150
Conversation
…ice unit file in consideration. All props to Wouter Wijngaards for this work.
|
Thank you for the contribution, and the fixup with explanations. Was a good read and also the file can be useful to people that want that systemd configuration. Merged it and added a contrib/README and Changelog note. |
|
Thanks a lot, it was a pleasure 😀 |
|
@Frzk when I was fixing original systemd service I tried to make it work with and without chroot alongside being backward compatible for existing users. In fact, without fixes service worked more reliable without chroot enabled. I just tested myself that it works without chroot for me but if it doesn't for you then it's a bug and we should try to investigate and fix it instead of proliferating services with minimal diffs from each other which will be confusing for users. Unless I'm mistaken when Your BindPaths/BindReadOnlyPaths doesn't make sense as they are bind to the very same dir. Also they are pointless here as they are needed only for chroot. You may misunderstood the purpose of those rules. @wcawijngaards I recommend reverting this one and work collaboratively on fixing original service instead. |
|
Also all systemd hardenings are available in original systemd service as well but the description suggest they are exclusive for non-chroot version. |
|
@Frzk could you test if adding |
|
Sure, if possible, we can have one file that can do it with and without chroot. So that both types of use can be supported. |
|
@wcawijngaards Could you revert it then before someone start using it? As I said original service works right now for me with |
|
Hi there, First things first, I'd like to apologize if I hurt someone with this PR. As I stated in the original post (#149), these are changes I was suggesting (basically : let's discuss them). I originally made the changes against the existing file, but having another unit file seemed a good idea to me. Sounds like it wasn't.
That's a mistake of mine, they can be removed. (They are useful when running with
Can't see where I suggest something like this (?). That being said, I also made a bunch of tests all this morning, regardless of my use case (which is using I'm using ArchLinux, so All tests were done with the provided original unit file. For some tests I used a drop-in to add a few directives, keeping the unit file untouched. Here is what I'm experiencing (tried to be synthetic): 1. with chroot (
|
systemctl start |
systemctl stop |
Result |
|---|---|---|
| Creates /run/unbound.pid | Leaves /run/unbound.pid | Not OK |
| Creates /etc/unbound/dev | Leaves /etc/unbound/dev | ~OK |
| Creates /etc/unbound/run | Leaves /etc/unbound/run | ~OK |
Results are the same when specifying pidfile: /run/unbound.pid in unbound configuration file.
1.b Setting pidfile: /run/unbound/unbound.pid:
systemctl start |
systemctl stop |
Result |
|---|---|---|
| Fails to create /run/unbound/unbound.pid | Leaves /run/unbound | Not OK |
| Creates /etc/unbound/dev | Leaves /etc/unbound/dev | ~OK |
| Creates /etc/unbound/run | Leaves /etc/unbound/run | ~OK |
1.c Base config file and adding RuntimeDirectory=unbound in a drop-in:
systemctl start |
systemctl stop |
Result |
|---|---|---|
| Creates an empty /run/unbound directory | Removes /run/unbound directory | OK |
| Writes pid file in /run/unbound.pid | Leaves /run/unbound.pid | Not OK |
| Creates /etc/unbound/dev | Leaves /etc/unbound/dev | ~OK |
| Creates /etc/unbound/run | Leaves /etc/unbound/run | ~OK |
1.d Setting pidfile: /run/unbound/unbound.pid and adding RuntimeDirectory=unbound in a drop-in:
systemctl start |
systemctl stop |
Result |
|---|---|---|
| Creates /run/unbound/unbound.pid | Removes /run/unbound | OK |
| Creates /etc/unbound/dev | Leaves /etc/unbound/dev | ~OK |
| Creates /etc/unbound/run | Leaves /etc/unbound/run | ~OK |
We are finally OK.
2. without chroot (chroot: "")
2.a Setting chroot: "":
systemctl start |
systemctl stop |
Result |
|---|---|---|
| Creates /run/unbound.pid | Leaves /run/unbound.pid | Not OK |
| Creates /etc/unbound/dev | Leaves /etc/unbound/dev | ~OK |
| Creates /etc/unbound/run | Leaves /etc/unbound/run | ~OK |
2.b Setting chroot: "" and pidfile: /run/unbound/unbound.pid:
systemctl start |
systemctl stop |
Result |
|---|---|---|
| Fails unless /run/unbound is created first | - | ~OK |
| Creates /run/unbound/unbound.pid | Leaves /run/unbound/unbound.pid | Not OK |
| Creates /etc/unbound/dev | Leaves /etc/unbound/dev | ~OK |
| Creates /etc/unbound/run | Leaves /etc/unbound/run | ~OK |
2.c Setting chroot: "", pidfile: /run/unbound/unbound.pid and adding RuntimeDirectory=unbound in a drop-in:
systemctl start |
systemctl stop |
Result |
|---|---|---|
| Creates /run/unbound/unbound.pid | Removes /run/unbound | OK |
| Creates /etc/unbound/dev | Leaves /etc/unbound/dev | ~OK |
| Creates /etc/unbound/run | Leaves /etc/unbound/run | ~OK |
We are OK here.
As you can see, it seems to me that we are pretty far from working out of the box.
I guess the results could be improved if unbound was compiled with --with-pidfile=/run/unbound/unbound.pid because it would mean we'd just have to set RuntimeDirectory=unbound to get something working in these cases.
But again, I'm not speaking about my use case. I'll try to provide further inputs about it later, I've already spent 5 or 6 hours on this, way more than I can afford.
No problem, you raised a valid issue. I only wish that proposed fix wasn't merged so fast before any discussion could happen 😄 . Anyway I hope we can agree about finding better solution.
Hm, I don't see how binding them to the same dir would help in any case.
You wrote that:
That suggests chroot and systemd hardening are alternatives despite they can be used together just fine. Also same systemd hardening rules are elaborately described only in new unit while those are exactly same rules like those in other service. I think we can just put those descriptions to main service.
Same as me so finding common ground should be easy 😄
Your testing shows that situation is exactly the same with and without Some minor issue happens when The only real issue happens when I hope above analysis make situation clearer.
I spent many hours within several weeks in order to make this service usable for official Arch package and I'm open to help with improving it further. I'll prepare changes with adding |
|
@Maryse47 Constantly deleting/reposting your comments makes it very difficult to answer with context and precisions 😕 I'll try to answer everything here later this evening with further explanations about my use cases. |
Sorry, I'm continuously improving my answers in quest for perfection (not deleting but editing). I want be be as specific and to the point as I can to minimize misunderstandings. I didn't know you read them already. |
This is basically why they exists : making a part of the filesystem tree available in a mount namespace... Why would I want them somewhere else ?
I think you are over-interpreting here :sorry: Also, (real, serious question), what's the benefit of the chroot in this case ?
Please do.
I don't know if it was appropriate or not, but at least it was working (pid removed), filesystem kept clean, working with portable services.
Having a leftover .pid is far from being a minor issue IMHO : I'm basically relying on them to monitor my services... And it also leaves a dev and run directories in my config directory...
My unit file fixes this at the cost of specifying
So is it severe or not ?
Nope, that's not the "only real issue". Please see below.
I'm really sorry, but it doesn't. Let's get back to testing, this time trying to make unbound a portable service. As root, without modifying the provided unit file: cd ~
mkdir ./unbound_test
pacstrap -i ./unbound_test/ base unbound
mksquashfs ./unbound_test/ /var/lib/portables/unbound_test.raw
portablectl attach /var/lib/portables/unbound_test.raw
systemctl start unboundFails with: unbound.service: Failed to set up mount namespacing: /run/systemd/unit-root/etc/unbound/dev: No such file or directory
unbound.service: Failed at step NAMESPACE spawning /usr/bin/unbound: No such file or directoryWhich is expected, portablectl detach /var/lib/portables/unbound_test.raw
rm /var/lib/portables/unbound_test.raw
mkdir -p ./unbound_test/etc/unbound/{dev,run}
mksquashfs ./unbound_test/ /var/lib/portables/unbound_test.raw
portablectl attach /var/lib/portables/unbound_test.raw
systemctl start unboundIt now fails with the following: unbound.service: Failed to set up mount namespacing: /run/systemd/unit-root/run/dbus/system_bus_socket: No such file or directory
unbound.service: Failed at step NAMESPACE spawning /usr/bin/unbound: No such file or directory(Which is quite logic too since How am I supposed to proceed thereafter ? Now if I edit the provided unit file, according to the comment in it (still with the chroot), it works ! Results are the same with In fact to make it work I'd have to create a drop-in with a I suspect using it with I understand that keeping backward compatibility is important, but here, IMHO, we are keeping it for no reason (what is the benefit ?) at the cost of a complex unit file which prevents legitimate usage. Again, my proposal was about providing a unit file that is simple, yet providing enough security and allowing for other use cases like the one presented here without all the hassle of understanding what the unit file does (and yes, maybe breaking compatibility with people running with Now please @wcawijngaards could you please merge #151 to end all this non sense ? |
Ok, for
Most systemd hardenings make various path read-only while chroot make them invisible. That's why we have to hack around some things to make them visible again like
Yes, but you fixed not the problem that you described. Description says "This unit file is provided to run unbound without chroot." This is misleading because chroot works just fine with original unit. It may not work with portable services so if you described it as "This unit file is provided to run unbound as portable service." instead it would make more sense.
The point is if I specify
It is a bad practice but it is what many distros and users do and I wanted this unit be usable for as much people as possible.
IMHO it's not. By core I meant core unbound code in opposition to contrib stuff like systemd units.
Unbound leaves those pids all the time in most configurations and I didn't saw complaints here or in Arch so I think vast majority of people don't care. Is leaving dev and run in config directory another serious issue for you? They are needed for chroot to work but I don't see how they're causing problems.
The only real issue your first tests showed. I thought they were complete.
Wait, what still doesn't work with first tests you presented here? The pidfiles are left? Please open separate issue about that. Portable services are different case and I didn't said it will work with them. I think we shouldn't jump to the next case before we can settle on the previous one. Saying "it doesn't work" isn't helpful.
It's possible that chroot workarounds are incompatible with portable services. It's quite possible if you put portable services instead of
Do you mean
Do you know how many users reported broken chroot in unbound when upstream unit was used for the first time in official package? Enabled
Again, your proposal didn't provide any meaningful new security and was totally orthogonal to chroot. As I understand now it was all about portable services from the beginning but that fact was lost while your prepared final PR. Your unit doesn't bring any meaningful value for users who don't use portable services and it's not something that distros want to package. If unbound maintainers think that additional portable services version is valuable then it's ok but please name and describe things accordingly. |
@Frzk Please take a look at modifications I did for #151 . I kept your unit but renamed it and changed its description to be more appropriate. I hope this is solution we can finally agree about. |
So it confirms what I suspected : no benefit.
Quoting the very first lines in #149:
How doesn't it fix the problem I described ?
That's not misleading at all, that's what it does:
As you just said, you would need it for
The purpose is to bring an alternative. Then by modifying the compile options (store pidfile in /run/unbound/unbound.pid instead of /run/unbound.pid + remove the chroot) we would have better defaults IMHO (easy to understand unit file, filesystem tree kept clean, runs out of the box, doesn't break systemd functionalities).
Again I understand this. That's why the unit was proposed as an alternative. Not replacing yours
That may sound harsh but I don't really care if other users don't mind running bad practice.
I just demonstrated that it doesn't work with a complete set of reproducible instructions and explanations. So telling me that "saying «it doesn't work» isn't helpful" just sounds like trolling to me.
Demonstrated before.
Yes, instead we would have a discussion about the leftover pid file, then a discussion about the leftovers run and dev directories (even when you don't use chroot), and then a discussion about portable services (and maybe one about systemd-nspawn). And we'd finally end up with 2 unit files: one that cares with backward compatibility and another one that follows nowadays best practices. Which is where we are now. That'd be really productive.
Reality is the provided unit file only works if:
Whereas mine (+ some config or modified build options) was given as an alternative to circumvent all this. So yes, I don't see the point of putting "portable service" everywhere in it.
I'll let you do the tests 😀
Actually I'm not. It's provided as an alternative. And yes, I'll get in touch with Gaetan to discuss the build options for ArchLinux.
Never was the goal. Never stated it does.
Sure. That's why it's suffixed nochroot. How should have we made it clearer ?!
This is just how I discovered things. I would probably have noticed later when setting up monitoring (because of the leftover pid file). So, nope, it wasn't all about portable services.
At least it fixes the missing
I don't know, I'm (thankfully !) not in packagers minds.
IMO that's what we've done here:
|
Making various paths not accessible instead of read-only has no benefit?
Did you seriously think user will dig into a comment for old PR which was rejected in order to know context about shipped code? Merged PR has absolutely no such context.
The bad practice comes from unbound itself perhaps for some historical reasons. I don't think the purpose of systemd unit is to mandate how unbound works. Arch (and possibly nobody else) wouldn't adopt upstream systemd unit if it broke existing users and everyone would be left with basic unit with no hardening shipped by ditros.
I explained that your issue from your first set of instructions could fixed by
You demonstrated it in the same post.
I already adviced to open issue about leftover pids. Unbound never removes those even while using your unit, it's just systemd which removes whole directory. You may easily check this with
Original service is chroot agnostic - it works the same regardless if chroot is enabled or not. If you know how to make chroot working without creating those files then just show it, otherwise there is nothing to discuss.
Isn't there agreement about those? I don't see anything to discuss and that was my point - if you didn't stubbornly repeated claims about not working chroot (which is false) and instead focus on portable services then agreement could be reached immediately.
I'll much prefer that we end up with one unit working 99% of time that may be packaged by distros and one for special case like portable services which may be used for experimentation. Producing separate units with subtle differences with misleading descriptions is counterproductive as nobody will use more than one.
As said above this is unbound issue., not sytemd unit. It can be solved with
Question for @wcawijngaards do you think
As I said those are needed for chroot to work. Unbound uses chroot by default. Breaking default configuration in systemd unit sounds awkward to me.
Do you mean portable services? They are incompatible with chroot, but see above.
Perhaps unbound developers would want to hear more about it.
With exactly same some config or modified build options the original service works as well as yours, including all nochroot cases except for portable services. This is fundamental issue with your unit and what make it pointless for all users who don't use portable services. This is also the message I'm trying to pass to you without much success.
I'm sorry but I spent so many hours on this issue and won't sacrifice more only because you refrain from providing simple answer about things you already know.
Good luck. Improving Arch package will be certainly beneficial.
Sorry, I was just annoyed by your mentions of security so many times. Perhaps I should ignore it.
The nochroot suffix suggest that original unit is incompatible with no-chroot which is utterly false. That makes its purpose not clear. The portable services are incompatible with chroot which is the reason you have to disable chroot for them. That's why portable suffix will be more appropriate and more to the point.
Without portable services not single point of yours stands.
Could you at least say if my corrections are acceptable for you? You already agreed for complete removal so I hope keeping your unit intact with just description change will be even more welcomed.
It makes little sense as original service work without chroot just fine. It's much easier for users to modify something with overrides than grabbing whole new unit (obviously distro can package only one).
From packager perspective, breaking backward compatibility would be quite troublesome. Original unit allows for best practices but not enforce them which is more sensible for general audience and involves no risk and almost no downsides. |
|
Hi Maryse47 and Frzk, Sure a rename of _nochroot to _portable sounds fine to me. So #151 looks reasonable. About chroot, chroot predates systemd. And it is there to protect the system. This is why I try to enable this to encourage safer configurations. If systemd can provide protection just as good, that is fine with me, just like SELinux might. And some users find that too hard, because chroot makes things difficult, and there is the chroot: "" option, (that I also use in testing). Chroot is valuable, because it protects the system against a compromised server. About pidfile removal. Unbound actually attempts to remove the pidfile. It needs permissions, on the file and directory (to remove a file from that directory). For that the file and directory need to be inside the chroot path, otherwise the pidfile has disappeared from unbound after the chroot, eg. unbound locked itself out of accessing the pidfile with the chroot security call. Unbound also attempts to truncate the file (to 0 bytes) and then delete it, and also attempts to chown the pidfile (to config username). Perhaps you could put the pidfile location inside the chroot, or mount the pidfile directory inside the chroot eg. bindmount /etc/unbound/run/unbound, (for /etc/unbound as chroot and /run/unbound as pidfile dir). Thanks for the work in figuring out the best possible systemd options! There seems to be a lot of possibilities. And use cases for the unbound server. I think right now merging #151 allows both the original case of #149 (portable use), and we have the normal file for regular, chroot or nonchroot use. I believe a lot of packaging systems ship their own systemd service file, anyway, as people want to store stuff like pidfiles in different places. @Maryse47, you asked if unbound needs write permissions. Unbound needs write permissions for two reasons, the pidfile removal and trust anchor updates. It depends on where you keep those files, and then unbound needs write permission on that. If chroot is used, those locations have to be inside the chroot. Unbound exits if the trustanchor cannot be written to, if that trustanchor is configured with auto updates, making sure that RFC5011 root trust anchor rollover works. Best regards, Wouter |
|
FYI: http://0pointer.de/blog/projects/changing-roots
Sound like a good candidate for the |
|
I just realised that RFC7706 root zone copies also can have a file, with the zonefile contents, that has to be written to. This is for auth-zone directives with a zonefile configured. (It can also be without zonefile; it then transfers every time, eg. zonefile: ""). |
For some reason chowning the pidfile doesn't seem to work: and this may be the problem - unbound lacks of permissions to remove the pidfile. It's possible to use
We did this before but it was causing issues for some people and was dropped. Removing pidfile didn't work anyway for above reasons.
Ok, so this is what i expected. We have to keep those dirs writable in generic config then
Using state dir under |
Thanks, I'm already doing this along with
Doesn't sound something really difficult to add a
Thanks, I'm already doing it in my unit file. It works well. |
|
@Frzk do you think we can add
Adding
I think we can add it to both generic & portable unit. |
|
About the pidfile, the chown error is shown if you increase verbosity to 3 or higher. Right at the start. If you use log to syslog, it should be written to syslog. Otherwise, use Perhaps the systemcallfilter stops chown. Or a bindpaths to make the pidfile appear in the chroot, but for that chowning the pidfile has to work too. Looking at the source, I see unbound only chowns, if the username is set in unbound config, (i.e. not disabled with You could cheat on it with But maybe this does not work, in which case I do not want to waste your time on it. |
|
So I can see |
|
You could try starting unbound from the commandline with |
|
Starting |
|
Could it be that the /run dir group permission of 5 (readonly) is affecting this? I.e. not user root, but only group? If /run/unbound.pid is owned by the same user that started it, then I would expect it to work just like manually. |
|
if you manually chown with the numbers from the error message |
|
Does unbound really make its file somewhere else, eg. it exists in some other directory, and the file in /run that you look at is an old file? (This happens to FreeBSD people a lot, because unbound is shipped in packages to /usr/local/etc/unbound and there is also a /etc/unbound in some cases). Also because of Debian's alternatives systems for systems like Ubuntu too, the files are somewhere else. |
If unbound runs as root user then group access shouldn't affect it. This is really weird.
When I chown this file with sudo it works without issues.
I removed /run/unbound.pid if it existed. Starting unbound creates /run/unbound.pid but chmod fails. |
|
@wcawijngaards ah, sorry. I had apparmor profile for unbound with |
|
Agah, I was already typing a reply, but apparmor could be it! |
|
If it can chown, I suspect it can also unlink later on (with ReadWrite Paths and BindPaths and so on configuration adjusted for it). Because it chowns after checking the chroot pathname and capabilities are sortof similar. (Unless there is a separate unlink(2) apparmor rule (that is the file delete call it uses to delete the pidfile when unbound stops)). |
|
It still doesn't chown itwhen started from systemd so I will investigate it further. |
|
How are you running it with systemd ? |
|
@wcawijngaards @Frzk so obviously Now, after stop pidfile is getting truncated but not removed not sure why but the same happens with Above still doesn't work when chroot is being used but making pidfile available in chroot without breaking someones config seems rather impossible. |
|
systemd doesn't need a pidfile. Just invoke unbound with the If you need a pidfile for some reason, use the systemd.service https://www.freedesktop.org/software/systemd/man/systemd.service.html
The The set of capabilities granted in In another DNS server, its usage of capabilities was re-designed recently so that it could be started unprivileged and with minimal capabilities, and then all capabilities dropped after binding to port 53. See https://gitlab.labs.nic.cz/knot/knot-dns/issues/546#note_57098 and https://gitlab.labs.nic.cz/knot/knot-dns/merge_requests/864 for details. |
That sounds like good idea. We can add
The problem with that is pidfile path is configurable by user and may vary across distros. Dropping pidfile completely sounds better then.
Yes, I already dropped it in my PR.
CAP_NET_RAW: needed for IP_TRANSPARENT socket option. CAP_IPC_LOCK: I don't know. @wcawijngaards do you think unbound uses it for something or should we drop it? CAP_SETGID, CAP_SETUID: as I pointed out in #150 (comment) running unbound completely rootless from the start need existing systems configured for that and I doubt they are. |
Not sure to follow here, but systemd will not create the pidfile (it's made clear in the quoted doc). It will only make sure it's removed:
|
|
CAP_IPC_LOCK: needed for mmap, shmctl and unbound can use that for statistics reporting. With No pidfile: yes we have that feature for people that want to have no pidfile. It does not do anything else. |
|
@wcawijngaards should I add |
If I understand correctly, |
|
Yes, I think you are correct, and unbound does not need it. |
|
@wcawijngaards I added |
Replaces #149.