[BUG] bastille rc script should require SERVERS

[avg-I]

On one of my systems I noticed that (host) syslogd stopped logging any kernel messages.
I see them at console but they do not get into any logs files (including /var/log/messages).
My syslogd configuration is pretty standard, so that was a big puzzle for me.

Long story short, it turns out that (host) syslogd fails to open /dev/klog:

 11051 syslogd  NAMI  "/dev/klog" 
 11051 syslogd  RET   openat -1 errno 16 Device busy

Unfortunately, syslogd is pretty quiet about that, so it took a while to notice it.

The reason for EBUSY is that bastille gets started before syslogd, it starts some jails and apparently one of jailed syslogd-s grabs /dev/klog:

root     syslogd     1032    5 /usr/local/bastille/jails/minio/root/dev     17 crw-------    klog  r

However, having said that, I am not sure why /dev/klog is not hidden in the jail.
I checked that /usr/local/bastille/jails/minio/jail.conf has the default jail ruleset:

/usr/local/bastille/jails/minio/jail.conf:  devfs_ruleset = 4;

Maybe that's because the device entry is created with MAKEDEV_ETERNAL flag.
I see that all devices created with that flag appear in jails, even though they are not "unhidden" in any devfs rules, e.g. /dev/geom.ctl.

Bastille has been installed from the FreeBSD package repo (quarterly):
0.10.20231125
13.3-RELEASE-p2
13.3-RELEASE-p2
13.3-RELEASE-p2

[avg-I]

Hmm, in fact, I think that I see all devices from /dev in /usr/local/bastille/jails/minio/root/dev as well.
It's if the jail ruleset hasn't been applied.
But if I check it, everything seems to be correct:

# devfs -m /usr/local/bastille/jails/minio/root/dev rule show
100 include 1
200 include 2
300 include 3
400 path fuse unhide
500 path zfs unhide

# devfs -m /usr/local/bastille/jails/minio/root/dev rule -s 1 show
100 hide

# devfs -m /usr/local/bastille/jails/minio/root/dev rule -s 2 show
100 path null unhide
200 path zero unhide
300 path crypto unhide
400 path random unhide
500 path urandom unhide

# devfs -m /usr/local/bastille/jails/minio/root/dev rule -s 3 show
100 path ptyp* unhide
200 path ptyq* unhide
300 path ptyr* unhide
400 path ptys* unhide
500 path ptyP* unhide
600 path ptyQ* unhide
...

[avg-I]

Okay, if I replace the rule in ruleset 1 with

path * hide

then it seems to fix the problem.

Looks like it's a regression in FreeBSD?

[avg-I]

Well, that change should not make any difference, what made the difference was re-applying the ruleset.

So, I circled back to the original rc script problem: bastille is started before rc.d/devfs is run, so bastille does not see the rulesets.

# rcorder /etc/rc.d/* /usr/local/etc/rc.d/* | grep -E 'devfs|syslog|bastille|NETWORK|SERVERS'
/etc/rc.d/NETWORKING
/usr/local/etc/rc.d/bastille
/etc/rc.d/devfs
/etc/rc.d/newsyslog
/etc/rc.d/syslogd
/etc/rc.d/SERVERS

[avg-I]

Seems like a regression from 77afbd1

CC @yaazkal

[putneybsd]

/me too - I've seen this behaviour when rebooting a bastille host. In view of the statement in the bastille docs https://docs.bastillebsd.org/en/stable/chapters/jail-config.html

NOTE: It is important that only appropriate device nodes in devfs
be exposed to a jail; access to disk devices in the jail may
permit processes in the jail to bypass the jail sandboxing by
modifying files outside of the jail.

this does seem to be a real issue. I'm wondering if it would also be possible to add a warning to jail startup if the jail.conf specifies a devfs ruleset that can't be found?

[avg-I]

@cedwards, could you please take a look at this?

[dsh2dsh]

JFYI, as a workaround, on FreeBSD I created /usr/local/etc/rc.d/_bastille file with content:

#!/bin/sh
#
# PROVIDE: _bastille
# REQUIRE: jail
# BEFORE: bastille securelevel
# KEYWORD: shutdown

and it moved start of bastille after /etc/rc.d/jail.

[avg-I]

That's a very nice trick. Thank you.

[yaazkal]

Thank you @avg-I for the finding and detailed information.

I'm not sure if making it require SERVERS will be enough. I can see other services after it that someone can argue that should be required too due to their given use case.

Now, I can see /etc/rc.d/jail says it must run before securelevel and securelevel is the last one to run in a vanilla FreeBSD system, so basically after everything is running. That said, I think we should choose then simply require jail or if we want to keep the responsibility from our side:

REQUIRE: NETWORKING devfs
BEFORE: securelevel

which way is preferable? cc @cedwards

Mine is to require jail and then let the Bastille user sets whatever jail order wants to prioritize, i.e a DNS server jail to run first. My original commit was to solve that issue, but afaik the rcorder in Bastille came later on after that commit, so maybe the change has sense now.

[avg-I]

@yaazkal requiring jail sounds like the simplest and safest approach, just like @dsh2dsh suggested with his workaround.

[avg-I]

Thank you!

[ivomarino]

I confirm this also helps when using nullfs mounts of mount points from the host which are mounted via network (like s3fs).