Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ruleset exists but iocage does not find it #952

Closed
dlangille opened this issue Jun 17, 2019 · 51 comments
Closed

Ruleset exists but iocage does not find it #952

dlangille opened this issue Jun 17, 2019 · 51 comments

Comments

@dlangille
Copy link
Contributor

This worked before upgrading iocage today and upgrading the jail to 12.0-RELEASE-p5

Make sure to follow and check these boxes before submitting an issue! Thank you.

$ iocage --version
Version	1.2 RC

The above is head from 2019-06-16:

$ pkg info -x iocage
py36-iocage-devel-1.0.0.20190616,1

Today I upgraded iocage from 1.0.0.20190519_1,1

When the jail is started, the rule set is not invoked/found:

[dan@knew:~] $ date
Mon Jun 17 01:17:55 UTC 2019
[dan@knew:~] $ sudo iocage start toiler
* Ruleset 7 does not exist, using defaults
* Starting toiler
  + Started OK
  + Using devfs_ruleset: 7
  + Using IP options: ip4.addr=10.55.0.13,127.1.0.53 ip6.addr=2001:470:8abf:[redacted] ip4.saddrsel=1 ip4=new ip6.saddrsel=1 ip6=new
  + Starting services OK
  + Executing poststart OK


[dan@knew:~] $ tail /var/log/iocage.log 
2019/06/17 01:17:01 (INFO)   + Removing devfs_ruleset: 7 OK
2019/06/17 01:17:01 (INFO)   + Removing jail process OK
2019/06/17 01:17:01 (INFO)   + Executing poststop OK
2019/06/17 01:18:05 (INFO) * Ruleset 7 does not exist, using defaults
2019/06/17 01:18:05 (INFO) * Starting toiler
2019/06/17 01:18:05 (INFO)   + Started OK
2019/06/17 01:18:05 (INFO)   + Using devfs_ruleset: 7
2019/06/17 01:18:05 (INFO)   + Using IP options: ip4.addr=10.55.0.13,127.1.0.53 ip6.addr=2001:470:8abf:[redacted] ip4.saddrsel=1 ip4=new ip6.saddrsel=1 ip6=new
2019/06/17 01:18:08 (INFO)   + Starting services OK
2019/06/17 01:18:08 (INFO)   + Executing poststart OK
[dan@knew:~] $ 

NOTE the 'does not exist' in the log extra above.

Within /etc/devfs.rules we have

# for dhcpd in jail toiler
#
[devfsrules_jail_bpf=7]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'bpf*' unhide

# added based on https://forums.freebsd.org/threads/jailed-dhcp-server-and-tftp-handoff.29934/
#add path net unhide
#add path 'net/*' unhide

# Support for TUN devices
#

At present, dhcpd in the jail cannot be started because:

Jun 17 01:18:07 toiler dhcpd[61983]: No bpf devices. Please read the README section for your operating system.

@dlangille
Copy link
Contributor Author

For fun, I did this:

[dan@knew:~] $ sudo iocage set devfs_ruleset=10 toiler
devfs_ruleset: 7 -> 10

Ruleset 10 does not exist in /etc/devfs.rules

Restarting the jail results and there is no 'does not exist' exist message, despite the fact that the ruleset does not in fact exist.

2019/06/17 13:50:16 (INFO) devfs_ruleset: 7 -> 10
2019/06/17 13:50:26 (INFO) * Stopping toiler
2019/06/17 13:50:26 (INFO)   + Executing prestop OK
2019/06/17 13:50:28 (INFO)   + Stopping services OK
2019/06/17 13:50:28 (INFO)   + Removing devfs_ruleset: 7 OK
2019/06/17 13:50:28 (INFO)   + Removing jail process OK
2019/06/17 13:50:28 (INFO)   + Executing poststop OK
2019/06/17 13:50:29 (INFO) * Starting toiler
2019/06/17 13:50:29 (INFO)   + Started OK
2019/06/17 13:50:29 (INFO)   + Using devfs_ruleset: 10
2019/06/17 13:50:29 (INFO)   + Using IP options: ip4.addr=10.55.0.13,127.1.0.53 ip6.addr=2001:470:8abf:[redacted] ip4.saddrsel=1 ip4=new ip6.saddrsel=1 ip6=new
2019/06/17 13:50:30 (INFO)   + Starting services OK
2019/06/17 13:50:30 (INFO)   + Executing poststart OK

Next, let us renumber the ruleset in the file:

[dan@knew:~] $ diff -ruN /etc/devfs.rules~ /etc/devfs.rules
--- /etc/devfs.rules~	2019-06-17 13:49:17.000000000 +0000
+++ /etc/devfs.rules	2019-06-17 13:50:57.257191000 +0000
@@ -33,7 +33,7 @@
 
 # for dhcpd in jail toiler
 #
-[devfsrules_jail_bpf=7]
+[devfsrules_jail_bpf=10]
 add include $devfsrules_hide_all
 add include $devfsrules_unhide_basic
 add include $devfsrules_unhide_login
[dan@knew:~] $ 

Restarting the jail, and again, no message about 'does not exist', which is correct this time.

2019/06/17 13:50:50 (INFO) * Stopping toiler
2019/06/17 13:50:50 (INFO) + Executing prestop OK
2019/06/17 13:50:52 (INFO) + Stopping services OK
2019/06/17 13:50:52 (INFO) + Removing devfs_ruleset: 10 OK
2019/06/17 13:50:52 (INFO) + Removing jail process OK
2019/06/17 13:50:52 (INFO) + Executing poststop OK
2019/06/17 13:51:04 (INFO) * Starting toiler
2019/06/17 13:51:04 (INFO) + Started OK
2019/06/17 13:51:04 (INFO) + Using devfs_ruleset: 10
2019/06/17 13:51:04 (INFO) + Using IP options: ip4.addr=10.55.0.13,127.1.0.53 ip6.addr=2001:470:8abf:[redacted] ip4.saddrsel=1 ip4=new ip6.saddrsel=1 ip6=new
2019/06/17 13:51:05 (INFO) + Starting services OK
2019/06/17 13:51:05 (INFO) + Executing poststart OK

Now dhcpd can start

My questions:

  • Why did this break?
  • Why does 7, which existed, not get detected?
  • Why does 10, which did not exist, get 'found'?
  • Why does 10, when it exists, get found but 7 did not?

@skarekrow
Copy link
Member

I recommend leaving that on the default 4 with dhcp on, as iocage will handle the devfs rulesets for you and unhide the same devices.

@dlangille
Copy link
Contributor Author

From man poudriere:

     dhcp=[1 | 0]  This controls starting the jail with the Dynamic Host
                   Configuration Protocol enabled.  To enable dhcp, vnet and
                   bpf must also be enabled.

                   Default: 0

                   Source: local

Do we both understand this the same way?

This jail has a static IP address.

dhcpd runs inside this jail.

I think the iocage dhcp configuration setting is not related to this use case.

@skarekrow
Copy link
Member

In which case just set the bpf property and vnet. Don't set dhcp. iocage will handle the rulesets for you as long as the property is at the default (4)

@dlangille
Copy link
Contributor Author

Here is something interesting:

$ iocage get all toiler | grep set
allow_set_hostname:1
cpuset:off
devfs_ruleset:10
jail_zfs_dataset:iocage/jails/toiler/data


$ sudo iocage set devfs_ruleset=4 toiler
devfs_ruleset: 10 -> 4


$ sudo iocage set devfs_ruleset=10 toiler
devfs_ruleset: 10 -> 10

Wait... I changed it to 4, then back to 10, and it's already 10?

Let's try it again:

[dan@knew:~] $ sudo iocage set devfs_ruleset=10 toiler
devfs_ruleset: 10 -> 10
[dan@knew:~] $ iocage get devfs_ruleset toiler
10
[dan@knew:~] $ sudo iocage set devfs_ruleset=4 toilerler
devfs_ruleset: 10 -> 4
[dan@knew:~] $ iocage get devfs_ruleset toiler
10
[dan@knew:~] $ 

OK, let's look in there again:

[dan@knew:~] $ cd /iocage/jails/toiler/
[dan@knew:/iocage/jails/toiler] $ ls -lt
total 29
-rw-r--r--   1 root  wheel  3493 Jun 24 18:16 config.json
drwxr-xr-x  19 root  wheel    24 Mar 18 17:08 root
-rw-r--r--   1 root  wheel     0 Dec 21  2018 fstab
[dan@knew:/iocage/jails/toiler] $ date
Mon Jun 24 18:17:50 UTC 2019
[dan@knew:/iocage/jails/toiler] $ grep devfs_ruleset config.json 
    "devfs_ruleset": "4",
[dan@knew:/iocage/jails/toiler] $ 

@dlangille
Copy link
Contributor Author

Trying that again, it seems to work. WTF is going on?

[dan@knew:/iocage/jails/toiler] $ grep bpf config.json 
    "bpf": 0,
[dan@knew:/iocage/jails/toiler] $ sudo iocage set bpf=1 toiler
bpf: 0 -> 1
[dan@knew:/iocage/jails/toiler] $ grep bpf config.json 
    "bpf": 1,
[dan@knew:/iocage/jails/toiler] $ sudo iocage set devfs_ruleset=4 toiler
devfs_ruleset: 10 -> 4
[dan@knew:/iocage/jails/toiler] $ grep devfs_ruleset config.json 
    "devfs_ruleset": "4",
[dan@knew:/iocage/jails/toiler] $ 

@skarekrow
Copy link
Member

That's because iocage get will return the dynamic ruleset, you know this as you commented in the issue (#694) :P

@skarekrow
Copy link
Member

TLDR; Set bpf and vnet to 1, set devfs_ruleset to 4 and enjoy it working as it should.

@dlangille
Copy link
Contributor Author

You attribute much to me but I have no memory of that one.

@skarekrow
Copy link
Member

;) Well it did indeed happen lol

@dlangille
Copy link
Contributor Author

Yeah, but expecting people to recall obscure bits is obscure.

@dlangille
Copy link
Contributor Author

I'm off to configure vnet for this dhcp jail.

@dlangille
Copy link
Contributor Author

"VNET is considered experimental. Unexpected system crashes can occur. " - https://iocage.readthedocs.io/en/latest/networking.html

Wait, what? I must now use this?

Is there another way?

@skarekrow
Copy link
Member

That's for earlier RELEASEs, it's stabilized significantly since then. To pass the bpf device in, vnet AFAIK is required.

@dlangille
Copy link
Contributor Author

This jail has used bpf since it was created. The requirement to use vnet must be new.

@dlangille
Copy link
Contributor Author

dlangille commented Jun 24, 2019

The interesting part here:

$ sudo iocage start toiler
* Starting toiler
  + Started OK
  + Using devfs_ruleset: 25
  + Configuring VNET FAILED
  route: writing to routing socket: Network is unreachable
add net default: gateway 10.55.0.1 fib 0: Network is unreachable

Stopped toiler due to VNET failure
                                                                               
Broadcast Message from root@knew.int.unixathome.org                            
        (no tty) at 18:44 UTC...                                               
                                                                               
Communications with UPS lost.                                                                               
                                                                               
Broadcast Message from root@knew.int.unixathome.org                            
        (no tty) at 18:44 UTC...                                               
                                                                               
Communciations with UPS restored.                                                                               

The way it takes down ix0:

Jun 24 18:44:29 knew kernel: epair0a: Ethernet address: 02:8a:bd:d4:9d:0a
Jun 24 18:44:29 knew kernel: epair0b: Ethernet address: 02:8a:bd:d4:9d:0b
Jun 24 18:44:29 knew kernel: epair0a: link state changed to UP
Jun 24 18:44:29 knew kernel: epair0b: link state changed to UP
Jun 24 18:44:29 knew kernel: epair0a: changing name to 'vnet0.20'
Jun 24 18:44:29 knew kernel: ix0: link state changed to DOWN
Jun 24 18:44:30 knew kernel: vnet0.20: promiscuous mode enabled
Jun 24 18:44:30 knew kernel: ix0: link state changed to UP
Jun 24 18:44:30 knew kernel: arp: 02:ff:60:23:ee:f1 is using my IP address 10.55.0.13 on epair0b!
Jun 24 18:44:30 knew kernel: vnet0.20: link state changed to DOWN
Jun 24 18:44:30 knew kernel: epair0b: link state changed to DOWN
Jun 24 18:44:30 knew kernel: ix0: link state changed to DOWN
Jun 24 18:44:30 knew apcupsd[1954]: Communications with UPS lost.
Jun 24 18:44:30 knew kernel: ix0: link state changed to UP
Jun 24 18:44:35 knew apcupsd[1954]: Communications with UPS restored.

I'm guessing that is because 10.55.0.13 is static on ix0 and must now be dynamic. OK, I can do that.

[dan@knew:/iocage/jails/toiler] $ sudo ifconfig ix0 10.55.0.13 delete

Add the IP addresses back in:

$ sudo iocage set ip4_addr="vnet0|10.55.0.13,127.1.0.53" toiler
ip4_addr: 10.55.0.13,127.1.0.53 -> vnet0|10.55.0.13,127.1.0.53

But still:

$ sudo iocage start toiler
* Starting toiler
  + Started OK
  + Using devfs_ruleset: 25
  + Configuring VNET FAILED
  route: writing to routing socket: Network is unreachable
add net default: gateway 10.55.0.1 fib 0: Network is unreachable

Stopped toiler due to VNET failure
Jun 24 18:53:28 knew kernel: epair0a: Ethernet address: 02:46:dd:b8:45:0a
Jun 24 18:53:28 knew kernel: epair0b: Ethernet address: 02:46:dd:b8:45:0b
Jun 24 18:53:28 knew kernel: epair0a: link state changed to UP
Jun 24 18:53:28 knew kernel: epair0b: link state changed to UP
Jun 24 18:53:28 knew kernel: epair0a: changing name to 'vnet0.21'
Jun 24 18:53:28 knew kernel: ix0: link state changed to DOWN
Jun 24 18:53:28 knew kernel: vnet0.21: promiscuous mode enabled
Jun 24 18:53:28 knew kernel: ix0: link state changed to UP
Jun 24 18:53:29 knew kernel: vnet0.21: link state changed to DOWN
Jun 24 18:53:29 knew kernel: epair0b: link state changed to DOWN
Jun 24 18:53:29 knew kernel: ix0: link state changed to DOWN
Jun 24 18:53:29 knew kernel: ix0: link state changed to UP
Jun 24 18:57:51 knew kernel: epair0a: Ethernet address: 02:3f:43:00:8c:0a
Jun 24 18:57:51 knew kernel: epair0b: Ethernet address: 02:3f:43:00:8c:0b
Jun 24 18:57:51 knew kernel: epair0a: link state changed to UP
Jun 24 18:57:51 knew kernel: epair0b: link state changed to UP
Jun 24 18:57:51 knew kernel: epair0a: changing name to 'vnet0.22'
Jun 24 18:57:51 knew kernel: ix0: link state changed to DOWN
Jun 24 18:57:51 knew kernel: vnet0.22: promiscuous mode enabled
Jun 24 18:57:51 knew kernel: ix0: link state changed to UP
Jun 24 18:57:52 knew kernel: vnet0.22: link state changed to DOWN
Jun 24 18:57:52 knew kernel: epair0b: link state changed to DOWN
Jun 24 18:57:52 knew kernel: ix0: link state changed to DOWN
Jun 24 18:57:52 knew kernel: ix0: link state changed to UP

I am not sure why this fails at all.

@skarekrow
Copy link
Member

Do you have defaultrouter set?

@dlangille
Copy link
Contributor Author

Yes, it is set to my gateway.

$ iocage get defaultrouter toiler
10.55.0.1

@dlangille
Copy link
Contributor Author

I have no idea what this should look like:

$ ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:ca:b7:29:ce:00
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 1 priority 128 path cost 2000
	groups: bridge 
	nd6 options=1<PERFORMNUD>

@dlangille
Copy link
Contributor Author

I am reading https://iocage.readthedocs.io/en/latest/networking.html

The examples under VIMAGE/VNET have references to what is required in configuration files but no mention of how to invoke that configuration item from the command line.

bridge0 already existed when I got here. That is likely related to:

cloned_interfaces="lo1"

Let's try what I found at https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-bridging.html

$ sudo ifconfig bridge create
bridge1

$ ifconfig bridge1
bridge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:ca:b7:29:ce:01
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
	groups: bridge 
	nd6 options=1<PERFORMNUD>

$ sudo ifconfig bridge1 addm ix0
ifconfig: BRDGADD ix0: Device busy

Oh, I have to take ix0 down for this?

@skarekrow
Copy link
Member

skarekrow commented Jun 24, 2019 via email

@dlangille
Copy link
Contributor Author

Oh. Is the following required?

# set up bridge interface for iocage
cloned_interfaces="bridge0"

# plumb interface em0 into bridge0
ifconfig_bridge0="addm em0 up"
ifconfig_em0="up"

Where em0 is my main interface? ix0 in my case. I ask because that's what the documentation says.

@skarekrow
Copy link
Member

skarekrow commented Jun 24, 2019 via email

@dlangille
Copy link
Contributor Author

I have the jail up and dhcpd is running. From within the jail, I can reach the outside world (e.g. ping, host, etc), but the outside world cannot reach it.

I can ssh to the jail from the jail host, but not from other hosts.

There is no firewall on this host.

ifconfig on the host does not show 10.55.0.13 (the IP address I am trying to reach)

If I go into the jail via iocage console, I see the expected IP addresses in there.

I think I am missing a step.

@dlangille
Copy link
Contributor Author

Got it.

$ sudo iocage set ip4_addr="vnet0|10.55.0.13/24" toiler
ip4_addr: vnet0|10.55.0.13 -> vnet0|10.55.0.13/24

Then:

[dan@knew:/iocage/jails/toiler] $ sudo iocage restart toiler
* Stopping toiler
  + Executing prestop OK
  + Stopping services OK
  + Tearing down VNET OK
  + Removing devfs_ruleset: 25 OK
  + Removing jail process OK
  + Executing poststop OK
* Starting toiler
  + Started OK
  + Using devfs_ruleset: 25
  + Configuring VNET OK
  + Using IP options: vnet
  + Starting services OK
  + Executing poststart OK
[dan@knew:/iocage/jails/toiler] $ 

NOTE the lack of UPS messages.

The missing mask is the key.

@dlangille
Copy link
Contributor Author

dlangille commented Aug 7, 2019

This is py36-iocage-devel-1.0.0.20190801,1 (built from master on that date)

After rebooting this server today, no jails started up:

2019/08/07 16:12:12 (INFO) * Ruleset 7 does not exist, using defaults
2019/08/07 16:12:12 (ERROR) No bridge for interface ix0found in configuration.

Trying manually, to reproduce the message:

[dan@knew:~] $ sudo iocage start toiler
No bridge for interface ix0found in configuration.

I first searched the code to fix that whitespace typo, see #1015

Based on the previous conversation, I thought iocage would take care of the bridge, hence:

[dan@knew:~] $ grep bridge /etc/rc.conf
[dan@knew:~] $ 

So I did this:

[dan@knew:~] $ ifconfig bridge0
ifconfig: interface bridge0 does not exist
[dan@knew:~] $ sudo ifconfig bridge create
bridge0
[dan@knew:~] $ 

Reading above, I thought this would work:

[dan@knew:~] $ sudo ifconfig bridge0 addm ix0
[dan@knew:~] $ sudo iocage start toiler
No bridge for interface ix0found in configuration.
[dan@knew:~] $ ifconfig bridge0
bridge0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:ca:b7:29:ce:00
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
	member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 1 priority 128 path cost 2000
	groups: bridge 
	nd6 options=1<PERFORMNUD>
[dan@knew:~] $ 

@dlangille
Copy link
Contributor Author

It looks like iocage start ALL does not honor boot=off. Shall I raise an issue?

[dan@knew:~] $ iocage get boot toiler
0
[dan@knew:~] $ sudo iocage start ALL
No bridge for interface ix0found in configuration.
[dan@knew:~] $ 

So much for not starting the jails one by one.

@dlangille
Copy link
Contributor Author

OK, that's the other jails started, let's see if I can figure this problem out.

@dlangille
Copy link
Contributor Author

dlangille commented Aug 7, 2019

I've been playing in the code.

get_interface_bridge_map() outputs: vnet0 bridge0

interface is: ix0

It seems like ix0 should be in the map too.

This seems OK though:

$ iocage get all toiler | grep vnet
interfaces:vnet0:bridge0
ip4_addr:vnet0|10.55.0.13/24
ip6_addr:vnet0|2001:470:8abf:7055:31da:7e45:0:53/64
vnet:1
vnet0_mac:02ff6023eef1 02ff6023eef2
vnet1_mac:none
vnet2_mac:none
vnet3_mac:none
vnet_default_interface:auto
vnet_interfaces:none

@dlangille
Copy link
Contributor Author

dlangille commented Dec 5, 2019

It seems like configuring any devfs_ruleset != 4 results on that ruleset being deleted on iocage stop. This needs a fix ASAP (I might look it myself, let's see).

That is exactly what is happening:

[dan@r720-01:/iocage/jails] $ sudo devfs rule -s 42 show
100 include 1
200 include 2
300 include 3
400 path zfs unhide
500 include 41

[dan@r720-01:/iocage/jails] $ sudo iocage set devfs_ruleset=42 bacula-sd-02.int.unixathome.org
devfs_ruleset: 45 -> 42

[dan@r720-01:/iocage/jails] $ sudo iocage start bacula-sd-02.int.unixathome.org
* Starting bacula-sd-02_int_unixathome_org
  + Started OK
  + Using devfs_ruleset: 42
  + Using IP options: ip4.addr=ix0|10.55.0.33 ip4.saddrsel=1 ip4=new ip6.saddrsel=1 ip6=new
  + Starting services OK
  + Executing poststart OK

[dan@r720-01:/iocage/jails] $ sudo devfs rule -s 42 show
100 include 1
200 include 2
300 include 3
400 path zfs unhide
500 include 41

[dan@r720-01:/iocage/jails] $ sudo iocage stop bacula-sd-02.int.unixathome.org
* Stopping bacula-sd-02_int_unixathome_org
  + Executing prestop OK
  + Stopping services OK
  + Removing devfs_ruleset: 42 OK
  + Removing jail process OK
  + Executing poststop OK
[dan@r720-01:/iocage/jails] $ sudo iocdevfs rule -s 42 show
[dan@r720-01:/iocage/jails] $ 

grembo added a commit to grembo/iocage that referenced this issue Dec 5, 2019
cloned/copied into dynamic ones.  This makes devfs rule handling more
symmetrical to what is done when the default ruleset 4 is configured (which
in fact never applies devfs ruleset 4, but creates an iocage specific
dynamic ruleset instead - which can be quite confusing).

As a result, this addresses the problem of non-dynamic rulesets being
removed on `iocage stop` raised in
iocage#952.

This also makes iocage fail to start a jail if the configured devfs ruleset
is not available - beforehand it would start with a default ruleset in this
case, which can have severe unwanted side effects.

Finally, this sets the minimum dynamically assigned devfs rule id to 1000 to
reserve the lower ids for static configuration by the admin in devfs.rules
(this is mostly for convenience).
grembo added a commit to grembo/iocage that referenced this issue Dec 5, 2019
cloned/copied into dynamic ones.  This makes devfs rule handling more
symmetrical to what is done when the default ruleset 4 is configured (which
in fact never applies devfs ruleset 4, but creates an iocage specific
dynamic ruleset instead - which can be quite confusing).

As a result, this addresses the problem of non-dynamic rulesets being
removed on `iocage stop` raised in
iocage#952.

This also makes iocage fail to start a jail if the configured devfs ruleset
is not available - beforehand it would start with a default ruleset in this
case, which can have severe unwanted side effects.

Finally, this sets the minimum dynamically assigned devfs rule id to 1000 to
reserve the lower ids for static configuration by the admin in devfs.rules
(this is mostly for convenience).
@grembo
Copy link
Contributor

grembo commented Dec 5, 2019

@dlangille I created a pull request that fixes the issue by creating dynamic rules from configured static rules, please see #1106 for details.

It also stops jail start in case a configured devfs rule isn't available (before it would start with default rules instead).

For a quick fix/testing, download and apply the raw patch to ioc_common.py.

@dlangille
Copy link
Contributor Author

dlangille commented Dec 5, 2019

Thank you for this work. I appreciate it.

This is the result.

EDIT: this was an invalid test. The jail must be started with the patched code. It was not.

[dan@r720-01:/usr/local/lib/python3.6/site-packages] $ sudo cp -ia iocage_lib/ioc_common.py iocage_lib/ioc_common.py.original
[dan@r720-01:/usr/local/lib/python3.6/site-packages] $ sudo patch < ~/tmp/87fb1675a43d26a4a4252e2ed3a48f3134ea1c5c.diff 
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/iocage_lib/ioc_common.py b/iocage_lib/ioc_common.py
|index 292dbe50..32455371 100644
|--- a/iocage_lib/ioc_common.py
|+++ b/iocage_lib/ioc_common.py
--------------------------
Patching file iocage_lib/ioc_common.py using Plan A...
Hunk #1 succeeded at 737 (offset -3 lines).
Hunk #2 succeeded at 745 (offset -3 lines).
done
[dan@r720-01:/usr/local/lib/python3.6/site-packages] $ sudo iocdevfs rule -s 42 show
sudo: iocdevfs: command not found
[dan@r720-01:/usr/local/lib/python3.6/site-packages] $ sudo devfs rule -s 42 show
100 include 1
200 include 2
300 include 3
400 path zfs unhide
500 include 41
[dan@r720-01:/usr/local/lib/python3.6/site-packages] $ sudo iocage stop bacula-sd-02.int.unixathome.org
* Stopping bacula-sd-02_int_unixathome_org
  + Executing prestop OK
  + Stopping services OK
  + Removing devfs_ruleset: 42 OK
  + Removing jail process OK
  + Executing poststop OK
[dan@r720-01:/usr/local/lib/python3.6/site-packages] $ sudo devfs rule -s 42 show
[dan@r720-01:/usr/local/lib/python3.6/site-packages] $ 

@dlangille
Copy link
Contributor Author

I will retest. I think the above was an invalid test.

@dlangille
Copy link
Contributor Author

After the above, I did this:

[dan@r720-01:/usr/local/lib/python3.6/site-packages] $ sudo service devfs restart
[dan@r720-01:/usr/local/lib/python3.6/site-packages] $ sudo devfs rule -s 42 show
100 include 1
200 include 2
300 include 3
400 path zfs unhide
500 include 41
[dan@r720-01:/usr/local/lib/python3.6/site-packages] $ sudo iocage start bacula-sd-02.int.unixathome.org
* Starting bacula-sd-02_int_unixathome_org
  + Started OK
  + Using devfs_ruleset: 1000
  + Using IP options: ip4.addr=ix0|10.55.0.33 ip4.saddrsel=1 ip4=new ip6.saddrsel=1 ip6=new
  + Starting services OK
  + Executing poststart OK
[dan@r720-01:/usr/local/lib/python3.6/site-packages] $ sudo devfs rule -s 42 show
100 include 1
200 include 2
300 include 3
400 path zfs unhide
500 include 41
[dan@r720-01:/usr/local/lib/python3.6/site-packages] $ sudo iocage stop bacula-sd-02.int.unixathome.org
* Stopping bacula-sd-02_int_unixathome_org
  + Executing prestop OK
  + Stopping services OK
  + Removing devfs_ruleset: 1000 OK
  + Removing jail process OK
  + Executing poststop OK
[dan@r720-01:/usr/local/lib/python3.6/site-packages] $ sudo devfs rule -s 42 show
100 include 1
200 include 2
300 include 3
400 path zfs unhide
500 include 41
[dan@r720-01:/usr/local/lib/python3.6/site-packages] $ 

Success. Thank you.

@grembo
Copy link
Contributor

grembo commented Dec 5, 2019

@dlangille Cool, thanks for testing/confirming.

Could you maybe also check sudo devfs rule -s 1000 show before, during, and after running the jail? (1000 being the ruleset id shown on jail start Using devfs_ruleset: 1xxx

grembo added a commit to grembo/iocage that referenced this issue Dec 5, 2019
cloned/copied into dynamic ones.  This makes devfs rule handling more
symmetrical to what is done when the default ruleset 4 is configured (which
in fact never applies devfs ruleset 4, but creates an iocage specific
dynamic ruleset instead - which can be quite confusing).

As a result, this addresses the problem of non-dynamic rulesets being
removed on `iocage stop` raised in
iocage#952.

This also makes iocage fail to start a jail if the configured devfs ruleset
is not available - beforehand it would start with a default ruleset in this
case, which can have severe unwanted side effects.

Finally, this sets the minimum dynamically assigned devfs rule id to 1000 to
reserve the lower ids for static configuration by the admin in devfs.rules
(this is mostly for convenience).
@dlangille
Copy link
Contributor Author

I picked the mqtt01 jail.

[dan@r720-01:~] $ sudo devfs rule -s 1000 show
[dan@r720-01:~] $ sudo devfs rule -s 6 show
100 hide
200 path null unhide
300 path zero unhide
400 path crypto unhide
500 path random unhide
600 path urandom unhide
700 path ptyp* unhide
800 path ptyq* unhide
900 path ptyr* unhide
1000 path ptys* unhide
1100 path ptyP* unhide
1200 path ptyQ* unhide
1300 path ptyR* unhide
1400 path ptyS* unhide
1500 path ptyl* unhide
1600 path ptym* unhide
1700 path ptyn* unhide
1800 path ptyo* unhide
1900 path ptyL* unhide
2000 path ptyM* unhide
2100 path ptyN* unhide
2200 path ptyO* unhide
2300 path ttyp* unhide
2400 path ttyq* unhide
2500 path ttyr* unhide
2600 path ttys* unhide
2700 path ttyP* unhide
2800 path ttyQ* unhide
2900 path ttyR* unhide
3000 path ttyS* unhide
3100 path ttyl* unhide
3200 path ttym* unhide
3300 path ttyn* unhide
3400 path ttyo* unhide
3500 path ttyL* unhide
3600 path ttyM* unhide
3700 path ttyN* unhide
3800 path ttyO* unhide
3900 path ptmx unhide
4000 path pts unhide
4100 path pts/* unhide
4200 path fd unhide
4300 path fd/* unhide
4400 path stdin unhide
4500 path stdout unhide
4600 path stderr unhide
4700 path zfs unhide

Yeah, that's not right is it?

[dan@r720-01:/iocage/jails] $ sudo iocage restart mqtt01
* Stopping mqtt01
  + Executing prestop OK
  + Stopping services OK
  + Removing devfs_ruleset: 6 OK
  + Removing jail process OK
  + Executing poststop OK
* Starting mqtt01
  + Started OK
  + Using devfs_ruleset: 1000
  + Using IP options: ip4.addr=127.1.0.201,ix0|[redacted] ip4.saddrsel=1 ip4=new ip6.saddrsel=1 ip6=new
  + Starting services OK
  + Executing poststart OK
[dan@r720-01:/iocage/jails] $ sudo devfs rule -s 1000 show
100 hide
200 path null unhide
300 path zero unhide
400 path crypto unhide
500 path random unhide
600 path urandom unhide
700 path ptyp* unhide
800 path ptyq* unhide
900 path ptyr* unhide
1000 path ptys* unhide
1100 path ptyP* unhide
1200 path ptyQ* unhide
1300 path ptyR* unhide
1400 path ptyS* unhide
1500 path ptyl* unhide
1600 path ptym* unhide
1700 path ptyn* unhide
1800 path ptyo* unhide
1900 path ptyL* unhide
2000 path ptyM* unhide
2100 path ptyN* unhide
2200 path ptyO* unhide
2300 path ttyp* unhide
2400 path ttyq* unhide
2500 path ttyr* unhide
2600 path ttys* unhide
2700 path ttyP* unhide
2800 path ttyQ* unhide
2900 path ttyR* unhide
3000 path ttyS* unhide
3100 path ttyl* unhide
3200 path ttym* unhide
3300 path ttyn* unhide
3400 path ttyo* unhide
3500 path ttyL* unhide
3600 path ttyM* unhide
3700 path ttyN* unhide
3800 path ttyO* unhide
3900 path ptmx unhide
4000 path pts unhide
4100 path pts/* unhide
4200 path fd unhide
4300 path fd/* unhide
4400 path stdin unhide
4500 path stdout unhide
4600 path stderr unhide
4700 path zfs unhide
[dan@r720-01:/iocage/jails] $ 

Oh. Same thing. Oh..

@grembo
Copy link
Contributor

grembo commented Dec 5, 2019

@dlangille Well, what is that jail's configuration? I assume that jail has ruleset=4 configured (it was still running from before the update, that's why it had rule id 6). So having the same rules here makes sense, right?

@dlangille
Copy link
Contributor Author

The jail is indeed configured with devfs_ ruleset=4

But given that, I would expect:

$ sudo devfs rule -s 4 show
100 include 1
200 include 2
300 include 3
400 path fuse unhide
500 path zfs unhide

@dlangille
Copy link
Contributor Author

Perhaps this is the result of the clone?

@grembo
Copy link
Contributor

grembo commented Dec 5, 2019

@dlangille

The jail is indeed configured with devfs_ ruleset=4

But given that, I would expect:

$ sudo devfs rule -s 4 show
100 include 1
200 include 2
300 include 3
400 path fuse unhide
500 path zfs unhide

Well, that's why I wrote in #1106 :

This makes devfs rule handling more
symmetrical to what is done when the default ruleset 4 is configured (which
in fact never applies devfs ruleset 4, but creates an iocage specific
dynamic ruleset instead - which can be quite confusing
).

This isn't something I introduced, that's how iocage behaved already (as you can compare with an unpatched version). That's why it never killed ruleset 4, but only custom ones: It always creates a dynamic custom ruleset if "4" is magically configured, but it didn't if you defined one yourself. The dynamic one was removed on stop, which didn't cause any harm (as it's not 4), but it caused harm with any other id. Therefore, any changes to ruleid 4 in devfs.rules has (and had) no impact whatsoever. That's a problem in itself, but I can't solve this, as it seems really hard to migrate away from it.

My patch basically treats statically configured ruleset ids != 4 the same - copy and create a dynamic one, which then can be safely disposed.

So what I was hoping for was a test like described with a custom ruleset != 4 configured.

p.s. There's also a problem with iocage get (if a jail isn't running it shoes different values than if it is), but that's also outside of the scope of this patch, as it worked like that beforehand.

@grembo
Copy link
Contributor

grembo commented Dec 5, 2019

@dlangille The unpatched/unaltered code in question can be found here:

if ruleset != '4':
if int(ruleset) in ruleset_list:
return str(ruleset)
logit({
"level": "INFO",
"message": f'* Ruleset {ruleset} does not exist, using defaults'
},
_callback=callback,
silent=silent)
ruleset = 5 # 0-4 is always reserved
while ruleset in ruleset_list:
ruleset += 1
ruleset = str(ruleset)
devfs_dict = dict((dev, None) for dev in (
'hide', 'null', 'zero', 'crypto', 'random', 'urandom', 'ptyp*',
'ptyq*', 'ptyr*', 'ptys*', 'ptyP*', 'ptyQ*', 'ptyR*', 'ptyS*', 'ptyl*',
'ptym*', 'ptyn*', 'ptyo*', 'ptyL*', 'ptyM*', 'ptyN*', 'ptyO*', 'ttyp*',
'ttyq*', 'ttyr*', 'ttys*', 'ttyP*', 'ttyQ*', 'ttyR*', 'ttyS*', 'ttyl*',
'ttym*', 'ttyn*', 'ttyo*', 'ttyL*', 'ttyM*', 'ttyN*', 'ttyO*', 'ptmx',
'pts', 'pts/*', 'fd', 'fd/*', 'stdin', 'stdout', 'stderr', 'zfs'
))
# We set these up by default above
skip_includes = ['$devfsrules_hide_all', '$devfsrules_unhide_basic',
'$devfsrules_unhide_login']
if includes is not None:
devfs_includes = [include for include in includes if include not in
skip_includes]
if paths is not None:
devfs_dict.update(paths)
# We may end up setting all of these.
if check_truthy(conf['allow_mount_fusefs']):
devfs_dict['fuse'] = None
if check_truthy(conf['bpf']):
devfs_dict['bpf*'] = None
if check_truthy(conf['allow_tun']):
devfs_dict['tun*'] = None
for include in devfs_includes:
su.run(
['devfs', 'rule', '-s', ruleset, 'add', 'include', include],
stdout=su.PIPE
)
for path, mode in devfs_dict.items():
# # Default hide all
if path == 'hide':
su.run(
['devfs', 'rule', '-s', ruleset, 'add', 'hide'],
stdout=su.PIPE
)
continue
path = ['add', 'path', path]
if mode is not None:
path += [mode]
else:
path += ['unhide']
su.run(['devfs', 'rule', '-s', ruleset] + path, stdout=su.PIPE)
return ruleset

There you can see how a ruleset configuration of "4" receives special treatment by synthesising a ruleset that's not related to what is actually configured as ruleset 4 in devfs. This ruleset is later deleted on iocage stop.

My patch alters this by cloning the configured devfs_ruleset into a dynamic one, so that the cleanup mechanism works the same as for ruleset "4".

https://github.com/grembo/iocage/blob/30c8f2d361fad1a11441301b95342e9cb7c6b80d/iocage_lib/ioc_common.py#L751-L782

It only does that for anything != 4 though, as it otherwise would break existing iocage installations by replacing what iocage does with what a default FreeBSD install has in devfs.rules. That's why my patch doesn't change any behaviour for devfs_ruleset=4.

iocage has that default behaviour for a reason. I'm unhappy that they used "4" as a magic number for that (it feels wrong on so many different levels and is super confusing), but that's just what it is.

@grembo
Copy link
Contributor

grembo commented Dec 6, 2019

Or in pseudo-code:

Original behaviour:

onstart:
    assign_rule(rule == 4 ? create_dynamic_rule_unrelated_to_rule_4() : ruleid)
onstop:
    delete_assigned_rule()

New behaviour:

onstart:
    assign_rule(rule == 4
        ? create_dynamic_rule_unrelated_to_rule_4() 
        : create_dynamic_rule_from_configured_static_rule(ruleid));
onstop:
    delete_assigned_rule()

@dlangille Does this make it more clear/understandable?
Update: Make pseudo-code more C-like ;)

@dlangille
Copy link
Contributor Author

I follow. Thank you.

@sonicaj
Copy link
Member

sonicaj commented Jan 18, 2020

@dlangille can you please confirm if it's safe to close this issue as I think @grembo's patch fixes your concern ? ( Thank you @grembo btw for the contribution ;) )

@dlangille
Copy link
Contributor Author

Confirmed, I have had no recurrence of this issue.

@sonicaj
Copy link
Member

sonicaj commented Jan 18, 2020

Thank you for confirming @dlangille. Closing this as #1106 will fix this.

@sonicaj sonicaj closed this as completed Jan 18, 2020
sonicaj pushed a commit that referenced this issue Jan 22, 2020
…rulesets != 4 are (#1106)

* Change devfs ruleset handling so that configured rulesets != 4 are
cloned/copied into dynamic ones.  This makes devfs rule handling more
symmetrical to what is done when the default ruleset 4 is configured (which
in fact never applies devfs ruleset 4, but creates an iocage specific
dynamic ruleset instead - which can be quite confusing).

As a result, this addresses the problem of non-dynamic rulesets being
removed on `iocage stop` raised in
#952.

This also makes iocage fail to start a jail if the configured devfs ruleset
is not available - beforehand it would start with a default ruleset in this
case, which can have severe unwanted side effects.

Finally, this sets the minimum dynamically assigned devfs rule id to 1000 to
reserve the lower ids for static configuration by the admin in devfs.rules
(this is mostly for convenience).

* Change devfs_ruleset config parsing on jail start, so that:
- Users get a helpful error message in case a configured devfs_ruleset
  doesn't exist (also shows the configured ruleset and not
  the dynamically created one, which was not helpful).
- Users learn on jail start about how the devfs_ruleset is created
  (show id it was cloned from or that it is based on
  iocage's default).
- Avoid leaking devfs_rulesets on starting plugins that define
  devfs_paths/devfs_includes (would lose one ruleset
  everytime otherwise).
- Show a warning in case a plugin with devfs_paths/devfs_includes
  is started with a manually configured devfs_ruleset (as
  this won't - and never did - apply those.
- Move magic numbers to constants in ioc_common.py.

This doesn't fix the iocage man page, which shows (and had shown)
inaccurate information on this for a while.

* Move "min_dyn_devfs_ruleset" to jail property, change jail start devfs_ruleset
message to be one-liner containing information on configured devfs ruleset
(configured or iocage generated).

* Validate 'devfs_ruleset' and 'min_dyn_devfs_ruleset' while setting.
bugclerk pushed a commit to truenas/iocage that referenced this issue Jan 22, 2020
cloned/copied into dynamic ones.  This makes devfs rule handling more
symmetrical to what is done when the default ruleset 4 is configured (which
in fact never applies devfs ruleset 4, but creates an iocage specific
dynamic ruleset instead - which can be quite confusing).

As a result, this addresses the problem of non-dynamic rulesets being
removed on `iocage stop` raised in
iocage#952.

This also makes iocage fail to start a jail if the configured devfs ruleset
is not available - beforehand it would start with a default ruleset in this
case, which can have severe unwanted side effects.

Finally, this sets the minimum dynamically assigned devfs rule id to 1000 to
reserve the lower ids for static configuration by the admin in devfs.rules
(this is mostly for convenience).

(cherry picked from commit 30c8f2d)
sonicaj pushed a commit to truenas/iocage that referenced this issue Jan 22, 2020
…rulesets != 4 are (iocage#1106)

* Change devfs ruleset handling so that configured rulesets != 4 are
cloned/copied into dynamic ones.  This makes devfs rule handling more
symmetrical to what is done when the default ruleset 4 is configured (which
in fact never applies devfs ruleset 4, but creates an iocage specific
dynamic ruleset instead - which can be quite confusing).

As a result, this addresses the problem of non-dynamic rulesets being
removed on `iocage stop` raised in
iocage#952.

This also makes iocage fail to start a jail if the configured devfs ruleset
is not available - beforehand it would start with a default ruleset in this
case, which can have severe unwanted side effects.

Finally, this sets the minimum dynamically assigned devfs rule id to 1000 to
reserve the lower ids for static configuration by the admin in devfs.rules
(this is mostly for convenience).

* Change devfs_ruleset config parsing on jail start, so that:
- Users get a helpful error message in case a configured devfs_ruleset
  doesn't exist (also shows the configured ruleset and not
  the dynamically created one, which was not helpful).
- Users learn on jail start about how the devfs_ruleset is created
  (show id it was cloned from or that it is based on
  iocage's default).
- Avoid leaking devfs_rulesets on starting plugins that define
  devfs_paths/devfs_includes (would lose one ruleset
  everytime otherwise).
- Show a warning in case a plugin with devfs_paths/devfs_includes
  is started with a manually configured devfs_ruleset (as
  this won't - and never did - apply those.
- Move magic numbers to constants in ioc_common.py.

This doesn't fix the iocage man page, which shows (and had shown)
inaccurate information on this for a while.

* Move "min_dyn_devfs_ruleset" to jail property, change jail start devfs_ruleset
message to be one-liner containing information on configured devfs ruleset
(configured or iocage generated).

* Validate 'devfs_ruleset' and 'min_dyn_devfs_ruleset' while setting.
bugclerk pushed a commit to truenas/iocage that referenced this issue Mar 13, 2020
…rulesets != 4 are (iocage#1106)

* Change devfs ruleset handling so that configured rulesets != 4 are
cloned/copied into dynamic ones.  This makes devfs rule handling more
symmetrical to what is done when the default ruleset 4 is configured (which
in fact never applies devfs ruleset 4, but creates an iocage specific
dynamic ruleset instead - which can be quite confusing).

As a result, this addresses the problem of non-dynamic rulesets being
removed on `iocage stop` raised in
iocage#952.

This also makes iocage fail to start a jail if the configured devfs ruleset
is not available - beforehand it would start with a default ruleset in this
case, which can have severe unwanted side effects.

Finally, this sets the minimum dynamically assigned devfs rule id to 1000 to
reserve the lower ids for static configuration by the admin in devfs.rules
(this is mostly for convenience).

* Change devfs_ruleset config parsing on jail start, so that:
- Users get a helpful error message in case a configured devfs_ruleset
  doesn't exist (also shows the configured ruleset and not
  the dynamically created one, which was not helpful).
- Users learn on jail start about how the devfs_ruleset is created
  (show id it was cloned from or that it is based on
  iocage's default).
- Avoid leaking devfs_rulesets on starting plugins that define
  devfs_paths/devfs_includes (would lose one ruleset
  everytime otherwise).
- Show a warning in case a plugin with devfs_paths/devfs_includes
  is started with a manually configured devfs_ruleset (as
  this won't - and never did - apply those.
- Move magic numbers to constants in ioc_common.py.

This doesn't fix the iocage man page, which shows (and had shown)
inaccurate information on this for a while.

* Move "min_dyn_devfs_ruleset" to jail property, change jail start devfs_ruleset
message to be one-liner containing information on configured devfs ruleset
(configured or iocage generated).

* Validate 'devfs_ruleset' and 'min_dyn_devfs_ruleset' while setting.

(cherry picked from commit a69d58b)
skarekrow pushed a commit that referenced this issue Sep 12, 2020
…rulesets != 4 are (#1106)

* Change devfs ruleset handling so that configured rulesets != 4 are
cloned/copied into dynamic ones.  This makes devfs rule handling more
symmetrical to what is done when the default ruleset 4 is configured (which
in fact never applies devfs ruleset 4, but creates an iocage specific
dynamic ruleset instead - which can be quite confusing).

As a result, this addresses the problem of non-dynamic rulesets being
removed on `iocage stop` raised in
#952.

This also makes iocage fail to start a jail if the configured devfs ruleset
is not available - beforehand it would start with a default ruleset in this
case, which can have severe unwanted side effects.

Finally, this sets the minimum dynamically assigned devfs rule id to 1000 to
reserve the lower ids for static configuration by the admin in devfs.rules
(this is mostly for convenience).

* Change devfs_ruleset config parsing on jail start, so that:
- Users get a helpful error message in case a configured devfs_ruleset
  doesn't exist (also shows the configured ruleset and not
  the dynamically created one, which was not helpful).
- Users learn on jail start about how the devfs_ruleset is created
  (show id it was cloned from or that it is based on
  iocage's default).
- Avoid leaking devfs_rulesets on starting plugins that define
  devfs_paths/devfs_includes (would lose one ruleset
  everytime otherwise).
- Show a warning in case a plugin with devfs_paths/devfs_includes
  is started with a manually configured devfs_ruleset (as
  this won't - and never did - apply those.
- Move magic numbers to constants in ioc_common.py.

This doesn't fix the iocage man page, which shows (and had shown)
inaccurate information on this for a while.

* Move "min_dyn_devfs_ruleset" to jail property, change jail start devfs_ruleset
message to be one-liner containing information on configured devfs ruleset
(configured or iocage generated).

* Validate 'devfs_ruleset' and 'min_dyn_devfs_ruleset' while setting.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants