Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ruleset exists but iocage does not find it #952

Open
dlangille opened this issue Jun 17, 2019 · 30 comments

Comments

@dlangille
Copy link
Contributor

commented Jun 17, 2019

This worked before upgrading iocage today and upgrading the jail to 12.0-RELEASE-p5

Make sure to follow and check these boxes before submitting an issue! Thank you.

$ iocage --version
Version	1.2 RC

The above is head from 2019-06-16:

$ pkg info -x iocage
py36-iocage-devel-1.0.0.20190616,1

Today I upgraded iocage from 1.0.0.20190519_1,1

When the jail is started, the rule set is not invoked/found:

[dan@knew:~] $ date
Mon Jun 17 01:17:55 UTC 2019
[dan@knew:~] $ sudo iocage start toiler
* Ruleset 7 does not exist, using defaults
* Starting toiler
  + Started OK
  + Using devfs_ruleset: 7
  + Using IP options: ip4.addr=10.55.0.13,127.1.0.53 ip6.addr=2001:470:8abf:[redacted] ip4.saddrsel=1 ip4=new ip6.saddrsel=1 ip6=new
  + Starting services OK
  + Executing poststart OK


[dan@knew:~] $ tail /var/log/iocage.log 
2019/06/17 01:17:01 (INFO)   + Removing devfs_ruleset: 7 OK
2019/06/17 01:17:01 (INFO)   + Removing jail process OK
2019/06/17 01:17:01 (INFO)   + Executing poststop OK
2019/06/17 01:18:05 (INFO) * Ruleset 7 does not exist, using defaults
2019/06/17 01:18:05 (INFO) * Starting toiler
2019/06/17 01:18:05 (INFO)   + Started OK
2019/06/17 01:18:05 (INFO)   + Using devfs_ruleset: 7
2019/06/17 01:18:05 (INFO)   + Using IP options: ip4.addr=10.55.0.13,127.1.0.53 ip6.addr=2001:470:8abf:[redacted] ip4.saddrsel=1 ip4=new ip6.saddrsel=1 ip6=new
2019/06/17 01:18:08 (INFO)   + Starting services OK
2019/06/17 01:18:08 (INFO)   + Executing poststart OK
[dan@knew:~] $ 

NOTE the 'does not exist' in the log extra above.

Within /etc/devfs.rules we have

# for dhcpd in jail toiler
#
[devfsrules_jail_bpf=7]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'bpf*' unhide

# added based on https://forums.freebsd.org/threads/jailed-dhcp-server-and-tftp-handoff.29934/
#add path net unhide
#add path 'net/*' unhide

# Support for TUN devices
#

At present, dhcpd in the jail cannot be started because:

Jun 17 01:18:07 toiler dhcpd[61983]: No bpf devices. Please read the README section for your operating system.

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Jun 17, 2019

For fun, I did this:

[dan@knew:~] $ sudo iocage set devfs_ruleset=10 toiler
devfs_ruleset: 7 -> 10

Ruleset 10 does not exist in /etc/devfs.rules

Restarting the jail results and there is no 'does not exist' exist message, despite the fact that the ruleset does not in fact exist.

2019/06/17 13:50:16 (INFO) devfs_ruleset: 7 -> 10
2019/06/17 13:50:26 (INFO) * Stopping toiler
2019/06/17 13:50:26 (INFO)   + Executing prestop OK
2019/06/17 13:50:28 (INFO)   + Stopping services OK
2019/06/17 13:50:28 (INFO)   + Removing devfs_ruleset: 7 OK
2019/06/17 13:50:28 (INFO)   + Removing jail process OK
2019/06/17 13:50:28 (INFO)   + Executing poststop OK
2019/06/17 13:50:29 (INFO) * Starting toiler
2019/06/17 13:50:29 (INFO)   + Started OK
2019/06/17 13:50:29 (INFO)   + Using devfs_ruleset: 10
2019/06/17 13:50:29 (INFO)   + Using IP options: ip4.addr=10.55.0.13,127.1.0.53 ip6.addr=2001:470:8abf:[redacted] ip4.saddrsel=1 ip4=new ip6.saddrsel=1 ip6=new
2019/06/17 13:50:30 (INFO)   + Starting services OK
2019/06/17 13:50:30 (INFO)   + Executing poststart OK

Next, let us renumber the ruleset in the file:

[dan@knew:~] $ diff -ruN /etc/devfs.rules~ /etc/devfs.rules
--- /etc/devfs.rules~	2019-06-17 13:49:17.000000000 +0000
+++ /etc/devfs.rules	2019-06-17 13:50:57.257191000 +0000
@@ -33,7 +33,7 @@
 
 # for dhcpd in jail toiler
 #
-[devfsrules_jail_bpf=7]
+[devfsrules_jail_bpf=10]
 add include $devfsrules_hide_all
 add include $devfsrules_unhide_basic
 add include $devfsrules_unhide_login
[dan@knew:~] $ 

Restarting the jail, and again, no message about 'does not exist', which is correct this time.

2019/06/17 13:50:50 (INFO) * Stopping toiler
2019/06/17 13:50:50 (INFO) + Executing prestop OK
2019/06/17 13:50:52 (INFO) + Stopping services OK
2019/06/17 13:50:52 (INFO) + Removing devfs_ruleset: 10 OK
2019/06/17 13:50:52 (INFO) + Removing jail process OK
2019/06/17 13:50:52 (INFO) + Executing poststop OK
2019/06/17 13:51:04 (INFO) * Starting toiler
2019/06/17 13:51:04 (INFO) + Started OK
2019/06/17 13:51:04 (INFO) + Using devfs_ruleset: 10
2019/06/17 13:51:04 (INFO) + Using IP options: ip4.addr=10.55.0.13,127.1.0.53 ip6.addr=2001:470:8abf:[redacted] ip4.saddrsel=1 ip4=new ip6.saddrsel=1 ip6=new
2019/06/17 13:51:05 (INFO) + Starting services OK
2019/06/17 13:51:05 (INFO) + Executing poststart OK

Now dhcpd can start

My questions:

  • Why did this break?
  • Why does 7, which existed, not get detected?
  • Why does 10, which did not exist, get 'found'?
  • Why does 10, when it exists, get found but 7 did not?
@skarekrow

This comment has been minimized.

Copy link
Member

commented Jun 24, 2019

I recommend leaving that on the default 4 with dhcp on, as iocage will handle the devfs rulesets for you and unhide the same devices.

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Jun 24, 2019

From man poudriere:

     dhcp=[1 | 0]  This controls starting the jail with the Dynamic Host
                   Configuration Protocol enabled.  To enable dhcp, vnet and
                   bpf must also be enabled.

                   Default: 0

                   Source: local

Do we both understand this the same way?

This jail has a static IP address.

dhcpd runs inside this jail.

I think the iocage dhcp configuration setting is not related to this use case.

@skarekrow

This comment has been minimized.

Copy link
Member

commented Jun 24, 2019

In which case just set the bpf property and vnet. Don't set dhcp. iocage will handle the rulesets for you as long as the property is at the default (4)

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Jun 24, 2019

Here is something interesting:

$ iocage get all toiler | grep set
allow_set_hostname:1
cpuset:off
devfs_ruleset:10
jail_zfs_dataset:iocage/jails/toiler/data


$ sudo iocage set devfs_ruleset=4 toiler
devfs_ruleset: 10 -> 4


$ sudo iocage set devfs_ruleset=10 toiler
devfs_ruleset: 10 -> 10

Wait... I changed it to 4, then back to 10, and it's already 10?

Let's try it again:

[dan@knew:~] $ sudo iocage set devfs_ruleset=10 toiler
devfs_ruleset: 10 -> 10
[dan@knew:~] $ iocage get devfs_ruleset toiler
10
[dan@knew:~] $ sudo iocage set devfs_ruleset=4 toilerler
devfs_ruleset: 10 -> 4
[dan@knew:~] $ iocage get devfs_ruleset toiler
10
[dan@knew:~] $ 

OK, let's look in there again:

[dan@knew:~] $ cd /iocage/jails/toiler/
[dan@knew:/iocage/jails/toiler] $ ls -lt
total 29
-rw-r--r--   1 root  wheel  3493 Jun 24 18:16 config.json
drwxr-xr-x  19 root  wheel    24 Mar 18 17:08 root
-rw-r--r--   1 root  wheel     0 Dec 21  2018 fstab
[dan@knew:/iocage/jails/toiler] $ date
Mon Jun 24 18:17:50 UTC 2019
[dan@knew:/iocage/jails/toiler] $ grep devfs_ruleset config.json 
    "devfs_ruleset": "4",
[dan@knew:/iocage/jails/toiler] $ 
@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Jun 24, 2019

Trying that again, it seems to work. WTF is going on?

[dan@knew:/iocage/jails/toiler] $ grep bpf config.json 
    "bpf": 0,
[dan@knew:/iocage/jails/toiler] $ sudo iocage set bpf=1 toiler
bpf: 0 -> 1
[dan@knew:/iocage/jails/toiler] $ grep bpf config.json 
    "bpf": 1,
[dan@knew:/iocage/jails/toiler] $ sudo iocage set devfs_ruleset=4 toiler
devfs_ruleset: 10 -> 4
[dan@knew:/iocage/jails/toiler] $ grep devfs_ruleset config.json 
    "devfs_ruleset": "4",
[dan@knew:/iocage/jails/toiler] $ 
@skarekrow

This comment has been minimized.

Copy link
Member

commented Jun 24, 2019

That's because iocage get will return the dynamic ruleset, you know this as you commented in the issue (#694) :P

@skarekrow

This comment has been minimized.

Copy link
Member

commented Jun 24, 2019

TLDR; Set bpf and vnet to 1, set devfs_ruleset to 4 and enjoy it working as it should.

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Jun 24, 2019

You attribute much to me but I have no memory of that one.

@skarekrow

This comment has been minimized.

Copy link
Member

commented Jun 24, 2019

;) Well it did indeed happen lol

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Jun 24, 2019

Yeah, but expecting people to recall obscure bits is obscure.

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Jun 24, 2019

I'm off to configure vnet for this dhcp jail.

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Jun 24, 2019

"VNET is considered experimental. Unexpected system crashes can occur. " - https://iocage.readthedocs.io/en/latest/networking.html

Wait, what? I must now use this?

Is there another way?

@skarekrow

This comment has been minimized.

Copy link
Member

commented Jun 24, 2019

That's for earlier RELEASEs, it's stabilized significantly since then. To pass the bpf device in, vnet AFAIK is required.

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Jun 24, 2019

This jail has used bpf since it was created. The requirement to use vnet must be new.

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Jun 24, 2019

The interesting part here:

$ sudo iocage start toiler
* Starting toiler
  + Started OK
  + Using devfs_ruleset: 25
  + Configuring VNET FAILED
  route: writing to routing socket: Network is unreachable
add net default: gateway 10.55.0.1 fib 0: Network is unreachable

Stopped toiler due to VNET failure
                                                                               
Broadcast Message from root@knew.int.unixathome.org                            
        (no tty) at 18:44 UTC...                                               
                                                                               
Communications with UPS lost.                                                                               
                                                                               
Broadcast Message from root@knew.int.unixathome.org                            
        (no tty) at 18:44 UTC...                                               
                                                                               
Communciations with UPS restored.                                                                               

The way it takes down ix0:

Jun 24 18:44:29 knew kernel: epair0a: Ethernet address: 02:8a:bd:d4:9d:0a
Jun 24 18:44:29 knew kernel: epair0b: Ethernet address: 02:8a:bd:d4:9d:0b
Jun 24 18:44:29 knew kernel: epair0a: link state changed to UP
Jun 24 18:44:29 knew kernel: epair0b: link state changed to UP
Jun 24 18:44:29 knew kernel: epair0a: changing name to 'vnet0.20'
Jun 24 18:44:29 knew kernel: ix0: link state changed to DOWN
Jun 24 18:44:30 knew kernel: vnet0.20: promiscuous mode enabled
Jun 24 18:44:30 knew kernel: ix0: link state changed to UP
Jun 24 18:44:30 knew kernel: arp: 02:ff:60:23:ee:f1 is using my IP address 10.55.0.13 on epair0b!
Jun 24 18:44:30 knew kernel: vnet0.20: link state changed to DOWN
Jun 24 18:44:30 knew kernel: epair0b: link state changed to DOWN
Jun 24 18:44:30 knew kernel: ix0: link state changed to DOWN
Jun 24 18:44:30 knew apcupsd[1954]: Communications with UPS lost.
Jun 24 18:44:30 knew kernel: ix0: link state changed to UP
Jun 24 18:44:35 knew apcupsd[1954]: Communications with UPS restored.

I'm guessing that is because 10.55.0.13 is static on ix0 and must now be dynamic. OK, I can do that.

[dan@knew:/iocage/jails/toiler] $ sudo ifconfig ix0 10.55.0.13 delete

Add the IP addresses back in:

$ sudo iocage set ip4_addr="vnet0|10.55.0.13,127.1.0.53" toiler
ip4_addr: 10.55.0.13,127.1.0.53 -> vnet0|10.55.0.13,127.1.0.53

But still:

$ sudo iocage start toiler
* Starting toiler
  + Started OK
  + Using devfs_ruleset: 25
  + Configuring VNET FAILED
  route: writing to routing socket: Network is unreachable
add net default: gateway 10.55.0.1 fib 0: Network is unreachable

Stopped toiler due to VNET failure
Jun 24 18:53:28 knew kernel: epair0a: Ethernet address: 02:46:dd:b8:45:0a
Jun 24 18:53:28 knew kernel: epair0b: Ethernet address: 02:46:dd:b8:45:0b
Jun 24 18:53:28 knew kernel: epair0a: link state changed to UP
Jun 24 18:53:28 knew kernel: epair0b: link state changed to UP
Jun 24 18:53:28 knew kernel: epair0a: changing name to 'vnet0.21'
Jun 24 18:53:28 knew kernel: ix0: link state changed to DOWN
Jun 24 18:53:28 knew kernel: vnet0.21: promiscuous mode enabled
Jun 24 18:53:28 knew kernel: ix0: link state changed to UP
Jun 24 18:53:29 knew kernel: vnet0.21: link state changed to DOWN
Jun 24 18:53:29 knew kernel: epair0b: link state changed to DOWN
Jun 24 18:53:29 knew kernel: ix0: link state changed to DOWN
Jun 24 18:53:29 knew kernel: ix0: link state changed to UP
Jun 24 18:57:51 knew kernel: epair0a: Ethernet address: 02:3f:43:00:8c:0a
Jun 24 18:57:51 knew kernel: epair0b: Ethernet address: 02:3f:43:00:8c:0b
Jun 24 18:57:51 knew kernel: epair0a: link state changed to UP
Jun 24 18:57:51 knew kernel: epair0b: link state changed to UP
Jun 24 18:57:51 knew kernel: epair0a: changing name to 'vnet0.22'
Jun 24 18:57:51 knew kernel: ix0: link state changed to DOWN
Jun 24 18:57:51 knew kernel: vnet0.22: promiscuous mode enabled
Jun 24 18:57:51 knew kernel: ix0: link state changed to UP
Jun 24 18:57:52 knew kernel: vnet0.22: link state changed to DOWN
Jun 24 18:57:52 knew kernel: epair0b: link state changed to DOWN
Jun 24 18:57:52 knew kernel: ix0: link state changed to DOWN
Jun 24 18:57:52 knew kernel: ix0: link state changed to UP

I am not sure why this fails at all.

@skarekrow

This comment has been minimized.

Copy link
Member

commented Jun 24, 2019

Do you have defaultrouter set?

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Jun 24, 2019

Yes, it is set to my gateway.

$ iocage get defaultrouter toiler
10.55.0.1
@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Jun 24, 2019

I have no idea what this should look like:

$ ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:ca:b7:29:ce:00
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 1 priority 128 path cost 2000
	groups: bridge 
	nd6 options=1<PERFORMNUD>
@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Jun 24, 2019

I am reading https://iocage.readthedocs.io/en/latest/networking.html

The examples under VIMAGE/VNET have references to what is required in configuration files but no mention of how to invoke that configuration item from the command line.

bridge0 already existed when I got here. That is likely related to:

cloned_interfaces="lo1"

Let's try what I found at https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-bridging.html

$ sudo ifconfig bridge create
bridge1

$ ifconfig bridge1
bridge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:ca:b7:29:ce:01
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
	groups: bridge 
	nd6 options=1<PERFORMNUD>

$ sudo ifconfig bridge1 addm ix0
ifconfig: BRDGADD ix0: Device busy

Oh, I have to take ix0 down for this?

@skarekrow

This comment has been minimized.

Copy link
Member

commented Jun 24, 2019

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Jun 24, 2019

Oh. Is the following required?

# set up bridge interface for iocage
cloned_interfaces="bridge0"

# plumb interface em0 into bridge0
ifconfig_bridge0="addm em0 up"
ifconfig_em0="up"

Where em0 is my main interface? ix0 in my case. I ask because that's what the documentation says.

@skarekrow

This comment has been minimized.

Copy link
Member

commented Jun 24, 2019

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Jun 24, 2019

I have the jail up and dhcpd is running. From within the jail, I can reach the outside world (e.g. ping, host, etc), but the outside world cannot reach it.

I can ssh to the jail from the jail host, but not from other hosts.

There is no firewall on this host.

ifconfig on the host does not show 10.55.0.13 (the IP address I am trying to reach)

If I go into the jail via iocage console, I see the expected IP addresses in there.

I think I am missing a step.

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Jun 24, 2019

Got it.

$ sudo iocage set ip4_addr="vnet0|10.55.0.13/24" toiler
ip4_addr: vnet0|10.55.0.13 -> vnet0|10.55.0.13/24

Then:

[dan@knew:/iocage/jails/toiler] $ sudo iocage restart toiler
* Stopping toiler
  + Executing prestop OK
  + Stopping services OK
  + Tearing down VNET OK
  + Removing devfs_ruleset: 25 OK
  + Removing jail process OK
  + Executing poststop OK
* Starting toiler
  + Started OK
  + Using devfs_ruleset: 25
  + Configuring VNET OK
  + Using IP options: vnet
  + Starting services OK
  + Executing poststart OK
[dan@knew:/iocage/jails/toiler] $ 

NOTE the lack of UPS messages.

The missing mask is the key.

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Aug 7, 2019

This is py36-iocage-devel-1.0.0.20190801,1 (built from master on that date)

After rebooting this server today, no jails started up:

2019/08/07 16:12:12 (INFO) * Ruleset 7 does not exist, using defaults
2019/08/07 16:12:12 (ERROR) No bridge for interface ix0found in configuration.

Trying manually, to reproduce the message:

[dan@knew:~] $ sudo iocage start toiler
No bridge for interface ix0found in configuration.

I first searched the code to fix that whitespace typo, see #1015

Based on the previous conversation, I thought iocage would take care of the bridge, hence:

[dan@knew:~] $ grep bridge /etc/rc.conf
[dan@knew:~] $ 

So I did this:

[dan@knew:~] $ ifconfig bridge0
ifconfig: interface bridge0 does not exist
[dan@knew:~] $ sudo ifconfig bridge create
bridge0
[dan@knew:~] $ 

Reading above, I thought this would work:

[dan@knew:~] $ sudo ifconfig bridge0 addm ix0
[dan@knew:~] $ sudo iocage start toiler
No bridge for interface ix0found in configuration.
[dan@knew:~] $ ifconfig bridge0
bridge0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:ca:b7:29:ce:00
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
	member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 1 priority 128 path cost 2000
	groups: bridge 
	nd6 options=1<PERFORMNUD>
[dan@knew:~] $ 

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Aug 7, 2019

It looks like iocage start ALL does not honor boot=off. Shall I raise an issue?

[dan@knew:~] $ iocage get boot toiler
0
[dan@knew:~] $ sudo iocage start ALL
No bridge for interface ix0found in configuration.
[dan@knew:~] $ 

So much for not starting the jails one by one.

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Aug 7, 2019

OK, that's the other jails started, let's see if I can figure this problem out.

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Aug 7, 2019

I've been playing in the code.

get_interface_bridge_map() outputs: vnet0 bridge0

interface is: ix0

It seems like ix0 should be in the map too.

This seems OK though:

$ iocage get all toiler | grep vnet
interfaces:vnet0:bridge0
ip4_addr:vnet0|10.55.0.13/24
ip6_addr:vnet0|2001:470:8abf:7055:31da:7e45:0:53/64
vnet:1
vnet0_mac:02ff6023eef1 02ff6023eef2
vnet1_mac:none
vnet2_mac:none
vnet3_mac:none
vnet_default_interface:auto
vnet_interfaces:none

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Aug 8, 2019

See also #1016

That got my jail working again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.