bone

The "[B]ag [O]f [N]etgraph [E]xtensions" is a set of evolving netgraph(4) kernel modules and their utilities.

Due to how ports are built on FreeBSD, it makes more sense to combine what were two separate repos into one. I'll leave archives behind of the separate kmods and utilities repos. That allows making a default and "invariants" version for the kmods (for when your kernel is built with invariants, as is the default for CURRENT). And it means we have just one package for the utilities.

These require patches merged into FreeBSD15. While kernel patches are available for FreeBSD14, there is no plan to merge them (although some were like the -j option for ngctl(8) others have not). A table of required patches to src and their status in the process is still here.

rough edges.

But you are going to run into some rough edges here. First you must add this to your loader.conf(5) because without it your physical network interfaces will not have an ng_ether(4) to manipulate (so can't be joined to any other netgraph(4) node):

netgraph_load="YES"
ng_ether_load="YES"

That is well known.

Another rough edge when using netgraph(4) with jails is that default MACs given to ng_eiface(4) can collide with other FreeBSD systems on your network also creating ng_eiface(4) for any purpose. Especially if you have a naming convention as they hash the interface name!

So if the ng_eiface(4) is ultimately connected to a physical network you really do need to assign it a unique MAC to avoid collisions of autogenerated MACs.

The last rough edge is that netgraph(4) is traditionally shown sharing an ng_ether(4) by connecting both the upper and the lower hooks to an ng_bridge(4). For smoothe IPv4 and IPv6 operation I recommend instead just connecting the lower hook, and to setpromisc on the ng_ether(4) but do not configure it for networking. Instead attach an ng_eiface(4) to the same ng_bridge(4) and configure that for networking instead. This is rock solid and lets you configure both IPv4 and IPv6 for any ng_eiface(4) in a jail (see jeiface below) connected to that same bridge via ng_wormhole(4) (also below).

jeiface

Using vnet(9) with jails with netgraph(4) prior to these modules required using ifconfig(8) with vnet to "move" an interfact into the jail. This utility takes a different route and is just a shell script to create an ng_eiface(4) in a jail, name it, and optionally assign a MAC address.

This is how it might appear in jail.conf(5):

jeiface $name lan0

wormhole

jeiface made it a one-liner to create the interface for your jail, but as it is its not connected to anything.

The ng_wormhole(4) node allows you to connect any two netgraph(4) nodes that each reside in separate vnet(9) from each other (so jails). While you can (and I do) use this to simply attach an ng_eiface(4) created in a jail to an ng_bridge(4) on the system, it does not care what data it shovels back and forth.

But creation of wormholes and attaching both ends is problematic for scripting, so there is a utility, ngportal(8), to simplifying creation, naming, and connecting ng_wormholes(4).

Connecting an ng_bridge(4) named br0 on the system to an ng_eiface(4) in a jail named lan0 and naming the two wormhole nodes br0$name and lan0system after what they are connected with is now one line (jail.conf(5) format again):

ngportal :br0$name:br0:link $name:lan0system:lan0:ether

pcap

The ng_pcap(4) node is mostly for debugging. It expects either layer 2 or layer 3 nodes which you must configure after connecting to source<N> nodes. You inform ng_pcap(4) what the hook provides: ethernet frames, IPv4, or IPv6. In the case of IPv4 and IPv6, fake ethernet frames will be added. This allows you to trace both layers at the same time. I don't know of any use for this besides debugging netgraph(4) nodes.

This node benefits from a standalone utility as well, ngpcap(8) to connect and pipe output to tcpdump(1):

ngpcap inet6:tee0:right2left ether:tee1:left2right | tcpdump -r -

To be of any use at all you have to plan out and insert some ng_tee(4) for the ng_pcap(4) to connect to. But its helped me while developing other netgraph(4) nodes. The man page has a detailed example.

ula4tag

This is an oddball for sure but I have to use it because my ISP only gives me a /64 prefix for IPv6. That is a massive range, but Android will not do DHCPv6 (probably a good thing IMO) and only does SLAAC. That takes up my whole GUA address space. And I want separate networks for my LAN and WiFi. That is where ng_ula4tag(4) comes in, it allows you to VLAN tag ULA and IPv4 traffic coming in while leaving GUA traffic untagged.

The idea is you connect ng_ether(4) to this, configure tags, and connect the ng_ula4tag(4) to an ng_bridge(4). Additionally on the bridge you need an ng_vlan(4) to separate things out. The man page has a detailed description of how to configure.

This is not ephemeral like ng_wormhole(4) or ng_pcap(4) and doesn't require any additional utility. But it is well suited for using the rc(8) command provided.

This is functionality I'm not aware of existing elsewhere, so there is a good chance its a bad idea (but hey its working for me). I use it because my ISP insists on a worse idea: giving out only /64 prefix.

Additional

Additional kernel modules are planned but not yet ready.

netgraph(4) rc(8) script

Don't get excited, this isn't the perfect netgraph(4) rc(8) script you are hoping for. In fact its a cop-out.

I've come to believe that even attempting such a thing is a fools errand. But who knows, I am not the worlds best sh(1) practitioner.

This is just going to pass a file to ngctl(8) and after that make sure netif interfaces (so ng_ether(4), ng_eiface(4), ng_iface(4) for example), get renamed to match the netgraph(4) node name (which you should have set in your config file with name command). So not much taken care of for you.

But by renaming netif interfaces to something you expect you can now finish all config using the usual ifconfig_<ifname>=... stanzas of rc.conf(5) or any other way you prefer to configure your network (I like net/dhcpcd).

That, surprisingly, is really all that is necessary to configure whatever you can dream up to put in an ngctl(8) command file.

Example setup for using wormholes with jails

After installing both bone-kmods (or bone-kmods@invariants) and bone-utils here is some sample config ideas. You will have to change interface names and ULA addresses to meet your needs, these are the values I use.

First go ahead and get netgraph loaded in loader.conf(5), edit your /boot/loader.conf and add:

netgraph_load="YES"
ng_ether_load="YES"

If ngctl ls does not show your network interfaces you have to reboot. Sorry.

Now you need /usr/loca/etc/ng/ngctl.conf. I'm assuming you want a private network for your jails as well as the ability to share your main NIC, which in this example is re0. So the bridge for your local area network will be br0 and the private bridge for jails will be br1 we will rename re0 to br0re0:

# you get to have comments in this file
# Start with bridges (create br0 and br1):
mkpeer .: bridge b link0
name .:b br0
msg br0: setpersistent
disconnect .: b
mkpeer .: bridge b link0
name .:b br1
msg br1: setpersistent
disconnect .: b

# connect re0 to our bridge and rename to indicate it should not be configured.
# you'll see dhcpcd ignores all interfaces named `br*` later.
name re0: br0re0
msg br0re0: setpromisc 1
connect br0re0: br0: lower link0

# create lan0 on br0 for system to use for networking
mkpeer br0: eiface link1 ether
name br0:link1 lan0
# basically everything connected to br0 needs a unique MAC set
msg lan0: set 56:9C:FC:00:00:15

# create jail0, this is local to machine and auto-generated MAC is fine
mkpeer br1: eiface link1 ether
name br1:link1 jail0

Normally that would leave you wondering how to find lan0 and jail0 since they started with ifconfig(8) names ngeth0 and so on. But our rc(8) script has ensured ifconfig(8) sees the same names as ngctl(8). That is now really valuable for rc.conf(5). We need to put our renamed br0re0 (see above that was re0) in the correct mode:

netgraph_enable="YES"
netgraph_config="ngctl.conf"
ifconfig_br0re0="-lro -tso4 -tso6 -rxcsum6 -txcsum6 -rxcsum -txcsum promisc up"

# The private network needs to have rtadvd running on it. Sooo much nicer than
# having workstation run a DHCP server.
rtadvd_enable="YES"
rtadvd_interfaces="jail0"

You are sharing the interface remember which is why you need some extra config for br0re0 above. You can now manually assign config lan0 and jail0:

ifconfig_lan0="inet ..." # however you would but s/re0/lan0/g
ifconfig_lan0_ipv6="inet6 ..."
ifconfig_jail0="inet ..." # this private jails comms
ifconfig_jail0_ipv6="inet6 ..."

I don't do that, I think net/dhcpcd is the best way of doing this here, and I use a minimal config for that in /usr/local/etc/dhcpcd.conf:

controlgroup wheel
clientid

vendorclassid

option domain_name_servers, domain_name, domain_search
option classless_static_routes
option interface_mtu
option rapid_commit

# By default we try to get v4 and v6, soon only v6...
ipv4
ipv6
ipv6rs
dhcp
dhcp6

# only handle these
denyinterfaces lo* br*
allowinterfaces lan0 jail0

# there has been a fix and you should be able to set "temporary" now
# before fix it would never delete any...
interface lan0
  slaac private

# For the jails they are on a different ULA and EUI-64 is desirable because a
# predictable MAC <-> IPv6 address is one less thing to sort out when putting
# a jail into DNS.
interface jail0
  persistent
  noipv4
  noipv6rs
  static ip6_address=fdc5:972:cd8c::1/64

I know that seems excessive. But this config will use either SLAAC or DHCPv6 for IPv6 and DHCP for IPv4. That covers all my bases as I use DHCP/DHCPv6 with net/kea for some networks and rtadvd(8) for others. It also shows how you might set a static address if you need that. I set the one static here so jails can find packages when I install them. Also notice how it ignores "br*" so br0re0 will not get messed up by net/dhcpcd.

And we need to actually set up rtadvd.conf(5) in /etc/rtadvd.conf:

# A word of extreme caution, the backslash character continues a comment as well
# as any configuration! That bit me for hours as I was trying different ways to
# set pinfoflags by commenting out lines. Be very careful, the way this file is
# written most configuration lines end with backslash.

default:\
        :maxinterval#120:mininterval#30:

# The jail0 is the only interface we are advertising on. This is a closed loop
# since my workstation does NOT route. Besides, it only has ULA addresses that
# won't go anywhere anyway. The purpose is for intra-jail traffic.
#
# This needs to be advertised as lower priority. Basically you use it when the
# route matches exactly. `jail0` has its address set statically in either the
# system rc.conf or in dhcpcd.conf.

jail0:\
        :raflags="l":rltime#0:tc=default:

Make sure lan0 is working and you can ping and ping6 out. So far we haven't used the new utiilities or modules. That is going to change with a jail.

For jails I like to have the following set up in the main jail.conf(5) file /etc/jail.conf:

# define interfaces for our jails. each jail can have the same interface names
# but change `lan0` to `lan0$name` if you don't like that.
# Also don't forget that lan0 needs a MAC added.
$lan0="jeiface $name lan0";
$jail0="jeiface $name jail0";

# create wormholes, assume there is a br[0|1] on the system and a lan0|jail0
# interface in the jail. Using `link` instead of `linkX` is what stable/14 is
# missing. That is vital so you don't have to track bridge links!
$wh0="ngportal :br0$name:br0:link $name:lan0system:lan0:ether";
$wh1="ngportal :br1$name:br1:link $name:jail0system:jail0:ether";

# NOTE: by naming the wormholes there is a race when restarting the same jail
#       after shutting it down. The jail vnet(9) is not guaranteed to be cleaned
#       up immediately. It can take some time. So we need to try to destroy the
#       wormhole manually. That way a restart won't have a name conflict.
$wh0end="ngctl shutdown br0$name: 2>/dev/null || :";
$wh1end="ngctl shutdown br1$name: 2>/dev/null || :";

That can be a bit clunky but the reason is in comments. When jails shutdown their vnet(9) lingers for a while (I believe to prevent a race). That can cause a problem if you start a jail, stop it, then restart it right away. Why? Because until the vnet(9) is gone you will collide on wormhole names!

There is no real reason to have wormhole names though. I just point it out as if you give wormholes names you have to be aware that they don't instantly go away at jail shutdown. If left un-named and you don't shut them down the wormhole is automatically closed when the vnet(9) is cleaned up.

Ok lets look at how that can be used in an individual jail conf:

dev15 {
        vnet;
        exec.created += "$lan0 56:9C:FC:10:02:15";
        exec.created += "$jail0";
        exec.created += "$wh0";
        exec.created += "$wh1";
        # ... whatever you usually do
        exec.start = "/bin/sh /etc/rc";
        exec.stop = "/bin/sh /etc/rc.shutdown jail";
        exec.poststop += "$wh0end"; # not needed if you don't name wormholes
        exec.poststop += "$wh1end";
}

Remember you MUST give all your $lan a MAC address to avoid collisions. I know you didn't even get to use the portal gun directly its in a conf file, this is a bummer man!