Can't get DHCP IP when using switch VLAN

[Drizzt321]

I'm trying to dual-home HomeAssistant I have running as a VM. I'm running 12.2-RELEASE with the latest vm-bhyve.

Router is running opnSense, switches are Ubiquiti 8-port managed switches. I have my laptop hardwired into one of the switch ports with the VLAN and it's getting a DHCP address just fine.

When I try using tcpdump -i em0.30 port 67 or port 68 -e -n -vv

I get the following which appears to be from the VM, and the correct interface MAC, however I don't see any response/reply and the VM still doesn't show the IP on that interface.

17:53:20.218764 58:9c:fc:5f:71:50 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 334: (tos 0xc0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 320)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 58:9c:fc:5f:71:50, length 292, xid 0x887c9c0f, secs 15, Flags [none] (0x0000)
          Client-Ethernet-Address 58:9c:fc:5f:71:50
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Discover
            Client-ID Option 61, length 7: ether 58:9c:fc:5f:71:50
            Parameter-Request Option 55, length 18:
              Subnet-Mask, Default-Gateway, Hostname, Domain-Name
              Domain-Name-Server, Time-Zone, MTU, BR
              Classless-Static-Route, Static-Route, YD, YS
              NTP, Server-ID, Option 119, Classless-Static-Route-Microsoft
              Option 252, RP
            MSZ Option 57, length 2: 576
            Hostname Option 12, length 13: "homeassistant"

When in HA I statically set a IP address on the interface, I can ping back and forth just fine to the router and my laptop, when it's on the VLAN.

Here's my configuration. Anything further I can check? Any alternatives to try? I've done a bunch of searches, so far haven't seen anything that's worked or seems to indicate my problem.

Switches

# vm switch list
NAME    TYPE      IFACE      ADDRESS  PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  -        no       -    -     em0
ha-iot  standard  vm-ha-iot  -        no       -    30    em0

VM configuration

loader="uefi"
graphics="yes"
xhci_mouse="yes"
graphics_listen="192.168.2.5"
graphics_port="5900"
graphics_wait="no"
graphics_res="800x600"

cpu="4"
memory="4GB"
network0_type="virtio-net"
network0_switch="public"
disk0_type="ahci-hd"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="9edcc9da-1b35-11eb-8576-0015170027d2"
network0_mac="58:9c:fc:06:f4:a0"

network1_type="virtio-net"
network1_switch="ha-iot"
network1_mac="58:9c:fc:5f:71:50"

# ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether 00:15:17:00:27:d2
        inet 192.168.2.5 netmask 0xffffff00 broadcast 192.168.2.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 70:85:c2:fc:1f:58
        media: Ethernet autoselect (none)
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        groups: lo
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 2e:36:b0:9f:36:7f
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 10 priority 128 path cost 2000000
        member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000000
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge vm-switch viid-4c918@
        nd6 options=1<PERFORMNUD>
vm-ha-iot: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 6e:19:b8:bb:03:25
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 11 priority 128 path cost 2000000
        member: em0.30 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 55
        groups: bridge vm-switch viid-f27b7@
        nd6 options=1<PERFORMNUD>
em0.30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vm-vlan-ha-iot-em0.30
        options=1<RXCSUM>
        ether 00:15:17:00:27:d2
        inet6 fe80::215:17ff:fe00:27d2%em0.30 prefixlen 64 scopeid 0x7
        groups: vlan vm-vlan viid-ccc4e@
        vlan: 30 vlanpcp: 0 parent interface: em0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-unifi-controller-0-public
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:ff:87
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 2153
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-pihole-0-public
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:ff:a9
        inet6 fe80::5a9c:fcff:fe10:ffa9%tap1 prefixlen 64 tentative scopeid 0x9
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 2454
tap2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-home-assistant-0-public
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:f2:21
        inet6 fe80::5a9c:fcff:fe10:f221%tap2 prefixlen 64 tentative scopeid 0xa
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 2385
tap3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-home-assistant-1-ha-iot
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:60:08
        inet6 fe80::5a9c:fcff:fe10:6008%tap3 prefixlen 64 scopeid 0xb
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 2385

EDIT: Upon further investigation from the router, using tcpdump, I'm seeing that the response IS being sent, or at least attempting to be.

18:05:57.580603 40:62:31:12:6b:95 > 58:9c:fc:5f:71:50, ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    10.30.0.1.67 > 10.30.122.30.68: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x89df6bca, secs 9, Flags [none] (0x0000)
          Your-IP 10.30.122.30
          Client-Ethernet-Address 58:9c:fc:5f:71:50
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Offer
            Server-ID Option 54, length 4: 10.30.0.1
            Lease-Time Option 51, length 4: 7200
            Subnet-Mask Option 1, length 4: 255.255.0.0
            Default-Gateway Option 3, length 4: 10.30.0.1
            Domain-Name Option 15, length 20: "home.darkobjects.net"
            Domain-Name-Server Option 6, length 4: 10.30.0.1

EDIT2: I don't think I have any firewall issues, my current firewall on the host machine

# ipfw list
00100 allow ip from any to any via lo0
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
00400 deny ip from any to ::1
00500 deny ip from ::1 to any
00600 allow ipv6-icmp from :: to ff02::/16
00700 allow ipv6-icmp from fe80::/10 to fe80::/10
00800 allow ipv6-icmp from fe80::/10 to ff02::/16
00900 allow ipv6-icmp from any to any icmp6types 1
01000 allow ipv6-icmp from any to any icmp6types 2,135,136
65000 allow ip from any to any
65535 deny ip from any to any

[BlueSpaceCanary]

I'm having similar issues. After some digging through the forum, I think it's down to the untagged bridge somehow interfering with ARP traffic on the tagged bridge: https://forums.freebsd.org/threads/bridge-epair-not-passing-through-tagged-vlan-traffic-between-host-and-vnet-jail.71646/#post-437147

A quick test I did:

1. ping the VM IP from my laptop ping is failing
2. switch off the VNET jails I had on my untagged bridge0 and ifconfig bridge0 destroy ping begins succeeding
3. restart the VNET jails, recreating the untagged bridge0 ping begins failing again

[Drizzt321]

@BlueSpaceCanary what's your interface configurations? Are you talking about a bridge on the untagged interface, and a 2nd bridge on the tagged interface, and when you turn off the bridge on the untagged interface the tagged bridge starts working?

[Drizzt321]

@BlueSpaceCanary so I think I reproduced what you're seeing. Bridge on igb0 and igb0.30, the VM doesn't connect with it. If I remove bridge on igb0, suddenly it works. Put the bridge back on igb0, and stops working.

[BlueSpaceCanary]

@Drizzt321 Yeah, that's the exact setup I had, one on igb0 and one on igb0.100

[Drizzt321]

@BlueSpaceCanary as per https://genneko.github.io/playing-with-bsd/networking/freebsd-vlan/#do-not-bridge-the-parent-interface-of-the-vlans, looks like indeed, there's just problems running a bridge on both untagged (default VLAN, id 1) interface as well as on a VLAN interface. Might just have to re-architect my network somewhat, or at least put my other VMs on a VLAN that most of the rest of my stuff is on as well. This is annoying.

[joeynwz]

This should be in bold right atop of the networking page of the wiki and in the man page.
Maybe print it 10 times on the console every time you create a switch...
All kidding aside, I just spent days troubleshooting this on and off, and without reading this I don't think I every would have suspected bridging the untagged interface was breaking the tagged ones.
I made the port on the cisco switch tagged only now, and created all individual vlans interfaces off off lagg0.
Finally everything is working as it should

And I don't care this is from 2020 =), it is still very relevant on 13.1.

[Drizzt321]

Unfortunate to see it's still an issue in FreeBSD, but happy you found this issue and got things working.

[nbari]

I am having the same issue, any workaround to have tagged and untagged bridges using the same interface?