Guest ingress packets broken when using Bridged interface with VLAN interface

[jkkataja]

Describe the issue
When creating guest and assigning its network mode to bridged that is attached to VLAN tagged interface, the ingress packets towards the guest contain additional bytes before the IP packet header. I suspect this is a bug on Apple's side, but reporting here, in case there is something you can amend from UTM side.

In my example I will be using vlan3 interface, same problem happens regardless of having one or multiple vlan interfaces.

Steps to reproduce:

1. Create VLAN interface from Network settings and assign it to some physical IF (in my case, VLAN 130 to bridge vlan3 using Physical interface USB-C to LAN adapter)
2. Create guest and assign network mode to 'Bridged' - Bridged Interface 'vlan3'
3. Start the guest and wait until it attempts to obtain network information (doesn't matter if static or DHCP) and see that packets are broken

I have three packet captures, can provide if needed:

• TCPDUMP from en4 (the physical interface) - scope of packets limited to those that are VLAN tagged with VLAN-ID 130, packets are all ok
• TCPDUMP from vlan3 (the vlan interface) - packets are all ok
• TCPDUMP from bridge100 - DHCP Reply packets are broken, before IP header there are the additional "00 82 08 00" bytes

Configuration

• UTM Version: 3.1.5 (53)
• OS Version: 12.3.1
• Apple Silicon

Ifconfig of the auto-created bridge100:
bridge100: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500 options=3<RXCSUM,TXCSUM> ether be:d0:74:24:13:64 Configuration: id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0 maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200 root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0 ipfilter disabled flags 0x0 member: vlan3 flags=3<LEARNING,DISCOVER> ifmaxaddr 0 port 30 priority 0 path cost 0 member: vmenet0 flags=3<LEARNING,DISCOVER> ifmaxaddr 0 port 25 priority 0 path cost 0 media: autoselect status: active

ifconfig of the vlan3 interface:
vlan3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500 options=6063<RXCSUM,TXCSUM,TSO4,TSO6,PARTIAL_CSUM,ZEROINVERT_CSUM> ether 3c:18:a0:51:01:4c inet6 fe80::c10:aa25:72d2:eacf%vlan3 prefixlen 64 secured scopeid 0x1e nd6 options=201<PERFORMNUD,DAD> vlan: 130 parent interface: en4 media: autoselect (1000baseT <full-duplex>) status: active

Attaching screenshot of two wireshark windows (left vlan3 interface, ok, right bridge100 interface, broken packet)

[zhangyoufu]

I ran into similar issue on macOS 13.0 Beta (22A5342f) with QEMU 7.1.0. (I'm not a UTM user)
I bridged Mac Mini (M1) wired Ethernet into VM.
The egress direction (guest->host) keeps VLAN tag as intended.
While the ingress direction (host->guest) always drops VLAN tag. (tcpdump inside the VM cannot see VLAN tag)
I didn't observe any broken packet though.

I submitted a bug report to Apple via Feedback Assistant FB11524674, and opened a Technical Support Incident via Apple Developer Portal.
Apple responds my TSI with "There is no workaround DTS can provide for Feedback ID #FB11524674; it is still under investigation."

My investigation:
The receiving data flow: en0->bridge100->vmenet0(IOUserEthernetController)->vmnet.framework->qemu->guest VM
Sniffing packet on vmenet0 shows that the packet has complete VLAN tag.
Both vmnet.framework (part of MobileInternetSharing project, /usr/libexec/InternetSharing) and qemu does not change packet content and their implementation is rather straight forward.