Multicast traffic cannot pass through bridge of virt-launcher pod.

[kononovn]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind enhancement

What happened:
Multicast traffic can not pass from/to VM via virt-launcher pod network.
What you expected to happen:
All kind of traffic can pass transparently from/to VM via virt-launcher pod network
How to reproduce it (as minimally and precisely as possible):
Create 2 VM and configure direct L2 link between them, use connection type 'bridge' and configure some multicast protocol(LLDP for example). You will see that traffic goes out of VM but doesn't pass via virt-launcher pod.
Anything else we need to know?:
Possible solution is to add following flag into virt-launcher pod bridge
echo 65528 > /sys/class/net/virbr0/bridge/group_fwd_mask
Environment:

• KubeVirt version (use virtctl version):
  Client Version: version.Info{GitVersion:"v0.16.0", GitCommit:"06d91198bededc7a9353ac774221c83580ee3373", GitTreeState:"clean", BuildDate:"2019-04-05T15:24:49Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{GitVersion:"v0.16.0-gb37c22bbf", GitCommit:"$Format:%H$", GitTreeState:"", BuildDate:"2019-04-18T08:47:30Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
• Kubernetes version (use kubectl version):
  [root@zeus06 cnv-tests]# kubectl version
  Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0+d4cacc0", GitCommit:"d4cacc0", GitTreeState:"clean", BuildDate:"2018-12-06T18:30:39Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.4+0ba401e", GitCommit:"0ba401e", GitTreeState:"clean", BuildDate:"2019-03-31T22:28:12Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
• VM or VMI specifications:
  VM-A

apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
metadata:
  name: vma
  namespace: myproject
spec:
  running: false
  template:
    spec:
      domain:
        devices:
          disks:
            - name: containerdisk
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
          interfaces:
            - name: default
              bridge: {}
            - name: br1
              bridge: {}
        resources:
          requests:
            memory: 1G
        cpu:
          cores: 2
      volumes:
        - name: containerdisk
          containerDisk:
            image: kubevirt/fedora30-cloud-container-disk-demo:latest
        - name: cloudinitdisk
          cloudInitNoCloud:
            userData: |-
              #cloud-config
              password: fedora
              chpasswd: { expire: False }
      networks:
        - name: default
          pod: {}
        - multus:
            networkName: br1
          name: br1

VM-B

apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
metadata:
  name: vmb
  namespace: myproject
spec:
  running: false
  template:
    spec:
      domain:
        devices:
          disks:
            - name: containerdisk
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
          interfaces:
            - name: default
              bridge: {}
            - name: br1
              bridge: {}
        resources:
          requests:
            memory: 1G
        cpu:
          cores: 2
      volumes:
        - name: containerdisk
          containerDisk:
            image: kubevirt/fedora30-cloud-container-disk-demo:latest
        - name: cloudinitdisk
          cloudInitNoCloud:
            userData: |-
              #cloud-config
              password: fedora
              chpasswd: { expire: False }
      networks:
        - name: default
          pod: {}
        - multus:
            networkName: br1
          name: br1

• Cloud provider or hardware configuration:
  Hardware
• OS (e.g. from /etc/os-release):
  cat /etc/os-release
  NAME=Fedora
  VERSION="29 (Twenty Nine)"
  ID=fedora
  VERSION_ID=29
  VERSION_CODENAME=""
  PLATFORM_ID="platform:f29"
  PRETTY_NAME="Fedora 29 (Twenty Nine)"
  ANSI_COLOR="0;34"
  LOGO=fedora-logo-icon
  CPE_NAME="cpe:/o:fedoraproject:fedora:29"
  HOME_URL="https://fedoraproject.org/"
  DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f29/system-administrators-guide/"
  SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
  BUG_REPORT_URL="https://bugzilla.redhat.com/"
  REDHAT_BUGZILLA_PRODUCT="Fedora"
  REDHAT_BUGZILLA_PRODUCT_VERSION=29
  REDHAT_SUPPORT_PRODUCT="Fedora"
  REDHAT_SUPPORT_PRODUCT_VERSION=29
  PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
• Kernel (e.g. uname -a):
  5.0.6-200.fc29.x86_64 Add travis support #1 SMP Wed Apr 3 15:09:51 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
• Others:

[fabiand]

@SchSeba what do we want to do? Permit by default or make it a flag on the api?

[rmohr]

Sounds like the pod network let's it through, so should we. Right?

[fabiand]

I think this depends on the used network provider. But whenever multicast is getting to the pod network (inside the pod), then it means that the provider also supported it, thus sounds reasonable to also pass it to VM.

[SchSeba]

This issue is related to Linux bridges silently drop LLDP messages (sent to the LLDP_Multicast address 01-80-C2-00-00-0E) and other control frames in the 01-80-C2-00-00-xx range.

https://thenetworkway.wordpress.com/2016/01/04/lldp-traffic-and-linux-bridges/

[gageorsburn]

This is an issue we're been fighting with for awhile but I haven't had the time to really implement a solution. @renner-osi Did some base work for a passthrough interface type where we create a device and pass it through. Last I heard we got this working however we are lacking some device plugin code to properly handle creating and managing the device. I'll see what I can scrounge up and see if I can put together a proper PR to address this. Unless @SchSeba you have another plan in mind?

Correction: We have it working with the virt-launcher container being privileged, which we need to come up with a workaround for.

[SchSeba]

Update: I take a look on this and the problem is because the pod is not privileged we can't change the file group forward mask for the bridge.

error output:

{"component":"virt-launcher","level":"fatal","msg":"failed to prepared pod networking","pos":"podinterface.go:88","reason":"open /sys/class/net/k6t-eth0/bridge/group_fwd_mask: read-only file system","timestamp":"2019-06-10T12:18:43.006218Z"}
panic: open /sys/class/net/k6t-eth0/bridge/group_fwd_mask: read-only file system

goroutine 25 [running]:
kubevirt.io/kubevirt/pkg/virt-launcher/virtwrap/network.(*PodInterface).Plug(0x22576b0, 0xc000c48000, 0xc000b86240, 0xc00049bb60, 0xc000318000, 0x14ade7d, 0x4, 0x10, 0x12c8fe0)
	pkg/virt-launcher/virtwrap/network/podinterface.go:89 +0x536
kubevirt.io/kubevirt/pkg/virt-launcher/virtwrap/network.SetupNetworkInterfaces(0xc000c48000, 0xc000318000, 0x0, 0x7)
	pkg/virt-launcher/virtwrap/network/network.go:83 +0x520
kubevirt.io/kubevirt/pkg/virt-launcher/virtwrap.(*LibvirtDomainManager).preStartHook(0xc0000806c0, 0xc000c48000, 0xc000318000, 0x16669c0, 0x0, 0x163a5e0)
	pkg/virt-launcher/virtwrap/manager.go:728 +0x3ad
kubevirt.io/kubevirt/pkg/virt-launcher/virtwrap.(*LibvirtDomainManager).SyncVMI(0xc0000806c0, 0xc000c48000, 0xc0003c8500, 0x0, 0x0, 0x0)
	pkg/virt-launcher/virtwrap/manager.go:881 +0x5e2
kubevirt.io/kubevirt/pkg/virt-launcher/virtwrap/cmd-server.(*Launcher).SyncVirtualMachine(0xc000623fa0, 0x16541c0, 0xc00049a780, 0xc00049a7b0, 0xc000623fa0, 0xc00049a6f0, 0x132f6e0)
	pkg/virt-launcher/virtwrap/cmd-server/server.go:159 +0x81
kubevirt.io/kubevirt/pkg/handler-launcher-com/cmd/v1._Cmd_SyncVirtualMachine_Handler(0x13f6480, 0xc000623fa0, 0x16541c0, 0xc00049a780, 0xc000218cd0, 0x0, 0x0, 0x0, 0xc00034b800, 0x7e7)
	bazel-out/k8-fastbuild/bin/pkg/handler-launcher-com/cmd/v1/linux_amd64_stripped/kubevirt_cmd_go_proto%/kubevirt.io/kubevirt/pkg/handler-launcher-com/cmd/v1/cmd.pb.go:515 +0x23e
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00046ad80, 0x165fd60, 0xc000483b00, 0xc000b8a200, 0xc00002a9f0, 0x2193600, 0x0, 0x0, 0x0)
	external/org_golang_google_grpc/server.go:971 +0x4a2
google.golang.org/grpc.(*Server).handleStream(0xc00046ad80, 0x165fd60, 0xc000483b00, 0xc000b8a200, 0x0)
	external/org_golang_google_grpc/server.go:1250 +0xd61
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0006461f0, 0xc00046ad80, 0x165fd60, 0xc000483b00, 0xc000b8a200)
	external/org_golang_google_grpc/server.go:690 +0x9f
created by google.golang.org/grpc.(*Server).serveStreams.func1
	external/org_golang_google_grpc/server.go:688 +0xa1
{"component":"virt-launcher","level":"error","msg":"dirty virt-launcher shutdown","pos":"virt-launcher.go:558","reason":"exit status 2","timestamp":"2019-06-10T12:18:43.028453Z"}

I think we should consider to back into @rmohr Idea to move the network preparation into the virt-handler.
PR: #1404

@vladikr @booxter @phoracek @dankenigsberg what you think about that?

[gageorsburn]

Note that with group forward mask its not going to be a perfect passthrough and allow all kinds of traffic unless

A slight catch is that in the default kernel distribution, bitmask values for the first three MAC addresses (-00, -01 and -02) are restricted, meaning we still cannot use this trick to enable STP and LACP protocols in our labs

STP I don't think anyone could care less about but LACP is how you do bonding.

[SchSeba]

LACP will not work if we add the 65528 to the bridge config?

[gageorsburn]

Not without patching the kernel.

[EdDev]

This issue subject about multicast and the content about LLDP is confusing.
LLDP and other similar protocols (e.g STP) use reserved mac addresses which are required by the standards not to be forwarded beyond a bridge or a local LAN (the same exist at the L3 level for protocols like VRRP and alike).
Regular multicast data forwarding is one thing, control frames like the ones LLDP is using is a different story.

I would not recommend forwarding LLDP frames over bridges, it is explicitly and intentionally blocked by the standard.
Quoting from the 802.1AD standard notes about the MAC multicast address:

This address was selected in order to make it possible to transmit an LLDP frame containing information specific to a single individual LAN, and for that information not to be propagated further than the extent of that individual LAN, to avoid the meaning of that information being misinterpreted. For example, a TLV containing the auto-negotiation status of an IEEE 802.3™ LAN would be open to misinterpretation if it were to be propagated via a Bridge onto a second IEEE 802.3 LAN, and would be meaningless if propagated onto a LAN based on a different MAC technology. This is the “LLDP_Multicast address” that was used in IEEE Std 802.1AB-2005. This address was
erroneously omitted from the table of S-VLAN component Reserved addresses added to IEEE Std 802.1Q by IEEEStd 802.1ad-2005, but has been added to that table by IEEE Std 802.1aj™-2009 [B1] and is included in subsequent revisions of IEEE Std 802.1Q.

There are cases where exceptions may exists, but even there the risk something wrong will be interpreted exists. One would expect LLDP to work with SRIOV VF, however, this is also questionable as an SRIOV enabled device uses an internal bridge for its operation and it may or may not forward such control frames. If it does, logic on what is supported probably exists (for example, sending LLDP frames from the VF out sounds unreasonable).

[gageorsburn]

I agree that a linux bridge is not the proper transport for this type of traffic.

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale