bridge: remove default pvid for veth interface

[bephinix]

Problem

Using the bridge plugin will create a seperate veth-pair per pod with one end of the veth-pair connected to the specified/configured bridge.
If one uses VLANs with the bridge plugin and a pod is configured to use the bridge network with e.g. VLAN 123, the veth-pair endpoint connected to the bridge has VLAN 123 and 1 configured:

port  vlan-id
br0   1 Egress Untagged
      123 PVID Egress Untagged

Idea/Solution

VLAN 1 is the DefaultPVID for a bridge or connected device, thus it is automatically configured.
Whereas the bridge (which has to be created and maintained manually) can be modified to not accept VLAN 1 traffic (e.g. by using ip link ... vlan_default_pvid 0 or systemd option DefaultPVID=none), this should also be possible for the veth-pair endpoint.

As current configurations should not be affected, one should add an option (e.g. nodefaultpvid) which will trigger the removal of PVID 1:

if vlanID != 0 {
    err = netlink.BridgeVlanDel(hostVeth, uint16(1), true, true, false, true)
    if err != nil {
        return nil, nil, fmt.Errorf("failed to delete default vlan tag on interface %q: %v", hostIface.Name, err)
    }
    err = netlink.BridgeVlanAdd(hostVeth, uint16(vlanID), true, true, false, true)
    if err != nil {
        return nil, nil, fmt.Errorf("failed to setup vlan tag on interface %q: %v", hostIface.Name, err)
    }
}

The proposed modification will also work if one wants to use VLAN 1 for a pod.

(Ref.: https://github.com/containernetworking/plugins/blob/master/plugins/main/bridge/bridge.go#L366)

[dcbw]

Seems like it may deserve a knob to prevent the untagged vlan1 behavior. I'd think the default expectation is that if you ask for a specific VLAN, you get that and only that VLAN, not that plus other stuff.