xde: mac_unicast_add failed for vnic on etherstub

[jmpesp]

My test sleds only have one physical 10G link to the fake scrimlet, but xde requires setting two devices as the underlay devices. My work around for this is to use the 10G link as the first underlay devices, and a vnic on an otherwise unconnected etherstub as the second underlay device:

pfexec dladm create-etherstub -t unattachedstub0
pfexec dladm create-vnic -t -l unattachedstub0 net1

This way, there's no ddm advertisement confusion.

This method didn't work after upgrading to OPTE 0.23.170. I saw EINVAL:

16:07:47.228Z WARN SledAgent (sprockets (BootstrapAgent)): Sled agent request failed
    err = Error managing sled agent: Could not start sled agent server: Error managing guest networking: Failure interacting with the OPTE ioctl(2) interface: command SetXdeUnderlay failed: System { errno: 14, msg: "mac_unicast_add failed for net1: 22" }

If I instead use a simnet link that's not connected to anything, or the machine's 1G link (that does also not participate in any ddm related activity), I do not see EINVAL.

[jmpesp]

Based on some dtracing, I think the EINVAL is coming from mac_client_datapath_setup:

    /*
     * A unicast flow already exists for that MAC client
     * so this flow must be the same MAC address but with
     * a different VID. It has been checked by
     * mac_addr_in_use().
     *
     * We will use the SRS etc. from the initial
     * mci_flent. We don't need to create a kstat for
     * this, as except for the fdesc, everything will be
     * used from the first flent.
     * 
     * The only time we should see multiple flents on the
     * same MAC client is on the sun4v vsw. If we removed
     * that code we should be able to remove the entire
     * notion of multiple flents on a MAC client (this
     * doesn't affect sub/user flows because they have
     * their own list unrelated to mci_flent_list).
     */
    if (bcmp(mac_addr, map->ma_addr, map->ma_len) != 0) {
      err = EINVAL;
      goto bail; 
    }

Using

fbt::mac_client_datapath_setup:entry {
  self->diag = (mac_client_impl_t *)(args[0]);
  self->diag2 = (uint8_t *)(args[2]);
}

fbt::mac_client_datapath_setup:return /arg1 == 22 && self->diag/ {
  mcip = (mac_client_impl_t *)self->diag;
  mip = (mac_impl_t *)(mcip->mci_mip);
  map = (mac_address_t *)(mcip->mci_unicast);
  print(*map);

  mac = self->diag2;
  printf("%0x:%0x:%0x:%0x:%0x:%0x\n", mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
}

fbt::mac_client_datapath_setup:return /self->diag/ { self->diag = NULL; }

shows that the requested mac address does not match the existing mac address:

CPU     ID                    FUNCTION:NAME
 26  38217 mac_client_datapath_setup:return mac_address_t {
    mac_address_type_t ma_type = MAC_ADDRESS_TYPE_UNICAST_PROMISC
    int ma_nusers = 0x1
    struct mac_address_s *ma_next = 0xfffffeb1e18c67d8
    uint8_t [20] ma_addr = [ 0x2, 0x8, 0x20, 0xc7, 0x3, 0x53, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]
    size_t ma_len = 0x6
    mac_vlan_t *ma_vlans = 0
    boolean_t ma_untagged = B_TRUE
    mac_group_t *ma_group = 0
    mac_impl_t *ma_mip = 0xfffffeb1edd62040
}a8:40:25:ff:0:0

where the net1 vnic's mac is 2:8:20:c7:3:53