direction decision using eni mac/ip not the vni alone

[mgheorghe]

we have the ENI (that ENI has an ENI_IP, an ENI_MAC, and some tunnel IDs)

now this IP/MAC can be transported over vxlan, over nvgre, over ipsec or something else.

now if the packet has SRC_IP == ENI_IP and SCR_MAC == ENI_MAC and tunnel_id that is an outbound packet
now if the packet has DST_IP == ENI_IP and DST_MAC == ENI_MAC and tunnel_ids that is an inbound packet

i see a few advantages to this approach:

1. removes the dependency on the tunneling protocol details (by removing vni as dependency and have it depend on a eni tunnel id irrespective of what protocol is choosing for tunneling)
2. gives the flexibility in case of vxlan to be able to send and receive the traffic on any tunnel with any vni. (a tunnel is uniquely identified by the 2 vtep ips and by the vni, so having 2 tunnels with same vni will be possible)

[KrisNey-MSFT]

Mircea to send packet captures and diagrams as this could be a bug possibly. (All packets were being tagged...)

Discussion from DASH Comm Meeting:
If we have a VXLan tunnel using VNI 3 (for example), and another one is using 3 as well, b/c direction is based on VNI, the pipeline will become confused.
Suggesting here to include IP and MAC of the ENI to help to uniquely define the ENI, AND a Tunnel-ID or Customer-ID or something more generic?
Or other tunneling protocols such as IPSec, NVGre, as a Protocol-Type?

Guohan: we use VNI + MAC address
Prince: host -> Appliance is where the special VNI is used.
Marian: VNI = Virtual Network Identifier, not a Tunnel Identifier. VNI is part of the Underlay allocated by the DataCenter, the Customer does not see this. According to DASH, MAC is not controlled by customer.

[mgheorghe]

SAI config showing multiple ENIs each using a different VNI for inbound and outbound.
DASH config does not allow this flexibility.
config.sai-baby-hero-out.json.zip

taking it further beyond what DASH SAI allows bellow config is a valid VxLAN scenario that allows the reuse of the same VNI over multiple tunnels.

vxlan RFC https://datatracker.ietf.org/doc/rfc7348/
2 ENIs can have same MAC "The VNI identifies the scope of the inner MAC frame originated by the
individual VM. Thus, you could have overlapping MAC addresses across
segments"

RFC 7348 page 16, shows also 2 VMs using same VNI, now with the DPU acting as "a bump in the wire" it should be possible to have 2 tunnels using same VNI, it should be up to the user to make the decision if they want to keep the traffic under same VNI or change the VNI ID., use of same VNI inbound and outbound is not allowed in SAI, since direction lookup is based on VNI alone.

{
"name": "direction_lookup_entry_#eni2",
"op": "create",
"type": "SAI_OBJECT_TYPE_DIRECTION_LOOKUP_ENTRY",
"key": {
"switch_id": "5",
"vni": "2"
},
"attributes": [
"SAI_DIRECTION_LOOKUP_ENTRY_ATTR_ACTION",
"SAI_DIRECTION_LOOKUP_ENTRY_ACTION_SET_OUTBOUND_DIRECTION"
]
},

[mgheorghe]

diagram showing a mixed scenarios where communication happens over tunnels with same VNI or different VNI