implement resource pool client

[pperiyasamy]

This repo contains resource pool server implementation which helps to select VF from right resource pool based on tokenID passed from client in the service request.
Can we also extend it to have client implementation so that VF attachment support available for endpoint container?

[denis-tingaikin]

Sounds good to me.

@Bolodya1997 Looks like it is related to your owning component. Could you also take a look at the suggestion?

[Bolodya1997]

We can extract some common logic and create a client chain element, but there is some basic problem with it.
When we are selecting VF for the Client requesting VFIO mechanism we have such scheme:

Client  --( token ID  )->  Forwarder
Client  <-(IOMMU group)--  Forwarder

With Endpoint there will be another scheme:

Forwarder  --(        )->  Endpoint
Forwarder  <-(token ID)--  Endpoint

So the only way to pass IOMMU group back to the Endpoint is to make second Request from the Forwarder. And there is no way to make this selection before we get token ID from the Endpoint.
So yes, we can add some client chain element to select a VF for the Endpoint and inject it in there, but there will be a problem to tell what we have selected back to the Endpoint.

@pperiyasamy is this a problem? Or probably you just don't want to pass something back to the Endpoint and so it is OK to you?

[pperiyasamy]

@Bolodya1997 Good find! in case of smartnic scenario, the VF would be just a kernel interface (also can be used as a DPDK device) configured inside endpoint container with IP address and routes (though interface name is derived in the forwarder) sent by endpoint upon service request. I hope that would be sufficient for endpoint to figure out which interface belongs to which p2p network in case if that's needed by endpoint application, it also has to have extra logic to figure out pci address of the VF device in dpdk scenario.

/cc @JanScheurich

[JanScheurich]

Mellanox SRIOV VFs are always bound to their mlx5_core kernel driver, so the netdev will be visible to the NSE pod.
As long as the NSE can map the interface name in the pod to the connected client, we should be fine. It is possible to map the netdev to the PCI device address for use with DPDK. Is the netdev interface name signaled to the NSE in the service request?

[edwarnicke]

@JanScheurich A question (to make sure I understand :) ).

Mellanox SRIOV VFs are always bound to their mlx5_core kernel driver, so the netdev will be visible to the NSE pod.

1. Am I correct that by 'netdev' you mean the kernel interface for the SRIOV VF?
2. If I'm correct about that, do you mean you want the netdev inserted into the NSE Pods Namespace, or do you mean it is inserted into the NSE Pods namespace?
3. If you mean it is inserted into the NSE Pods namespace, how exactly is that happening?

[JanScheurich]

Am I correct that by 'netdev' you mean the kernel interface for the SRIOV VF?

Yes

If I'm correct about that, do you mean you want the netdev inserted into the NSE Pods Namespace, or do you mean it is inserted into the NSE Pods namespace?

The SmartVF netdevs must be inserted into the NSE and NSC pod namespaces. Otherwise the NSC/NSE could not use it at all neither via kernel networking, nor as DPDK interface, nor as RDMA interface.

If you mean it is inserted into the NSE Pods namespace, how exactly is that happening?

That's the job of the OVS forwarder. It injects in into the pod namespace and applies all the usual interface configuration for a kernel interface.

[pperiyasamy]

Is the netdev interface name signaled to the NSE in the service request?

@JanScheurich currently its not, but in case of ovs forwarder, let it create a service request with interface name in mechanism preference parameter and let NSE use it for mapping afterwards. but application have to find a way to retrieve PCI device address using interface name.

[edwarnicke]

That's the job of the OVS forwarder. It injects in into the pod namespace and applies all the usual interface configuration for a kernel interface.

Got it... that's answers my question... the way you had phrased it sounded like you expected it to already be there before the Forwarder did its work. Happy to have misunderstood there :)

[edwarnicke]

@pperiyasamy

@JanScheurich currently its not, but in case of ovs forwarder, let it create a service request with interface name in mechanism preference parameter and let NSE use it for mapping afterwards. but application have to find a way to retrieve PCI device address using interface name.

I believe there are lots of ways of retrieving the PCI information for an interface:

$ ethtool -i eth0
driver: i40e
version: 1.5.16
firmware-version: 5.04 0x800024cd 0.0.0
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes

and https://github.com/safchain/ethtool