From b5b0f065915b3f44252658c0948b59eb02c5587e Mon Sep 17 00:00:00 2001 From: Eyal Lavee Date: Thu, 12 Jul 2018 13:41:13 +0000 Subject: [PATCH] Add Smart NIC Generic Offload Spec Blueprint: https://blueprints.launchpad.net/opencontrail/+spec/smart-nic-generic-offload Change-Id: I08960ee41f63a68f14fff8658791a191d70a4d25 Partial-Bug: #1781402 --- smart-nic-generic-offload.md | 168 +++++++++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 smart-nic-generic-offload.md diff --git a/smart-nic-generic-offload.md b/smart-nic-generic-offload.md new file mode 100644 index 0000000..6045602 --- /dev/null +++ b/smart-nic-generic-offload.md @@ -0,0 +1,168 @@ + +# 1. Introduction +This documents describes the proposed changes to introduce generic +smart NIC offload support for Tungsten Fabric vRouter data-path. + + +# 2. Problem statement +There are a wide range of smart/intelligent NICs emerging on the market, +implemented using various technologies. These NICs are capable of being +programed to perform advanced packet classification and modification +operations, allowing to implement flexible packet processing pipelines +implementing switching, routing, tunneling, etc. + +The goal of this feature is to define a framework and initial implementation +for accelerating vRouter packet processing/forwarding performance by utilizing +these capabilities. + +The requirement is to program the hardware NICs through existing standard +open source hardware offload APIs (e.g. DPDK rte_flow and/or Linux TC flower), +thereby enabling a single shared/generic vendor-agnostic offload implementation. + +# 3. Proposed solution + +The proposed solution defines a generic offload module layer in the vRouter. + +The generic offload layer logically resides between the vrouter datapath and +the smart NIC. It registers to receive notifications/callbacks on vrouter +configuration changes from the agent (these are based on previous work in +contrail vrouter 3.1 branch, see https://github.com/Juniper/contrail-vrouter/ +blob/R3.1/include/vr_offloads.h). + +The generic offload layer is responsible for programming offload rules to the +smart NIC through the existing standard open source hardware offload APIs +(e.g. DPDK rte_flow and/or Linux TC flower). The initial implementation will +add rules for classifying tunnel packets received on the fabric interface +matching established flows in the flow table, with an associated action to +mark/tag the packet with the flow id/index of the flow. + +Based on the programmed rules, packets received from the fabric interface are +checked to identify if they are marked/tagged. Packets identified with a +mark/tag are known to have hit the flow rule matching the specified index. +This information is then used to accelerate the vrouter data-path, bypassing +portions of the processing already handled in the NIC according to the +offload rules. The initial implementation proposed will bypass the majority +of the input classification process, including the input IP classification +the tunnel idenfifier classification (MPLS label or VXLAN VNI) and the flow +table classification. + +Any packet received without a mark/tag shall continue to be processed without +bypassing any of the existing vRouter implementation. + +Similarly, when offloads are not enabled, all packets will be considered +unmarked/untagged, and not subject to any offload/bypass, ensuring no +functional impact/risk to the non-accelerated mode. + +The offload layer shall be designed to support both DPDK and kernel based +vrouters, allowing a shared logical implementation with an OS callback layer +abstracting the OS specific functionality, in a manner similar to that +currently done within vRouter. The initial version will only implement the +DPDK OS callback layer, deferring TC offload to a second step. + +It is expected that future releases would continue to enhance this initial +implementation to further accelerate/offload vRouter by utilizing additional +offload capabilities as those are made available through enhancements pushed to +the standard offload APIs. + + +## 3.1 Alternatives considered +An alternative discussed in the community was to allow implementing the GOM +communication to the hardware NICs through vendor-specific hooks, instead of +the proposed existing standard APIs (Linux TC flower and DPDK rte_flow). + +The advantage of such an approach is that it would allow vendors to more +easily implement offloads, bypassing the need to agree on the required +extensions to the existing APIs and push these changes into the +repsective communities (Linux/DPDK). + +The disadvantage of such an approach is that it would inevitably lead to +multiple non-conformant vendor-specific offload implementations, instead of +a single shared/generic vendor-agnostic offload implementation. + +## 3.2 API schema changes +No expected changes to APIs. + +## 3.3 User workflow impact +The user will explicitly control enabling/disabling hardware offloads, +disabled by default. +For DPDK vrouter - enabling the offloads will be done via the vrouter +command line (new enable_offloads flag). +For kernel vrouter - TBD when implemented in future releases. +A deployment option should also be added to automate this process +when deploying compute nodes (method may be deployment specific, +initial support will be added to contrail-ansible-deployer process. + + +## 3.4 UI changes +- No expected changes to UI + +## 3.5 Notification impact +- Log if offload is enabled at startup +- Log any critical error while offloading rules + +# 4. Implementation +## 4.1 Work items + + - vrouter: Implement interface for an offload module to receive notifications + on vrouter configuration changes from the agent (based on previous work in + contrail vrouter 3.1 branch, see https://github.com/Juniper/ + contrail-vrouter/blob/R3.1/include/vr_offloads.h). +. + - vrouter: Implement DPDK command line for enabling generic offload layer, + triggering registration of generic offload layer callbacks with above + interface to receive notifications. Implementation for kernel based vrouter + layer may be done through kernel module config - deferred to future + releases implementing kernel offloads. + + - vrouter: Implement geenric offload layer control plane logic (tracking vrouter + configurations through GOM callback interfaces registers and programming + hardware rules and actions). + + - vrouter: Implement detection of tagged packets on packets received from the + fabric interface, and bypassing of input classification in vrouter datapath. + + - vrouter: Update vrouter-ansible-deployer to allow enabling vrouter offload + on a node. + + - vrouter: update vrouter to latest available stable DPDK version, targeting + DPDK 18.05 or 18.08 (depending on release schedule). + +# 5. Performance and scaling impact +## 5.1 API and control plane +- None + +## 5.2 Forwarding performance +- When offloading is disabled, there should be no performance impact. +- When offloading is enabled forwarding performance is expected to + increase for offloaded scenarios, and not have any significant performance + impact for scenarios not offloaded. + +# 6. Upgrade +- None + +# 7. Deprecations +- None + +# 8. Dependencies +- None + +# 9. Testing +## 9.1 Unit tests +- None +## 9.2 Dev tests +- None +## 9.3 System tests +- Run all existing system tests in CI with offload disabled. + - No functional impact + - No performance impact +- Run all existing system tests in CI with offload enabled. + - No functional impact + - Performance improvement or no performance impact + +# 10. Documentation Impact +- Document procedure for enabling smart NIC offloads. + +# 11. References +- TF community paravirt offload design doc: + Tungsten Fabric vRouter Smart NIC Offload for Para-virtualized VMs - Architecture and Design Proposal + https://docs.google.com/document/d/1gxzC6xz-qnTv5r6Gt5KZgNHER6eFI29DZkpKt4RxhY4/edit#