forked from majek/openonload
-
Notifications
You must be signed in to change notification settings - Fork 0
/
README.ef_vi
112 lines (83 loc) · 4.49 KB
/
README.ef_vi
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
ef_vi
=====
ef_vi is an API that gives applications access to the network adapter's
datapath. That is, it can be used to send and receive raw Ethernet frames
directly from user-level. The advantage of using this interface is that
it incurs very little CPU overhead, and so yields the best possible
performance.
ef_vi is a zero-copy interface. This means that the user gets direct
access to the buffers in memory that the hardware uses, avoiding a copy
that is usually incurred by higher-level interfaces. Zero copy can be a
big win for applications that:
- have very high throughput requirements, or
- are filtering received data, or
- are forwarding data.
The downside of using ef_vi is that you have to implement the upper-layer
protocols yourself, including IP, UDP, TCP etc. It is relatively
straighforward to use ef_vi for UDP, but much more difficult for TCP which
is very complex to implement from scratch. (Best to leave that to
OpenOnload!). ef_vi can be used in an application at the same time as
OpenOnload, so an application might use ef_vi for some especially
performance sensitive UDP traffic, and sockets with OpenOnload for TCP and
other traffic.
ef_vi is included in the OpenOnload distribution, and is used by the
OpenOnload user-level sockets library to access the hardware's datapath.
Overview
========
Users of ef_vi must first allocate a virtual interface (VI), encapsulated
by the type "ef_vi". A VI includes:
- A receive descriptor ring. (For receiving packets).
- A transmit descriptor ring. (For sending packets).
- An event queue.
To transmit a packet, the application writes the packet contents
(including all headers) into one or more packet buffers, and calls
ef_vi_transmit(). One or more descriptors that describe the packet are
queued in the transmit ring, and a doorbell is "rung" to tell the NIC that
the transmit ring is ready.
To receive, descriptors (each identifying a buffer) are queued in the
receive ring by calling ef_vi_receive_init() and _post(). When packets
arrive at the network adapter and are directed to a VI, they are written
into the buffers in fifo order.
The event queue is a channel from the NIC to software which notifies
software when packets arrive from the network, and when transmits complete
(so that the buffers can be freed or reused). The application retrieves
these events by calling ef_eventq_poll().
The buffers used for packet data are special: They must be pinned so that
they cannot be paged, and they must be registered for DMA using an
ef_memreg.
The NIC uses a special address space to identify locations in registered
memory, and such addresses are designated by the type "ef_addr".
Protection domains (ef_pd) provide an address space for registered memory.
All VIs allocated against a given protection domain can access any memory
registered within that protection domain.
Filters are the means by which the NIC decides where to deliver packets it
receives from the network. By default all packets are delivered to the
kernel network stack. Filters are added by the application to direct
received packets to a given VI.
You can request a protection domain be associated with an SR-IOV virtual
function (VF). This allows more memory to be registered. Latency and
packet rate are slightly worse when using VFs due to address translations
in the system IOMMU. See the Solarflare Server Adapter User's Guide for
details of how to enable SR-IOV.
It is also possible to use physical addressing. This mode gives the best
latency and allows unlimited amounts of memory to be registered. However,
an application using physical addressing can cause the network adapter to
read and write arbitrary host memory, and so breaks the system security
model.
Documentation
=============
The ef_vi documentation is generated by doxygen from the relevant
source files. To create this, install doxygen, and then run:
cd src/include/etherfabric/doxygen
doxygen doxyfile_ef_vi
This will generate HTML, RTF, and LaTex documents in the relevant
subdirectory. That latter can be processed to create a PDF if
necessary.
There is also a set of sample applications, which are located at:
openonload/src/tests/ef_vi
If you've any questions please contact <support@solarflare.com>.
License
=======
ef_vi is released under the GNU Lesser General Public License -- see the
LICENSE file for details. If this license does not meet your needs,
please contact us at <support@solarflare.com> to discuss other options.