Click-DPDK configuration - Clarifications

[gkatsikas]

Hi,

I am happily using Click for more than a year and I have been able to run complex chained configurations using both Linux and netmap I/O elements. Recently, I installed the latest Click version that supports DPDK and I want to repeat the above chains.

I managed to bind my 2 physical NICs to DPDK (as ports 0 and 1) and I am able to run a Click router that is attached to these interfaces as follows:

click -l 2,4 -n 4 --master-lcore 2 --proc-type=primary -v -- router.click
The From/ToDPDKDevice elements look like:
in0 :: FromDPDKDevice($iface0);
out0 :: ToDPDKDevice ($iface0);
in1 :: FromDPDKDevice($iface1);
out1 :: ToDPDKDevice ($iface1);
where iface0=0 and iface1=1

In case of chained Click modules, I need to create several interfaces and attach them to OVS along with my physical ones. I found that DPDK allows to create several rings on the same NIC using dpdkr type. However, if I start one Click application that uses From/ToDPDKDevice(0) as a primary process type (e.g., click -l 0,2,4 -n 4 --proc-type=primary -v -- ponger.click iface=0 macAddr=.. ipAddr=..), then when I start a secondary application on the same DPDK port (i.e., 0), it fails:

click -l 0,2,4 -n 4 --proc-type=secondary -v -- ponger.click iface=0 macAddr=.. ipAddr=..
EAL: memzone_reserve_aligned_thread_unsafe(): memzone <RG_MP_mbuf_pool_0> already exists
RING: Cannot reserve memory
ponger.click:32: While initializing ‘ponger/in :: FromDPDKDevice’:
Could not allocate packet MBuf pools
Router could not be initialized!

Before running these two applications, I successfully bound my NIC to OVS-DPDK using:
ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
Created and pinned 4 PMD threads to 4 cores as follows:
ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x115
and set the number of rx queues per NIC to 4 as follows:
ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=4

Do you have any suggestions on how one can configure DPDK to use multiple Click processes? I currently do not use the DPDKInfo element, might this be the problem? Do I need extra arguments in the FromToDPDKDevice elements to enable rings?

Sorry for the long message but it would be great to summarize an intuitive configuration for running a bit more complex Click applications using DPDK.

Best regards,
Georgios Katsikas

[tbarbette]

Hi,
I'm from the team behind support for DPDK in Click. Frankly, I never tried to run multiple DPDK software at the same time, including Click. The DPDKInfo element allows you to set the usual DPDK parameter that a programmer would have to set himself, you probably want the default... Maybe just change the NB_MBUF parameter which is the number of buffer that DPDK will allocate for your application. It's something like 2200 bytes per buffer, the default is 524288 buffers so 1,2GB. If you run multiple applications, you probably don't want each application to eat as much memory.

The error "EAL: memzone_reserve_aligned_thread_unsafe(): memzone already exists" seems to arrise because another software already owns a memory zone.
Looking at the example from http://dpdk.readthedocs.org/en/latest/sample_app_ug/multi_process.html, the memory error is consistant : one of the application has to be in a "secondary mode" where it does not allocate a memory zone but attach to the one already allocated by the primary process.
I'm pretty sure two software cannot open the same NIC anyway. You would have to use VFIO for that (create multiple virtual NIC).

What's your final goal? Click can do a good part of what OVS can do.

Tom

[gkatsikas]

Hi Tom,

Thanks for the prompt reply and feedback!

My goal is to create chains of Click-based modules that operate upon DPDK interfaces. Therefore, my testing scenario involves a traffic generator, a traffic sink and a series of Click modules in between. This is the reason I want to use a software switch as a backbone dataplane; to interconnect these Click modules.

Right now, when I execute ${DPDK_DIR}/tools/dpdk_nic_bind.py --status I can only see the physical interfaces of my machine. Using ${DPDK_DIR}/tools/dpdk_nic_bind.py --bind=igb_uio <nic_pci> I can successfully bind the DPDK driver to my physical NICs.

Your suggestion is to use vfio, however, what is the recommended way to create a virtual interface for DPDK? In other words, how can I create an interface that can be used by this command?:
${DPDK_DIR}/tools/dpdk_nic_bind.py --bind=vfio-pci . Then I would like to attach this interface to OVS so that I can use OpenFlow rules to steer the traffic across the chained Click modules (that is the ultimate goal).

Thanks in advance and regards,
Georgios

[tbarbette]

Why not chaining the modules inside Click?

Sorry I did not ment the VFIO driver as is, but the Virtual Function drivers ( ... ). http://dpdk.org/doc/guides/nics/intel_vf.html . I nerver used it in practice... The idea is to configure virtual "function" nics with virtual mac addresses and get packets for those directly to the matching virtual function nic. This is intended for receiving packets directly inside VMs to do NFV, but could be useful for multiple DPDK applications as you can open a virtual function NIC independently for each applications. But if you use it for chaining, the packet will go back to the NIC through the PCI-express link to be switched inside the NIC and sent back to the host through the PCI-express link each time... So it's probably not your solution...

What I do to generate packets is to use another computer, running DPDK or Netmap to generate the traffic towards the one running Click, so it does not impact the results. And if I've got to run multiple functions on the traffic, I chain them inside Click itself. If I really wanted multiple process, I would implement a packet-passing mechanism through mapped memory myself...

If you really want to use OVS-DPDK, I can try it a little.

[gkatsikas]

The reason I want to deploy my chain as a multitude of processes (Click modules) is only for research purposes, to measure the overhead and impact of deploying standalone chained modules. Then, I can compare the multi-process vs. single-process chain and quantify the benefits :)

The setup you described is exactly what I have in mind. Two machines are connected back to back using 10Gb interfaces. One machine is saturating the link using pktgen (netmap or dpdk) and the other machine uses e.g., a software switch such as OVS on which the physical interface of this machine is attached (this is where the packets come from the source machine) and several virtual interfaces are also attached (used by my Click modules to process the traffic). Then, using openflow rules, I can guide the traffic across the Click processes until it reaches a destination.

How is this packet passing mechanism can work with Click though? Will it be a different element? My concern right now is "how can I have several pairs of FromDPDKDevice and ToDPDKDevice elements, exchanging packets?" and it seems that if OVS can host all these elements (through) virtual interfaces, then I can afford to pay I a "tax" doing this ping-pong between OVS and Click.

I appreciate your ideas and support :) If you want to discuss this issue off-line you can use my academic e-mail (katsikas@kth.se)

Regards,
Georgios

[bcronje]

Please if possible keep the discussion online so everyone else can benefit from it.

[gkatsikas]

Hi again,

Working around the problem I came up with the following idea based on my sketch below:

                        _____________________________   ______________________________
                        |                            |  |                             |
                        | FromDevice.u -> ToDevice.u |  | FromDevice.u  -> ToDevice.u |
                        |____________________________|  |_____________________________|
                            |                             |                    |  
                       _____|_______________|_____________|____________________|________
                       |                                                                |
          _____________|                                                                |
         |  eth0(phy) ||                            OVS-DPDK                            |
         |____dpdk0___||                                                                |
                       |________________________________________________________________|

Do I have performance limitations if I use normal userspace FromDevice/ToDevice elements in my Click functions with SNIFFER=false (to avoid the kernel sniffers)? I can also set the BURST size to match the DPDK requirements.

Following the above setup, I believe that I can create regular virtual interfaces on the OVS bridge, pass them to lightweight containers and let my Click functions run in the containers as pure userspace modules.

Other alternatives for such a chain?

[tbarbette]

Yes, FromDevice/ToDevice are far far slower. I'll look this week if we can run Click-DPDK with OVS-DPDK simultaneously, or even just multiple Click-DPDK simultaneously.

What I would personally do (if I want separate process) is to replace OVS-DPDK by a Click instance in your scheme, using an IPRoute or an EtherSwitch element to dispatch packets to other Click process. To send packets between process, I'd suggest implementing a new Element using shared memory between process, so without using DPDK.

[gkatsikas]

Thanks a lot, it would be great if you can come up with a quick solution to have Click-DPDK with OVS-DPDK, or multiple Click-DPDK modules simultaneously.

I thought about your solutions (e.g. EtherSwitch) as well but it would be much faster to classify your traffic in hardware (e.g. OpenFlow rules in a TCAM) and then dispatch it to several Click functions that will do some extra processing per traffic class. Doing everything in Click is not the fastest solution.
That's why, having multiple Click Functions atop an OVS-DPDK bridge sounds a very nice solution :)

The idea of a SharedMemory element is not bad, I will consider doing this. Once you pass packets to Click via DPDK, then you don\t need extra I/O using such an element.

[tbarbette]

I made a branch with support for being a DPDK secondary process : https://github.com/tbarbette/click/tree/PRClickSecondaryProcess

If you run click with the EAL argument --proc-type=secondary, it will attach to the mempool of OVS, instead of trying to allocate a new one. But after that, I'm still stucked. The problem with OVS is that it is built using DPDK shared libraries, but Click is using DPDK static libraries (faster). You can already try this version. I think the solution is around giving the "--veth" argument tool to create virtual interfaces between applications...

[tbarbette]

I guess this can be closed. We'll discuss the issue of ovs-DPDK in the referenced thread.