Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PF_RING ZC: ixgbe virtual function support #15

Closed
pavel-odintsov opened this issue Jun 16, 2015 · 30 comments
Closed

PF_RING ZC: ixgbe virtual function support #15

pavel-odintsov opened this issue Jun 16, 2015 · 30 comments
Labels
enhancement

Comments

@pavel-odintsov
Copy link

Hello, folks!

I have multiple virtual functions with ixgbe.

I have changed following lines in load_driver.sh:

-insmod ./ixgbe.ko RSS=1,1,1,1
+insmod ./ixgbe.ko RSS=1,1,1,1 max_vfs=2,2

Thus, I tuned virtual functions and set mac's for they:

ip link set eth6 vf 0 mac 90:e2:ba:55:aa:bb
ip link set eth6 vf 1 mac 90:e2:ba:55:bb:cc

And apply changes:

rmmod ixgbevf
modprobe ixgbevf

Thus, I have single physical NIC and two virtual NIC's:

21: eth6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 90:e2:ba:4a:d8:e8 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 90:e2:ba:55:aa:bb, spoof checking on, link-state auto
    vf 1 MAC 90:e2:ba:55:bb:cc, spoof checking on, link-state auto
29: eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 90:e2:ba:55:aa:bb brd ff:ff:ff:ff:ff:ff
30: eth11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 90:e2:ba:55:bb:cc brd ff:ff:ff:ff:ff:ff

And I have enabled all NICs:

ifconfig eth6 up
ifconfig eth10 up
ifconfig eth11 up

And all nic's become ready:

root@filteredclient /usr/src/PF_RING-6.0.3/userland/examples_zc # ethtool eth10
Settings for eth10:
    Supported ports: [ ]
    Supported link modes:   10000baseT/Full 
    Supported pause frame use: No
    Supports auto-negotiation: No
    Advertised link modes:  Not reported
    Advertised pause frame use: No
    Advertised auto-negotiation: No
    Speed: 10000Mb/s
    Duplex: Full
    Port: Other
    PHYAD: 0
    Transceiver: Unknown!
    Auto-negotiation: off
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes
root@filteredclient /usr/src/PF_RING-6.0.3/userland/examples_zc # ethtool eth11
Settings for eth11:
    Supported ports: [ ]
    Supported link modes:   10000baseT/Full 
    Supported pause frame use: No
    Supports auto-negotiation: No
    Advertised link modes:  Not reported
    Advertised pause frame use: No
    Advertised auto-negotiation: No
    Speed: 10000Mb/s
    Duplex: Full
    Port: Other
    PHYAD: 0
    Transceiver: Unknown!
    Auto-negotiation: off
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes
root@filteredclient /usr/src/PF_RING-6.0.3/userland/examples_zc # ethtool eth6
Settings for eth6:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseT/Full 
                            10000baseT/Full 
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  1000baseT/Full 
                            10000baseT/Full 
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Speed: 10000Mb/s
    Duplex: Full
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    Supports Wake-on: d
    Wake-on: d
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

Well! I could run PF_RING ZC.

It works perfectly for physical interface:

./zcount -c 0 -i zc:eth6
#########################################################################
# ERROR: You do not seem to have a valid PF_RING ZC license 6.0.3.150330 for eth6 [Intel 10 Gbit ixgbe 82599-based]
# ERROR: Please get one at http://shop.ntop.org/.
#########################################################################
# We're now working in demo mode with packet capture and          
# transmission limited to 5 minutes
#########################################################################
#########################################################################
# ERROR: You do not seem to have a valid PF_RING ZC license 6.0.3.150330 for eth6 [Intel 10 Gbit ixgbe 82599-based]
# ERROR: Please get one at http://shop.ntop.org/.
#########################################################################
=========================
Absolute Stats: 0 pkts (0 drops) - 0 bytes
=========================

^CLeaving...
=========================
Absolute Stats: 0 pkts (0 drops) - 0 bytes
Actual Stats: 0.00 pps (0.00 drops) - 0.00 Gbps
=========================

=========================
Absolute Stats: 0 pkts (0 drops) - 0 bytes
Actual Stats: 0.00 pps (0.00 drops) - 0.00 Gbps

But for virtual functions it did not work at all:

./zcount -c 0 -i zc:eth10
pfring_zc_open_device error [Socket operation on non-socket] Please check that zc:eth10 is up and not already used

root@filteredclient /usr/src/PF_RING-6.0.3/userland/examples_zc # ./zcount -c 0 -i zc:eth11
pfring_zc_open_device error [Socket operation on non-socket] Please check that zc:eth11 is up and not already used

Could you add support for PF_RING which run on top of virtual functions?
@pavel-odintsov pavel-odintsov mentioned this issue Jun 16, 2015
Closed
@cardigliano
Copy link
Member

This requires patching the ixgbevf driver to support ZC, we already started working on this some time ago but need to complete it. We will try to allocate more time on the roadmap for this as soon as possible.

@pavel-odintsov
Copy link
Author

Thank you so much! I really appreciate your help! :)

On Fri, Jun 19, 2015 at 10:33 AM, Alfredo Cardigliano <
notifications@github.com> wrote:

This requires patching the ixgbevf driver to support ZC, we already
started working on this some time ago but need to complete it. We will try
to allocate more time on the roadmap for this as soon as possible.

—
Reply to this email directly or view it on GitHub
#15 (comment).

Sincerely yours, Pavel Odintsov

@majek
Copy link

majek commented Jun 19, 2015

I'm also very much interested in ixgbevf ZC driver.

@edsealing
Copy link

Very interested in this as well.

@dnj12345
Copy link

Any update on this?

@cardigliano
Copy link
Member

Not yet, we are busy with other projects with higher priority. sorry.

@jeremyerb jeremyerb mentioned this issue Oct 30, 2017
Closed
@cardigliano cardigliano changed the title Could not run PF_RING on ixgbe's virtual function PF_RING ZC: ixgbe virtual function support Nov 13, 2017
@jcmathews
Copy link

@cardigliano: Is this supported now ?

@jcmathews
Copy link

Hi Alfredo,

Does the VF support is included latest PF_RING? I do see some commits related to it, please do confirm.

Regards,
Jessel.

@cardigliano
Copy link
Member

@jcmathews this is still in progress, I will update this issue as soon as it is ready

@jcmathews
Copy link

Thanks

@cardigliano
Copy link
Member

Support for ixgbe-vf has been added to ZC and it is available on latest dev branch. This is currently under testing, if you want to start playing with it you are welcome.

@jcmathews
Copy link

jcmathews commented Nov 30, 2018 •
edited
Loading

Hi Alfredo,

Do you have configuration details to hook vf's by using qemu cmd line(but not by libvirt)? Any script changes to examples_zc/kvm/host/vm-boot.sh

Regards,
Jessel.

@cardigliano
Copy link
Member

@jcmathews I do not have a sample configuration for Qemu atm, I will work on that and update the documentation asap. (however you should be able to find instructions on google as it's a generic sriov configuration)

@jcmathews
Copy link

@cardigliano
Qemu v2.5 have some issue w.r.t to IOMMU, when using argument pci-assign.
Qemu v2.12 will boot up using vfio-pci, but ipc will not work since ivshmem-doorbell is not in place.
Generic SRIO configuration unbinds existing driver(say ixgbevf) and hook vf's using vfio-pci. Do you have idea about some other configuration.

@cardigliano
Copy link
Member

@jcmathews could you provide more details about the configuration/processing pattern you want to achieve? Virtual Functions are not supposed to be used with IPC/ivshmem as they are two differnt ways of achieving the same, bringing some traffic to the VM

@jcmathews
Copy link

jcmathews commented Nov 30, 2018 •
edited
Loading

@cardigliano :
i have 2 10G cards (X540 & 82599ES) cards,
i will receive the traffic from X540, then pass it to VM's via multiple QMP sockets. The VM's does some packet processing and sent back traffic to 82599 virtual functions.
For this case, my infrastructure look forward to use QMP sockets for traffic distribution towards VM and SRIOV based configuration for sending out the traffic to outside world.

@cardigliano
Copy link
Member

Got it. Could you comment more about "Generic SRIO configuration unbinds existing driver(say ixgbevf) and hook vf's using vfio-pci"? VF can be either on the host or assigned to a VM (unbinding it from the host), I guess you need the latter for sending from the VM, what is the problem with that?

@jcmathews
Copy link

@cardigliano :

Sorry for late reply.

The below step are from
https://www.linux-kvm.org/page/10G_NIC_performance:_VFIO_vs_virtio

Assign the VF to a guest
Unbind from igbvf driver and Bind to VFIO driver

unbind from previous driver (take igbvf device for example)
echo <vf_BDF> > /sys/bus/pci/device/<vf_BDF>/driver/unbind
lspci -s <vf_BDF> -n //to get its number
//it will return like below
0a:13.3 0200: 8086:1520 (rev 01)
//8086 1520 is its numeric number
bind to vfio-pci driver
echo 8086 1520 > /sys/bus/pci/drivers/vfio-pci/new_id
Now you can see this device is bound to vfio-pci driver
lspci -s <vf_BDF> -k

Create guest with direct passthrough via VFIO framework
qemu-kvm -m 16G -smp 8 -net none -device vfio-pci,host=81:10.0 -drive file=/var/lib/libvirt/images/rhel7.1.img,if=virtio -nographic
'-net none' tells qemu not emulate network devices
'-device vfio-pci,host=' designate a vfio-pci device and the device's host BDF

Problem 1:
Our qemu version 2.5 has issues with registering VFs and throws error as follows,
qemu-system-x86_64: -device pci-assign,host=08:10.3: No IOMMU found. Unable to assign device "(null)"
However if 2.12 is used, this error will not come.

Problem 2:
ivshmem to ivshmem-doorbell is not in place, so Qemu 2.12 version will not be compatible with PFRING.

Is above mentioned registration correct for VF's or any other way you used to use?

@cardigliano
Copy link
Member

Hi @jcmathews, we added instructions for assigning a VF without virsh at http://www.ntop.org/guides/pf_ring/vm/sriov.html (which is what you are doing).
Could you confirm that this is working for you on Qemu 2.12 and you are able to use the ixgbevf on the VM? (besides the fact that ivshmem is not working on this Qemu version)
Thank you.

@jcmathews
Copy link

jcmathews commented Dec 7, 2018 •
edited
Loading

I got some errors as follows

WARNING: "ixgbevf_hv_set_rar_vf" [/root/PF_RING/drivers/intel/ixgbevf/ixgbevf-4.5.1-zc/src/ixgbevf.ko] undefined!
WARNING: "ixgbevf_hv_reset_hw_vf" [/root/PF_RING/drivers/intel/ixgbevf/ixgbevf-4.5.1-zc/src/ixgbevf.ko] undefined!

Kernel Version Used : 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
RHEL7

#if IS_ENABLED(CONFIG_PCI_HYPERV) not enabled in VM. while compiling it throws error.

@cardigliano
Copy link
Member

cardigliano commented Dec 7, 2018 •
edited
Loading

Strange this is working for me on a VM with 3.10.0-862.14.4.el7.x86_64, I am investigating..
Please provide cat /boot/config-3.10.0-327.el7.x86_64 | grep CONFIG_PCI_HYPERV

@jcmathews
Copy link

jcmathews commented Dec 7, 2018 •
edited
Loading

config-3.10.0-327.el7.x86_64.zip
CONFIG_PCI_HYPERV Not found

@cardigliano
Copy link
Member

I pushed a patch to handle that case, please try compiling latest code from github:
cd PF_RING/drivers/intel/ixgbevf/ixgbevf-4.5.1-zc && make

@jcmathews
Copy link

jcmathews commented Dec 7, 2018 •
edited
Loading

48.253334] ixgbevf: Intel(R) 10GbE PCI Express Virtual Function Driver - version 4.5.1
[ 48.255431] Copyright(c) 1999 - 2018 Intel Corporation.
[ 48.322206] ixgbevf 0000:00:04.0: MAC address not assigned by administrator.
[ 48.324320] ixgbevf 0000:00:04.0: Assigning random MAC address
[ 48.327969] ixgbevf 0000:00:04.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
[ 48.331626] ixgbevf 0000:00:04.0: 12:64:14:f9:18:65
[ 48.333524] ixgbevf: eth1: ixgbevf_probe: Intel(R) 82599 Virtual Function
[ 48.334782] ixgbevf: eth1: ixgbevf_probe: GRO is enabled
[ 48.336058] ixgbevf: eth1: ixgbevf_probe: Intel(R) 10GbE PCI Express Virtual Function Driver
[ 48.348392] [PF_RING] removing dev=ens4 ifindex=5 (2)
[ 48.352923] IPv6: ADDRCONF(NETDEV_UP): ens4: link is not ready.

zsend if failing to send packets out.

[root@localhost examples_zc]# ./zsend -i zc:ens4 -r 2 -l 256
#########################################################################

ERROR: You do not seem to have a valid PF_RING ZC 7.3.0.181122

license for ens4 (MAC 12:64:14:F9:18:65) [Intel 10 Gbit ixgbe VF]

ERROR: Please get one at http://shop.ntop.org/.

#########################################################################

We're now working in demo mode with packet capture and

transmission limited to 5 minutes

#########################################################################
#########################################################################

ERROR: You do not seem to have a valid PF_RING ZC 7.3.0.181122

license for ens4 (MAC 12:64:14:F9:18:65) [Intel 10 Gbit ixgbe VF]

ERROR: Please get one at http://shop.ntop.org/.

#########################################################################
Sending packets to zc:ens4
Estimated CPU freq: 1745089500 Hz
Rate set to 2.00 Gbit/s, 64-byte packets, 2976190 pps

Absolute Stats: 4'096 pkts - 1'146'880 bytes

=========================
Absolute Stats: 4'096 pkts - 1'146'880 bytes
Actual Stats: 0.00 pps - 0.00 Gbps [0 bytes / 1.0 sec]

My host kernel cmd line is not having these "msi=1 pci=assign-busses", its an RHEL7 machine.
VF's are created.
75: ens3f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 1000
link/ether 14:02:ec:96:ed:65 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:01:02:03:04:06, spoof checking on, link-state auto, trust off, query_rss off
vf 1 MAC 12:64:14:f9:18:65, spoof checking on, link-state auto, trust off, query_rss off

@cardigliano
Copy link
Member

@jcmathews could you try with pfsend (which is using a different memory allocation method) to figure out if this is related to hugepages?
Did you also check pfcount to receive traffic?

@jcmathews
Copy link

Host Logs:
[340246.523724] ixgbe 0000:08:00.1 ens3f1: 1934439 Spoofed packets detected
[340248.528037] ixgbe 0000:08:00.1 ens3f1: 2287641 Spoofed packets detected
[340250.531983] ixgbe 0000:08:00.1 ens3f1: 2287730 Spoofed packets detected
[340250.532131] ixgbe 0000:08:00.1 ens3f1: initiating reset due to lost link with pending Tx work
[340252.536104] ixgbe 0000:08:00.1 ens3f1: Reset adapter
[340254.801276] ixgbe 0000:08:00.1 ens3f1: detected SFP+: 5

VM Logs:

[ 196.394348] Copyright(c) 1999 - 2018 Intel Corporation.
[ 196.460808] ixgbevf 0000:00:04.0: MAC address not assigned by administrator.
[ 196.463247] ixgbevf 0000:00:04.0: Assigning random MAC address
[ 196.466332] ixgbevf 0000:00:04.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
[ 196.468624] ixgbevf 0000:00:04.0: 5a:27:c9:bb:f6:a7
[ 196.469158] ixgbevf: eth1: ixgbevf_probe: Intel(R) 82599 Virtual Function
[ 196.470125] ixgbevf: eth1: ixgbevf_probe: GRO is enabled
[ 196.471265] ixgbevf: eth1: ixgbevf_probe: Intel(R) 10GbE PCI Express Virtual Function Driver
[ 196.505308] [PF_RING] removing dev=ens4 ifindex=6 (2)
[ 196.511071] IPv6: ADDRCONF(NETDEV_UP): ens4: link is not ready
[ 196.529759] IPv6: ADDRCONF(NETDEV_UP): ens4: link is not ready
[ 196.531917] IPv6: ADDRCONF(NETDEV_UP): ens4: link is not ready
[ 198.619689] load_driver.sh (14458): drop_caches: 3
[ 243.393594] bash (14613): drop_caches: 3
[ 277.957133] Bits 55-60 of /proc/PID/pagemap entries are about to stop being page-shift some time soon. See the linux/Documentation/vm.

Pfsend:
Using PF_RING v.7.3.0
Estimated CPU freq: 2656320000 Hz
Rate set to 2.00 Gbit/s, 260-byte packets, 892857.14 pps
Dumping statistics on /proc/net/pf_ring/stats/15070-ens4.2
TX rate: [current 1'093'826.08 pps/2.45 Gbps][average 1'093'826.08 pps/2.45 Gbps][total 1'093'895.00 pkts]
TX rate: [current 1'141'535.75 pps/2.56 Gbps][average 1'117'681.79 pps/2.50 Gbps][total 2'235'586.00 pkts]
TX rate: [current 1'141'567.48 pps/2.56 Gbps][average 1'125'644.56 pps/2.52 Gbps][total 3'377'456.00 pkts]
TX rate: [current 1'141'538.66 pps/2.56 Gbps][average 1'129'618.16 pps/2.53 Gbps][total 4'519'199.00 pkts]
TX rate: [current 1'141'561.76 pps/2.56 Gbps][average 1'132'007.06 pps/2.54 Gbps][total 5'661'053.00 pkts]
TX rate: [current 1'141'545.48 pps/2.56 Gbps][average 1'133'596.76 pps/2.54 Gbps][total 6'802'772.00 pkts]
TX rate: [current 1'141'555.41 pps/2.56 Gbps][average 1'134'733.63 pps/2.54 Gbps][total 7'944'437.00 pkts]
TX rate: [current 880'062.71 pps/1.97 Gbps][average 1'102'901.38 pps/2.47 Gbps][total 8'824'593.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 980'362.18 pps/2.20 Gbps][total 8'824'593.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 882'330.56 pps/1.98 Gbps][total 8'824'593.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 802'122.55 pps/1.80 Gbps][total 8'824'593.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 735'281.58 pps/1.65 Gbps][total 8'824'593.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 678'716.90 pps/1.52 Gbps][total 8'824'593.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 630'239.11 pps/1.41 Gbps][total 8'824'593.00 pkts]

Packets started flowing out and then immediately stopped. Retries also didnt work

@cardigliano
Copy link
Member

@jcmathews are you reforging the MAC address to match what is configured for the VF? It seems the adapter detects spoofed packets. Please note pfsend supports that:
-M Reforge source MAC (format AA:BB:CC:DD:EE:FF)
-m Reforge destination MAC (format AA:BB:CC:DD:EE:FF)

@jcmathews
Copy link

@cardigliano
Irrespective of mac address, pfsend should be able to send packets down to adapter. Here i am able to send some packets at start(only once after VM is up) then it stop's sending. Say my case is to work in promiscuous mode here. Am i missing something?

kernel cmd line is not having these "msi=1 pci=assign-busses. Do you see any impacts?

@cardigliano
Copy link
Member

@jcmathews please note that spoof check is a generic feature the adapter provide (not related to ZC), you can disable it with:
ip link set vf 0 spoofchk off
Please let me know if it works for you.

@seandex
Copy link

seandex commented Jan 25, 2020

is there a way this zc working on OVH public clouds?
I don't have KVM management access since public clouds are all guest VMs.
nfnetlink_queue 24576 0
nfnetlink_log 20480 0
nfnetlink 16384 2 nfnetlink_queue,nfnetlink_log
virtio_net 49152 0

@xyzheng0706 xyzheng0706 mentioned this issue Oct 26, 2021
Closed
@ronygut ronygut mentioned this issue Jul 31, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@majek @jcmathews @pavel-odintsov @cardigliano @edsealing @dnj12345 @seandex