*** ndpi library version mismatch, expected API rev.10926 (4.9.0) found rev.11086 (4.9.0-4778-c2b657f) ***

[ronygut]

Using the latest versions of PF_RING and nDPI
PF_RING 8.6.1 FROM October 23
nDPI 4.8 October 23

PF_RING FT is configured to use static nDPI
./configure --disable-ft-dl

When building PF_RING FT example with pcap and run ./ftflow_pcap -7 -i eth0
receiving:
*** ndpi library version mismatch, expected API rev.10926 (4.9.0) found rev.11086 (4.9.0-4778-c2b657f) ***

Please advise.
Thanks,
Rony.

[cardigliano]

Please try pulling the latest code, we just updated the FT library to match latest nDPI

[ronygut]

It works now.

But now I see another problem.
The sample program ftflow_pcap uses static ndpi library
The sample program ftflow_pcap is crashing (with the following parameters -i ens160 -7)
I also uncomment the line in the sample code :
pfring_ft_set_l7_detected_callback(ft, l7Detected, NULL);

here are 2 crash back trace from GDB
By the way it always crash with segmentation fault when I click on ctrl-c to stop the program.

Program received signal SIGSEGV, Segmentation fault.
0x00005555556b9281 in ndpi_serialize_binary_raw ()
(gdb) bt
#0 0x00005555556b9281 in ndpi_serialize_binary_raw ()
#1 0x00005555556c7112 in ndpi_flow2json ()
#2 0x00005555556260c5 in processFlow ()
#3 0x0000555555691ec0 in pfring_ft_housekeeping ()
#4 0x0000555555692e4c in pfring_ft_process ()
#5 0x000055555562551a in process_packet ()
#6 0x000055555562a589 in pcap_handle_packet_mmap (handle=handle@entry=0x555555e0a8a0, callback=callback@entry=0x5555556254d0 <process_packet>, user=user@entry=0x0,
frame=frame@entry=0x7ffff72151d0 "\b\006", tp_len=, tp_mac=, tp_snaplen=1454, tp_sec=1722416485, tp_usec=48, tp_vlan_tci_valid=0, tp_vlan_tci=0, tp_vlan_tpid=33024)
at ./pcap-linux.c:4194
#7 0x000055555562a994 in pcap_read_linux_mmap_v3 (handle=0x555555e0a8a0, max_packets=, callback=0x5555556254d0 <process_packet>, user=0x0) at ./pcap-linux.c:4344

and:

munmap_chunk(): invalid pointer

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140737350596416) at ./nptl/pthread_kill.c:44
44 ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737350596416) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=140737350596416) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=140737350596416, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007ffff7ce4476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x00007ffff7cca7f3 in __GI_abort () at ./stdlib/abort.c:79
#5 0x00007ffff7d2b676 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7e7db8c "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#6 0x00007ffff7d42cfc in malloc_printerr (str=str@entry=0x7ffff7e80230 "munmap_chunk(): invalid pointer") at ./malloc/malloc.c:5664
#7 0x00007ffff7d42fdc in munmap_chunk (p=) at ./malloc/malloc.c:3060
#8 0x00007ffff7d4749a in __GI___libc_free (mem=) at ./malloc/malloc.c:3381
#9 0x00005555556b8fc9 in ndpi_realloc ()
#10 0x00005555556bd2bd in ndpi_serialize_binary_uint32 ()
#11 0x00005555556c727e in ndpi_flow2json ()
#12 0x00005555556260c5 in processFlow ()
#13 0x312e3237310c0070 in ?? ()
#14 0xbb35322e31362e37 in ?? ()
#15 0x695f747365640700 in ?? ()
#16 0x32312e34370d0070 in ?? ()
#17 0x3636312e38392e35 in ?? ()
#18 0x0000000000000000 in ?? ()

[cardigliano]

@ronygut it seems I am not able to reproduce it, I will analyse the code, thank you for the trace

[cardigliano]

I was finally able to reproduce and fix this, please git pull the latest code and let me know

[ronygut]

Which git project to pull? nDPI , PF_RING, or both?

[cardigliano]

PF_RING

[ronygut]

Fixed!
Thanks

[ronygut]

can I use pfring_ft_process from different threads?
Is it thread safe?
If I use PF_RING ZC with multiple consumer threads with pfring_zc_run_balancer_v2 can I use one flow table, or do I need to use flow table per consumer thread?

[cardigliano]

For performance reason FT does not implement locking and it has been designed to be used with one instance per thread in case of multithreading, rather than sharing a single table. In short, please allocate multiple tables, one per thread.

[ronygut]

in pfring_ft_create_table what does user_metadata_size is used for?
If I set it to 0, how this influence the flow table behavior?
v = pfring_ft_flow_get_value(flow);
I use v->user in flow value to hold context for sessions from start of the session until the end of the session.

[ronygut]

v = pfring_ft_flow_get_value(flow);
is v->user can be used to allocate application flow context?
Is it the responsibility of the application to free any allocation pointed by v->user before calling pfring_ft_flow_free
I use it to allocate an instance of a c++ class
How do I use the v->user field to store application flow context?

[cardigliano]

in pfring_ft_create_table what does user_metadata_size is used for? If I set it to 0, how this influence the flow table behavior? v = pfring_ft_flow_get_value(flow); I use v->user in flow value to hold context for sessions from start of the session until the end of the session.

If you use v->user to store data, the size of that data should be provided via the user_metadata_size parameter, otherwise you access memory out of boundaries

[cardigliano]

v = pfring_ft_flow_get_value(flow); is v->user can be used to allocate application flow context? Is it the responsibility of the application to free any allocation pointed by v->user before calling pfring_ft_flow_free I use it to allocate an instance of a c++ class How do I use the v->user field to store application flow context?

FT takes care of allocating v->user (you can cast it to your data structure), just please pay attention if you place there pointers to other memory that you allocate dynamically (FT is not aware of those and you have to free them)

[ronygut]

pf_ring ft is responsible to allocate that size of bytes, and also to delete it at end of flow?
The application just uses this buffer?

[cardigliano]

Correct

[ronygut]

Do the callback from pfring_ft_set_flow_packet_callback is always called for each packet added to the flow?
when l7 classification callback is called is it the same packet that will be called on the callback from pfring_ft_set_flow_packet_callback?
Which callback from the two will be called first?
the l7 callback is not called if the flow is short and it was not possible to do DPI? In this case the flow protocol is set to Unknown, or l7 callback should always be called even on short flows?

[cardigliano]

Do the callback from pfring_ft_set_flow_packet_callback is always called for each packet added to the flow?
Yes
when l7 classification callback is called is it the same packet that will be called on the callback from pfring_ft_set_flow_packet_callback?
L7 classification is called one per flow, as soon as the application is detected
Which callback from the two will be called first?
flow_packet_callback after
the l7 callback is not called if the flow is short and it was not possible to do DPI?
No
In this case the flow protocol is set to Unknown, or l7 callback should always be called even on short flows?
l7 callback is not called in that case, you should use the export callback to check the protocol at flow termination

[ronygut]

is HBO - host order?
source and destination is changed in each packet in the flow.
When do you determine the client and server setting for IPs and ports?
e.g. client is the initiating of the flow in TCP.

In pfring_ft_flow_key , the saddr ,sport and daddr, dport are the client and server for the flow?

The direction in metadata is changed to s2d_direction and d2s_direction according to the packet itself?

The saddr ,sport and daddr, dport in pfring_ft_flow_key will always be the same in the flow? are they indicate client/server?

[cardigliano]

is HBO - host order?
Yes, Host Byte Order

source and destination is changed in each packet in the flow. When do you determine the client and server setting for IPs and ports? e.g. client is the initiating of the flow in TCP.
Client is the src IP from the first packet of the flow

In pfring_ft_flow_key , the saddr ,sport and daddr, dport are the client and server for the flow?
Yes

The direction in metadata is changed to s2d_direction and d2s_direction according to the packet itself?
Yes

The saddr ,sport and daddr, dport in pfring_ft_flow_key will always be the same in the flow? are they indicate client/server?
Yes

[ronygut]

Do I need to call pfring_ft_housekeeping in the loop that process packets every x seconds for each flow table I use?
I currently do not call it and if I remember correctly I saw that it was called by the Flow Table implicitly? (saw it in the back trace of the debugger)

[ronygut]

Does VLAN ID part of the flow key by default? I know it is part of the key structure, but does the flow key consists of vlan id + src ip + src port + dst ip + dst port + protocol?
If yes, is it possible to configure that vlan id will not be part of the unique flow key?

I use only IPV4

I'm creating 12 threads and each has a reference to different pfring_ft_table*
All threads are using the same callbacks (start flow, end flow etc...) routines that will be called from different threads.
The callbacks know to which flow table they are using as they have the right reference in the user* pointer of the callback. (flow table index)
I can see that I get the same flow with different vlan ID (0 and 200) called on the new flow callback for the same thread.
The strange thing is that if I use TCPDUMP to see packets with VLAN ID 0 I don't see such packets at all.
tcpdump -i ens160 -nn vlan 0 and ip -e
I can only see packets with VLAN ID 200
tcpdump -i ens160 -nn vlan 200 and ip -e
How come PF_RING FT see such flows with VLAN ID 0?
By the way I can see VLAN ID 0 also in the example_ft program ftflow.
vlanId: 0, srcIp: 10.168.233.137, dstIp: 10.201.55.103, srcPort: 43832, dstPort: 9093

Maybe, VLAN ID 0 indicates that there was no VLAN in the packet?
If so, how do you distinguish between no VLAN ID and VLAN ID 0 in the packet?

[ronygut]

What does pfring_ft_ext_pkthdr is used for in pfring_ft_create_table.
I see that in the ftflow.c you pass the packet hash

[cardigliano]

Do I need to call pfring_ft_housekeeping in the loop that process packets every x seconds for each flow table I use? I currently do not call it and if I remember correctly I saw that it was called by the Flow Table implicitly? (saw it in the back trace of the debugger)

pfring_ft_housekeeping is called automatically once per second when calling pfring_ft_process(), however it is a good practice to call it by the caller when idle (in case pfring_ft_process is not called). However it is not mandatory to call it.

[cardigliano]

What does pfring_ft_ext_pkthdr is used for in pfring_ft_create_table. I see that in the ftflow.c you pass the packet hash

This is not actually used by FT itseld, it is for external use.

[cardigliano]

Does VLAN ID part of the flow key by default? I know it is part of the key structure, but does the flow key consists of vlan id + src ip + src port + dst ip + dst port + protocol? If yes, is it possible to configure that vlan id will not be part of the unique flow key?

Yes, vlan id is part of the flow key. We can make it optional (open a separate ticket with this feature request)

I use only IPV4

I'm creating 12 threads and each has a reference to different pfring_ft_table* All threads are using the same callbacks (start flow, end flow etc...) routines that will be called from different threads. The callbacks know to which flow table they are using as they have the right reference in the user* pointer of the callback. (flow table index) I can see that I get the same flow with different vlan ID (0 and 200) called on the new flow callback for the same thread. The strange thing is that if I use TCPDUMP to see packets with VLAN ID 0 I don't see such packets at all. tcpdump -i ens160 -nn vlan 0 and ip -e I can only see packets with VLAN ID 200 tcpdump -i ens160 -nn vlan 200 and ip -e How come PF_RING FT see such flows with VLAN ID 0? By the way I can see VLAN ID 0 also in the example_ft program ftflow. vlanId: 0, srcIp: 10.168.233.137, dstIp: 10.201.55.103, srcPort: 43832, dstPort: 9093

Could you provide a pcap to reproduce this?

Maybe, VLAN ID 0 indicates that there was no VLAN in the packet? If so, how do you distinguish between no VLAN ID and VLAN ID 0 in the packet?

Yes, 0 means no vlan for us. Do you have a vlan with id 0?

[ronygut]

Thank you for your answers, I really appreciate it!
Another important question.
I want to use pf_ring ft in multithreaded environment.
I use PF_RINF ZC to create multiple consumer threads depending on configuration and expected load of the environment.
I use the default Built in IP based hash to distribute the load across threads.
Each threads is bind to a core.
Each thread gets its own traffic and handle it.
For performance , I want to use PF_RING FT (create Flow Table) for each thread.
How do you recommend creating each flow table that will be used in each consumer thread?
Do I need to declare __thread pfring_ft_tableto* ft to use by pfring_ft_create_table? in each consumer thread?( I mean to use thread local storage declaration?)

Can I use an array of pfring_ft_tableto* , according to the number of consumer threads that I create, dynamically, and each thread will have an index to this array? (I will call pfring_ft_create_table in a main thread and fill it up before consumer threads are created and each can use one table according to an index in the array)

What is the recommended way? I want to avoid synchronizations between PF_RING ZC consumer threads on the flow table?
Also all callback functions that I declare , can they be shared across threads by using the index that is dedicated to a specific consumer thread?

If you have an example of using multiple pf_ring flow table in multiple threads I would appreciate that you can share the code as an example on how to correctly do it.

[ronygut]

Does VLAN ID part of the flow key by default? I know it is part of the key structure, but does the flow key consists of vlan id + src ip + src port + dst ip + dst port + protocol? If yes, is it possible to configure that vlan id will not be part of the unique flow key?

Yes, vlan id is part of the flow key. We can make it optional (open a separate ticket with this feature request)

I use only IPV4
I'm creating 12 threads and each has a reference to different pfring_ft_table* All threads are using the same callbacks (start flow, end flow etc...) routines that will be called from different threads. The callbacks know to which flow table they are using as they have the right reference in the user* pointer of the callback. (flow table index) I can see that I get the same flow with different vlan ID (0 and 200) called on the new flow callback for the same thread. The strange thing is that if I use TCPDUMP to see packets with VLAN ID 0 I don't see such packets at all. tcpdump -i ens160 -nn vlan 0 and ip -e I can only see packets with VLAN ID 200 tcpdump -i ens160 -nn vlan 200 and ip -e How come PF_RING FT see such flows with VLAN ID 0? By the way I can see VLAN ID 0 also in the example_ft program ftflow. vlanId: 0, srcIp: 10.168.233.137, dstIp: 10.201.55.103, srcPort: 43832, dstPort: 9093

Could you provide a pcap to reproduce this?

Maybe, VLAN ID 0 indicates that there was no VLAN in the packet? If so, how do you distinguish between no VLAN ID and VLAN ID 0 in the packet?

Yes, 0 means no vlan for us. Do you have a vlan with id 0?

No we don't have VLAN ID 0.
I understand that in this case it is populate with default 0 in the VLAN ID

[ronygut]

Where can I open the ticket for the VLAN ID field that will be an optional in the flow key?

[ronygut]

@cardigliano I think I know what is the main difference between my program and ftflow.c
After enabling the slicing, In the callback of export flow (processFlow) I check for the status and if it is PFRING_FT_FLOW_STATUS_SLICE_TIMEOUT I don't free the flow as the flow is not ended yet. (e.g. not calling pfring_ft_flow_free(flow);)
In the ftflow.c on every call to processFlow callback it calls pfring_ft_flow_free to free the flow.
However in case of slicing you should not free the flow and continue.
Did you free the flow also on slicing?

[cardigliano]

Please free the flow, regardless of the status. Slices are is just a copy of the original flow.

[ronygut]

@cardigliano , In this case I'm not sure I understand the use of slices.
I thought that slices are used for long term flows and allow me to get a callback every x seconds until the flow is ended.
This allow me to do things for long running flows until the flow is ended. (not get the end flow only after the flow is ended, that can be after a day in some cases)
calling free flow will delete the user data and all the flow data, correct?
So, what if the use of slices in this case?
Please elaborate.

[cardigliano]

@ronygut you are correct: slices are used for long term flows to get updates without expiring the flow. The main flow is not touched until it expires, and it keeps the reference to the user data. Exported slices are like clones of the flow with a snapshot of the counters at the moment they are exported.

[ronygut]

@cardigliano , so I need to call free flow each time the export callback is called regardless if the flow status is PFRING_FT_FLOW_STATUS_SLICE_TIMEOUT?

[cardigliano]

Correct

[ronygut]

@cardigliano , I don't have an internet connection from the machine I build PF_RING
Is it possible to create a new zip file, so I can use it to get the API mismatch fix?

[ronygut]

@cardigliano , I used git clone and pulled the repository and still getting the same error:
*** ndpi library version mismatch, expected API rev.11192 (4.11.0) found rev.4108 (4.11.0) ***

I build PF_RING like this:
cd userland
./configure --disable-ft-dl
make

Used new libpfring.a file in my program

I built ndpi as static:
./autogen.sh --with-only-libndpi
/configure --enable-static
make

Used new libndpi.a in my program

Please advise

[ronygut]

@cardigliano , tried to call free flow also when getting slice timeout PFRING_FT_FLOW_STATUS_SLICE_TIMEOUT and exactly after timeout expired and getting indication of status = PFRING_FT_FLOW_STATUS_SLICE_TIMEOUT , the program crashes probably because I'm trying to access the user data in the flow.

[cardigliano]

@cardigliano , I used git clone and pulled the repository and still getting the same error: *** ndpi library version mismatch, expected API rev.11192 (4.11.0) found rev.4108 (4.11.0) ***

I build PF_RING like this: cd userland ./configure --disable-ft-dl make

Used new libpfring.a file in my program

I built ndpi as static: ./autogen.sh --with-only-libndpi /configure --enable-static make

Used new libndpi.a in my program

Please advise

FT is dynamically loading ndpi from the installation paths (/usr/lib/libndpi.so.* or /usr/lib64/libndpi.so.*), alternatively you can provide a custom path via env var (NDPI_LIB_PATH)

[ronygut]

@cardigliano , I'm using static linking and not dynamic linking for both ndpi and pfring.
I don't want to use shared objects.
It worked before.
I can't use it anymore?

[cardigliano]

@cardigliano , tried to call free flow also when getting slice timeout PFRING_FT_FLOW_STATUS_SLICE_TIMEOUT and exactly after timeout expired and getting indication of status = PFRING_FT_FLOW_STATUS_SLICE_TIMEOUT , the program crashes probably because I'm trying to access the user data in the flow.

Please do not release the user data when a flow slice is exported, as it is still referenced by the original flow

[cardigliano]

@cardigliano , I'm using static linking and not dynamic linking for both ndpi and pfring. I can't use it anymore?

FT is using dynamic linking (dlopen) anyway, please use .so

[ronygut]

@cardigliano , tried to call free flow also when getting slice timeout PFRING_FT_FLOW_STATUS_SLICE_TIMEOUT and exactly after timeout expired and getting indication of status = PFRING_FT_FLOW_STATUS_SLICE_TIMEOUT , the program crashes probably because I'm trying to access the user data in the flow.

Please do not release the user data when a flow slice is exported, as it is still referenced by the original flow

I'm not releasing the user data on flow export.
I'm trying to access it and it crashes.

[ronygut]

./autogen.sh --with-only-libndpi /configure --enable-static make

I can't use ./autogen.sh --with-only-libndpi /configure --enable-static make
It worked before

[ronygut]

NDPI_LIB_PATH

As I'm running from root NDPI_LIB_PATH will not work

[ronygut]

copied the libndpi.so* to /usr/lib
./ftflow -7 -i ens160
*** ndpi library version mismatch, expected API rev.11192 (4.11.0) found rev.42642 (4.11.0) ***
pfring_ft_create_table error

[ronygut]

@cardigliano I would appreciate it if you can fix the mismatch problem as I can't get the new IGNORE VLAN feature checked.
Thanks for your support.

[cardigliano]

You said you are using the stable branch (PF_RING 8.8.0 and nDPI 4.10) right?
It is strange in the output you provided it is looking for 4.11.0.
Please provide the output of ls -al /usr/lib/libndpi.so.*

[ronygut]

Yes, I'm using the stable (PF_RING 8.8.0 and nDPI 4.10)

-rwxr-xr-x 1 root root 3046360 Sep 5 08:34 /usr/lib/libndpi.so.4
-rwxr-xr-x 1 root root 3046360 Sep 5 08:34 /usr/lib/libndpi.so.4.11.0

[cardigliano]

It seems you did not install the ndpi .so from ndpi 4.10 (use packages or run make install)

[ronygut]

I downloaded the zip file from nDPI 4.10 and build everything myself.
the so files that was built from source are the ones I sent.
I can't do make install as the machine that I run it on is not a build machine.
I copied the so files as you instructed in the previous mail.
I prefer to built it myself and copy the relevant files before running the program.

[ronygut]

@cardigliano
Just checked ran the make in the ubuntu packages and got the same so file from deb content.
It contains the same 4.11.0 so version
Please advise.

dpkg -c ndpi_4.11.0-0_amd64.deb
/
drwxrwxr-x root/root 0 2024-09-05 11:04 ./usr/
drwxrwxr-x root/root 0 2024-09-05 11:04 ./usr/bin/
-rwxrwxr-x root/root 2916216 2024-09-05 11:04 ./usr/bin/ndpiReader
drwxrwxr-x root/root 0 2024-09-05 11:04 ./usr/lib/
-rwxrwxr-x root/root 2911608 2024-09-05 11:04 ./usr/lib/libndpi.so.4.11.0
drwxrwxr-x root/root 0 2024-09-05 11:04 ./usr/share/
drwxrwxr-x root/root 0 2024-09-05 11:04 ./usr/share/ndpi/
-rw-rw-r-- root/root 314587 2024-09-05 11:04 ./usr/share/ndpi/public_suffix_list.dat
lrwxrwxrwx root/root 0 2024-09-05 11:04 ./usr/lib/libndpi.so -> libndpi.so.4.11.0
lrwxrwxrwx root/root 0 2024-09-05 11:04 ./usr/lib/libndpi.so.4 -> libndpi.so.4.11.0

[ronygut]

@cardigliano , I probably downloaded the 4.10 development branch and not the the 4.10-stable branch.
I will clone 4.10-stable and check it again

[ronygut]

@cardigliano , OK, it works!
After I made sure to use the right branches.
Thanks for the support!

[ronygut]

I'm running the ftflow program and when ICMP protocol is decoded I see that sometimes the destination port is not 0. (in this case 771)
As ICMP has no ports, the source and destination ports should always be 0.
Why do I see non 0 destination ports?

[Flow] l7: ICMP, category: 0, srcMac: A8:46:9D:38:96:D5, dstMac: 00:08:E3:FF:FD:90, vlanId: 0, srcIp: 10.61.230.130, dstIp: 10.21.210.240, srcPort: 0, dstPort: 771, protocol: 1, tcpFlags: 0x00, c2s: { Packets: 1, Bytes: 146, First: 1725804668.687727, Last: 1725804668.687727 }, s2c: { Packets: 0, Bytes: 0, First: 0.0, Last: 0.0 }, status: idle-timeout, action: default

[cardigliano]

In case of ICMP, the destination port is set to (icmp_type * 256) + icmp_code, this is used to identify icmp requests between the same client/server

[ronygut]

@cardigliano

What was the exact hardware that you used to accomplish ~100Gbit/second with PF_RING FT + nDPI?
Number of CPU and type of CPU?
How much memory on the machine?
Did you use PF_RING ZC to get the packets?
How many consumer threads you used in PF_RING ZC?
How much Huge pages?
I would appreciate it if you can elaborate on this.
Can you send me a link to source code that you used for testing it?
Thanks,
Rony.

[cardigliano]

A sample configuration we used during recent tests in our lab is Intel Xeon Gold 6526Y with 8 x 16 GB DDR5 memory modules. As adapter at 100Gbit you can use Napatech (NT200A02) for best performance, NVIDIA ConnectX or Intel are also supported and perform good with PF_RING ZC drivers on average traffic at 100 Gbps.
As of the queues/threads configuration, we suggest 16 streams / RSS queues.

[ronygut]

@cardigliano

Do you mean 16 consumer threads bind to specific cores?
Regarding RSS, can you elaborate more on this?
Do you mean to load the Intel driver with RSS queues? (e.g. insmod ixgbe.ko RSS=16,16,16,16)
and do the hash in hardware instead of using hash in PF_RING ZC?
We will use Intel E810 NIC.