Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ndpi-netfilter (netfilter-2.6 branch) doesn't recognize YouTube, GMail, sometimes WhatsApp #51

Closed
clebig opened this issue Mar 26, 2019 · 27 comments
Closed

ndpi-netfilter (netfilter-2.6 branch) doesn't recognize YouTube, GMail, sometimes WhatsApp #51

clebig opened this issue Mar 26, 2019 · 27 comments

Comments

@clebig
Copy link

clebig commented Mar 26, 2019

Hi,

I cannot get ndpi-netfilter (netfilter-2.6 branch) to recognize GMail, YouTube and WhatsApp.
I guess there are other protocols that are not recognized but at least I'm sure of those ones.

It's likely a libndpi related problem because example/ndpiReader cannot detect them neither.

This is true at least since 0a0ff0d.

Clément
@vel21ripn
Copy link
Owner

The flow_info branch is based on nDPI 2.8-stable. Maybe the problem is fixed there?

@clebig
Copy link
Author

clebig commented Apr 1, 2019 via email •
edited
Loading

@clebig
Copy link
Author

clebig commented Apr 1, 2019

By the way, I have opened an issue on ntop/nDPI regarding WhatsApp, I don't know when and if it would be treated someday : ntop#683

@clebig
Copy link
Author

clebig commented Apr 2, 2019

I confirm that with the branch flow_info, it still doesn't work correctly.

I have compiled nDPI 2.8, dev, 2.6, 2.2 and fed ndpiReader with my pcaps for Maps, Youtube, Whatsapp Gmail recognition, and I couldn't get them detected in any version.

Do you have these protocols detected ?

@vel21ripn
Copy link
Owner 

egrep -i 'whatsapp|youtube|maps|gmail' /proc/net/xt_ndpi/proto
7a        7a/000001ff GMail            # 0 debug=0
7b        7b/000001ff GoogleMaps       # 0 debug=0
7c        7c/000001ff YouTube          # 0 debug=0
88        88/000001ff YouTubeUpload    # 0 debug=0
8e        8e/000001ff WhatsApp         # 1658 debug=0
bd        bd/000001ff WhatsAppVoice    # 1689751 debug=0
f2        f2/000001ff WhatsAppFiles    # 117 debug=0

Gmail, GoogleMaps and YouTube* not detected :(
Perhaps the file ndpi_content_match.c.inc is very outdated.
Can you send sample traffic files?

@clebig
Copy link
Author

clebig commented Apr 2, 2019 •
edited
Loading

Oups... my bad !
I was using the VM as gateway but... for IPv4 ! And I have a dual stack enabled. Facebook, Whatsapp, Google/Maps/Drive/Docs/Gmail are provided on IPv6 too ! So the traffic wasn't caught by the VM during tcpdump.
Now I have deactivated IPv6 on my computer, these traffics are well recognized by ndpiReader.

Unfortunately, xt_ndpi doesn't detect them. Any google service is accounted only by the Google entry. Whatsapp, Youtube, Docs, Drive, Gmail as still not detected.

@clebig
Copy link
Author

clebig commented Apr 2, 2019

What's your GLIBC version ?

@vel21ripn
Copy link
Owner

glibc 2.21 and 2.27 (Slackware 14.1+ and Slackware-15.0 pre)
When building with glibc-2.27, a strange error occurs, but it can be avoided if you add the tzset () call at the beginning of main() in ndpiReader.c. Then there is no sigsegv in the output of statistical data after processing the file.
IPv6 is supported by xt_ndpi. I have not had time to test IPv6 operation.
It is unfortunate that the nTOP project does not collect ipv6 traffic samples for testing.

@clebig
Copy link
Author

clebig commented Apr 2, 2019

Yep, I wasn't saying that IPv6 was not caught by xt_ndpi, just that my ipv6 traffic was not routed through this VM because I only added an ipv4 default route to it and because the services like Google and WhatsApp are also IPv6, the usual ipv6 gateway was used instead of the VM.

@clebig
Copy link
Author

clebig commented Apr 5, 2019 •
edited
Loading

I've just tested on Ubuntu Server 18.04 LTS (first time of my life I install it :-D), kernel 4.15 glibc 2.27, GCC 7.3, ndpi-netfilter flow_info branch. It gets built without error but It still doesn't work.

@vel21ripn
Copy link
Owner

Part of the traffic cannot be determined by signatures, it can be determined indirectly by addresses ( ip/AS ) and ports. This information in ntop/nDPI is extremely rarely updated.
Can you send traffic samples that are defined incorrectly?
If you have data on the availability of ip / port for a specific protocol or on the inconsistency of the available data in ndpi_content_match.c.inc, make a “Push request”.
This is the only way to improve this project.

@clebig
Copy link
Author

clebig commented Apr 5, 2019 via email •
edited
Loading

@clebig
Copy link
Author

clebig commented Apr 5, 2019 via email •
edited
Loading

@vel21ripn
Copy link
Owner

But the various services are correctly detected by ndpiReader with the pcaps recorded when these apps where not detected in live by xt_ndpi. So it doesn't seem related to ndpi itself.

Can you send me traffic to the address vel21ripn at gmail dot com in pcap format which is not detected by xt_ndpi and is defined in ndpiReader?

@vel21ripn
Copy link
Owner

vel21ripn commented Apr 9, 2019 •
edited
Loading

I applied ntop#687 and ntop#688 (branch flow_info).

@clebig
Copy link
Author

clebig commented Apr 9, 2019 via email

@vel21ripn
Copy link
Owner

I can process the pcap file in ndpi-netfilter. I have a working set of scripts and a program similar to tcpreplay, with which I tested the identity of the ndpiReader and ndpi-netfilter results.
The scheme is as follows:
pcap -> tapsend -> netfilter -> -j NDPI - set-mark -j NFLOG -> ndpiReader
I have a modified ndpiReader that can get a protocol label from the NFLOG and compare it with the protocol detected by ndpiReader.
ndpi-nefilter gives bad results if it does not see traffic in both directions (for example, only incoming or outgoing traffic passes through it).

@vel21ripn
Copy link
Owner

See #54

@clebig
Copy link
Author

clebig commented May 6, 2019

This seems everything is mostly solved. I can get YouTube, WhatsApp, GMail and other services recognized but WhatsApp voice is still not detected by xt_ndpi. It is however correctly detected by ndpiReader.

@vel21ripn
Copy link
Owner

I did not backporting changes from flow_info to netfilter-2.6

@clebig
Copy link
Author

clebig commented May 7, 2019 via email

@vel21ripn
Copy link
Owner

We have "WhatsApp voice" traffic examples ?

@clebig
Copy link
Author

clebig commented May 9, 2019

sure !
whatsapp_voice.pcap.gz

@vel21ripn
Copy link
Owner

I deleted the packages with addresses 239.255.255.250 and 10.144.172.200.
ndpiReader result

Detected protocols:
        Unknown              packets: 41            bytes: 3526          flows: 3
        Google               packets: 4             bytes: 252           flows: 1
        WhatsApp             packets: 260           bytes: 31767         flows: 3
        Messenger            packets: 61            bytes: 6996          flows: 13
        WhatsAppVoice        packets: 1804          bytes: 281947        flows: 2

        1       UDP 192.168.0.175:40613 <-> 157.240.21.51:3478 [proto: 189/WhatsAppVoice][cat: VoIP/10][844 pkts/162698 bytes <-> 882 pkts/105582 bytes]
        3       UDP 192.168.0.175:40911 <-> 157.240.21.51:3478 [proto: 189/WhatsAppVoice][cat: VoIP/10][8 pkts/896 bytes <-> 70 pkts/12771 bytes]

/proc/net/xt_ndpi/flows

1557734980 1557734980 4 17 192.168.0.175 40613 157.240.21.51 3478 150630 93234 844 882 I=129,0 P=WhatsAppVoice
1557734980 1557734980 4 17 192.168.0.175 40911 157.240.21.51 3478 772 11791 8 70 I=129,0 P=WhatsAppVoice

I do not see a problem yet.

@clebig
Copy link
Author

clebig commented May 13, 2019

Do you use your modified ndpiReader that reads labels from NFLOG ?

How can I read /proc/net/xt_ndpi/flows ? I can't cat on it. I have tried to set ndpi_enable_flow at insmod time but it's giving me something that is not what I'm looking for.

@vel21ripn
Copy link
Owner

Do you use your modified ndpiReader that reads labels from NFLOG ?

With the advent of flow_info, reading the NFLOG can only be useful for debugging purposes.
ndpiReader from my branches can read NFLOG.

How can I read /proc/net/xt_ndpi/flows ? I can't cat on it. I have tried to set ndpi_enable_flow at insmod

Please read the first 20 lines from ndpi-netfilter/FLOW_INFO.txt

@clebig
Copy link
Author

clebig commented May 13, 2019

Oups, sorry, wonderful explanation :)

@someonebw someonebw mentioned this issue Apr 16, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vel21ripn @clebig