[Question] how to do traffic analysis on fritzbox

[joerg-wille]

Hi,
I want to log and visualize my private fritz!box router and found following script:
https://github.com/ntop/ntopng/blob/dev/tools/fritzdump.sh
Since I am on a Mac I installed ntopng with brew and started redis and followed this blog:
https://www.ntop.org/ntopng/how-to-use-ntopng-for-realtime-traffic-analysis-on-fritzbox-routers/

For some reason I then "should" start logging "lan" in the URL:
http://fritz.box/html/capture.html

Then I start the script and the WebUI.
As I understand, the script captures traffic on the Fritz!Box WWAN Interface "2-0" and forwards this to stdout where ntopng is reading it.

Trying to login into http://fritz.box as user dslf-config
Capturing traffic on Fritz!Box interface 2-0 ...
13/Jan/2020 18:26:28 [Ntop.cpp:1902] Setting local networks to 127.0.0.0/8
13/Jan/2020 18:26:28 [Redis.cpp:132] Successfully connected to redis 127.0.0.1@0
13/Jan/2020 18:26:28 [Redis.cpp:132] Successfully connected to redis 127.0.0.1@0
13/Jan/2020 18:26:28 [NetworkInterface.cpp:125] WARNING: Unable to read IPv4 address of stdin: SIOCGIFADDR: stdin: Device not configured
13/Jan/2020 18:26:29 [Ntop.cpp:1994] Registered interface stdin [id: 0]
13/Jan/2020 18:26:29 [main.cpp:302] PID stored in file /var/run/ntopng.pid
13/Jan/2020 18:26:29 [HTTPserver.cpp:1030] HTTPS Disabled: missing SSL certificate /usr/local/Cellar/ntopng/3.8.1/share/ntopng/httpdocs/ssl/ntopng-cert.pem
13/Jan/2020 18:26:29 [HTTPserver.cpp:1032] Please read https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to enable SSL.
13/Jan/2020 18:26:29 [Utils.cpp:592] User changed to nobody
13/Jan/2020 18:26:29 [HTTPserver.cpp:1199] Web server dirs [/usr/local/Cellar/ntopng/3.8.1/share/ntopng/httpdocs][/usr/local/Cellar/ntopng/3.8.1/share/ntopng/scripts]
13/Jan/2020 18:26:29 [HTTPserver.cpp:1202] HTTP server listening on 3000
13/Jan/2020 18:26:29 [main.cpp:399] Working directory: /var/tmp/ntopng
13/Jan/2020 18:26:29 [main.cpp:401] Scripts/HTML pages directory: /usr/local/Cellar/ntopng/3.8.1/share/ntopng
13/Jan/2020 18:26:29 [Ntop.cpp:403] Welcome to ntopng x86_64 v.3.8.191231 - (C) 1998-18 ntop.org
13/Jan/2020 18:26:29 [Ntop.cpp:413] Built on MacOSX 10.14.6
13/Jan/2020 18:26:29 [PeriodicActivities.cpp:72] Started periodic activities loop...
13/Jan/2020 18:26:29 [PeriodicActivities.cpp:113] Each periodic activity script will use 2 threads
13/Jan/2020 18:26:29 [NetworkInterface.cpp:2597] Started packet polling on interface stdin [id: 0]...

I do see "some" traffic in the WebUI but nothing useful e.g.: no active flows...

(I have asked the same question on Telegram ntop_community and will mark this "issue" solved if I do get support there.)

[cardigliano]

@joerg-wille where do you see "some traffic" in the web interface? Are you checking traffic stats from the Interface page? Could you provide some more detail? Thank you.

[joerg-wille]

But that's the only indication of the setup doing "something". If I modify the script to analyze other interfaces than the Fritz!Box WWAN Interface "2-0" or if I do not start logging "Lan" in the Fritz!Box then this number in the ntop WebUI does not change.

[cardigliano]

are you able to provide a small pcap, by replacing ntopng with tcpdump (e.g. | tcpdump -i - -w dump.pcap)

[joerg-wille]

I have tried:
| tcpdump -i - -w dump.pcap
and got following error:
tcpdump: -: No such device exists

Then I replaced -I with -r
wget --no-check-certificate -qO- $FRITZIP/cgi-bin/capture_notimeout?ifaceorminor=$IFACE&snaplen=&capture=Start&sid=$SID | tcpdump -w ./dump.pcap -r -

and this time I got following error:
tcpdump: unknown file format

I have no experience with tcpdump...

[joerg-wille]

I saw yesterday that on the interface side in the WebUI there also was a pcap download button, but it only creates an empty file.
How can it be that I see "Received Traffic" some MB, but in "Traffic Breakdown" "No traffic yet".

[simonemainardi]

as you are on OS X, it seems this is the reason why you get the unrecognized output error:

https://apple.stackexchange.com/questions/152682/why-does-tcpdump-not-recognise-piped-input/354139#354139

please follow the link to solve and generate the pcap. Thank you

[joerg-wille]

thanks, for this tip. But I am still not able to log any data.
I have installed current version from tcpdump with brew and added it before macOS version to path. So:
which tcpdump
/usr/local/opt/tcpdump/sbin/tcpdump

But, when I call the script after I exchanged the line to:
wget --no-check-certificate -qO- $FRITZIP/cgi-bin/capture_notimeout?ifaceorminor=$IFACE&snaplen=&capture=Start&sid=$SID| tcpdump -r - -w dump.pcap

sudo ./fritzdump.sh dslf-config "password"
Trying to login into http://fritz.box as user dslf-config
Capturing traffic on Fritz!Box interface 1-lan ...
tcpdump: truncated dump file; tried to read 4 file header bytes, only got 0

Did not find any solution to this.
I am using Lan Interface:
IFACE="1-lan"

and started logging in FritzPox on "lan" interface.

[simonemainardi]

please, try and download the pcap directly from inside ntopng. Go to the interface page and the click pcap download and attach it here.

[joerg-wille]

I have tried this already.
As I stated above, using the WebUI to download pcap produces empty files.
Or am using the WebUI wrong?

[cardigliano]

@joerg-wille you should be able to go to an hidden page in your fritzbox which is fritz.box/html/capture.html and download a pcap directly from there, could you try that?

[joerg-wille]

@cardigliano yes, this works. I have done this couple of times. I can download the file and then inspect this with Wireshark. Capturing with fritzbox gives the expected result.
The idea of using ntopng was to do this in real time and with a more user-friendly UI.
I am on MacOS Mojave 10.14.6 and I am doing following steps:

• start redis: redis-server /usr/local/etc/redis.conf &
• open fritz.box/html/capture.html and start logging on IF "lan"
• run sudo ./fritzdump.sh dslf-config "password"
• open localhost:3000 in Browser

And this is the output in terminal:
Trying to login into http://fritz.box as user dslf-config
Capturing traffic on Fritz!Box interface 1-lan ...
30/Jan/2020 15:22:15 [Ntop.cpp:1902] Setting local networks to 127.0.0.0/8
30/Jan/2020 15:22:15 [Redis.cpp:132] Successfully connected to redis 127.0.0.1@0
30/Jan/2020 15:22:15 [Redis.cpp:132] Successfully connected to redis 127.0.0.1@0
30/Jan/2020 15:22:15 [NetworkInterface.cpp:125] WARNING: Unable to read IPv4 address of stdin: SIOCGIFADDR: stdin: Device not configured
30/Jan/2020 15:22:16 [Ntop.cpp:1994] Registered interface stdin [id: 0]
30/Jan/2020 15:22:16 [main.cpp:302] PID stored in file /var/run/ntopng.pid
30/Jan/2020 15:22:16 [HTTPserver.cpp:1030] HTTPS Disabled: missing SSL certificate /usr/local/Cellar/ntopng/3.8.1/share/ntopng/httpdocs/ssl/ntopng-cert.pem
30/Jan/2020 15:22:16 [HTTPserver.cpp:1032] Please read https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to enable SSL.
30/Jan/2020 15:22:16 [Utils.cpp:592] User changed to nobody
30/Jan/2020 15:22:16 [HTTPserver.cpp:1199] Web server dirs [/usr/local/Cellar/ntopng/3.8.1/share/ntopng/httpdocs][/usr/local/Cellar/ntopng/3.8.1/share/ntopng/scripts]
30/Jan/2020 15:22:16 [HTTPserver.cpp:1202] HTTP server listening on 3000
30/Jan/2020 15:22:16 [main.cpp:399] Working directory: /var/tmp/ntopng
30/Jan/2020 15:22:16 [main.cpp:401] Scripts/HTML pages directory: /usr/local/Cellar/ntopng/3.8.1/share/ntopng
30/Jan/2020 15:22:16 [Ntop.cpp:403] Welcome to ntopng x86_64 v.3.8.191231 - (C) 1998-18 ntop.org
30/Jan/2020 15:22:16 [Ntop.cpp:413] Built on MacOSX 10.14.6
30/Jan/2020 15:22:16 [PeriodicActivities.cpp:72] Started periodic activities loop...
30/Jan/2020 15:22:16 [PeriodicActivities.cpp:113] Each periodic activity script will use 2 threads
30/Jan/2020 15:22:16 [NetworkInterface.cpp:2597] Started packet polling on interface stdin [id: 0]...
30/Jan/2020 15:22:17 [PcapInterface.cpp:253] Terminated packet polling for stdin
30/Jan/2020 15:22:21 [LuaEngine.cpp:1713] ERROR: Interface didn't reload custom categories on time [iface: stdin]

So, there is an Error reported. How can I fix that? Why gets polling terminated?

[cardigliano]

@joerg-wille it is likely polling terminates because of a truncated/unrecognzed pcap stream from the fritzbox (same as tcpdump). We need to figure out why the pcap you download from the fritzbox gui is well formed, while the one downloaded with wget is not recognized by tcpdump/ntopng..

[joerg-wille]

This make sense - how can I help?

[th0u]

still open this issue?
I have (successfully) tested set up capturing Fritzbox traffic in Ntopng via several ways on Centos.
To test if the "wget" works and receives traffic, remove in the script the piping to what ever behind wget and add -O /path/to/your/fb.pcap. This will tell wget to output to file an you can check if that works.
Additionally, while the wget is running (observe it using ps axf), go to FritzBox Gui and navigate to packet capture. There you should see that a capture is running.

If that works, you can use wget ... | /path/to/ntopng -i - ..
On my side, it looks this
/bin/wget --no-check-certificate -qO- $FRITZIP/cgi-bin/capture_notimeout?ifaceorminor=$IFACE&snaplen=&capture=Start&sid=$SID| /usr/local/bin/ntopng --community -m 192.168.178.0/24,200x:xx:xxxx:xx00::/56 -G /var/run/ntopng.pid -d /var/lib/ntopng -1 /usr/share/ntopng/httpdocs -2 /usr/share/ntopng/scripts -i -

I also have a monitor script which (hopefully) restarts the capture if it fails.

Another way, that I tested was to output wget using -O into a FIFO (mkfifo /run/ntopfb.pcap) and use nprobe to read out the fifo and forward it via zmq to Ntopng. But it missed some traffic and had no pcap download in ntopng.
Hope it helps.
Kind regards, T.

[cardigliano]

@th0u thank you (we do not have an active fritzbox router atm for testing, thus your help is really useful)

[henfri]

@th0u

I also have a monitor script which (hopefully) restarts the capture if it fails.

For me, it fails rather regularly. Would you mind sharing the script (maybe even via a PR)?

Regards,
Hendrik

[th0u]

Hi Hendrik

the main script is in this repository (ntopng/tools/fritzdump.sh).
Its is enhanced to create a PID File, so I can monitor it.
I put this code in Line 2 ff
mypidfile=/run/fritzdump2.sh.pid
#Ensure PID file is removed on program exit.
trap "rm -f -- '$mypidfile'" EXIT

and line 47
echo $$ > $mypidfile

The actual monitoring script is test-fritzdump.sh:

#!/bin/bash

for pid in $(pidof -x fritzdump2.sh); do
if [ $pid != $$ ]; then
echo "[$(date)] : fritzdump2.sh : Process is already running with PID $pid"
exit 1
fi
done

echo "start fritzdump"

/usr/sbin/fritzdump2.sh &

test-fritzdump.sh is run by cron every minute.

Hope it helps.
T.

[henfri]

Hello @th0u ,

thank you!
In my case it seems that ntopng -and consequently fritzdump.sh- does NOT end. Instead it seems that no more data comes via wget from FritzBox.
ntopng keeps running, so that -as far as I understand your script and the modification- the pidfile would not be deleted in my case.

Or do I miss something?

Greetings,
Hendrik

[th0u]

yes, in that case, it would unfortunately not help.

[henfri]

Here my example:

root@homeserver:~/fritzdump# ps aux | grep wget
root 7162 0.0 0.0 54620 5228 pts/1 S+ 15:24 0:00 wget --no-check-certificate -qO- http://fritz.box/cgi-bin/capture_notimeout?ifaceorminor=2-0&snaplen=&capture=Start&sid=xxxxx

and in case of failure

ps aux | grep "wget|fritz|ntop"
root 7122 0.0 0.0 12744 3092 pts/1 S+ 15:24 0:00 /bin/bash ./fritzdump.sh henfri till1234
nobody 7163 70.0 1.5 1584828 193548 pts/1 Sl+ 15:24 173:49 ntopng -w 3001 -i -
root 21868 1.4 0.2 343784 34864 ? Ds 19:32 0:00 /usr/local/bin/ntopng /run/ntopng.conf
root 22103 0.0 0.0 14320 976 pts/5 S+ 19:32 0:00 grep wget|fritz|ntop

A pidfile for wget would be needed.

[th0u]

Hello Hendrik,
the output line with pid 21868: Do you run ntopng as a service too? It seems it runs twice (pid 7163 and 21868)
In my case, I do not run ntopng as a service.

[henfri]

Hello,

not knowingly. Maybe it was started at install... Although it should then not run due to port 3000 used already by grafana on my computer.

Greetings,
Hendrik

[th0u]

I'd disable ntopng service then.
This is how I start ntopng in fritzdump.sh:
/bin/wget --no-check-certificate -qO- $FRITZIP/cgi-bin/capture_notimeout?ifaceorminor=$IFACE\&snaplen=\&capture=Start\&sid=$SID | /usr/local/bin/ntopng --community -m 192.168.178.0/24 -G /var/run/ntopng.pid -d /var/lib/ntopng -1 /usr/share/ntopng/httpdocs -2 /usr/share/ntopng/scripts -i -

[henfri]

Thanks.

I have edited the script now like this:

[no change up to here]
# In case you want to use tshark instead of ntopng
#wget --no-check-certificate -qO- $FRITZIP/cgi-bin/capture_notimeout?ifaceorminor=$IFACE\&snaplen=\&capture=Start\&sid=$SID | /usr/bin/tshark -r -

start_ntop () {
  echo "Starting ntopng and wget"
  (wget --no-check-certificate -qO- $FRITZIP/cgi-bin/capture_notimeout?ifaceorminor=$IFACE\&snaplen=\&capture=Start\&sid=$SID & echo $! >&3 ) 3>pidwget | ntopng -w 3001 -i - &
  PID_WGET=$(<pidwget)
  PID_NTOP=$!
}

while :
do
  if [ -n "$PID_NTOP" -a -e /proc/$PID_NTOP ] ; then
    echo "NTOP is running"
    if [ -n "$PID_WGET" -a -e /proc/$PID_WGET ] ; then
     echo "wget is running"
    else
     echo "wget is not running"
     echo $PID_WGET
     start_ntop
    fi
  else
    echo "NTOP not running"
    echo $PID_NTOP
    start_ntop
  fi
    sleep 60
done

[henfri]

Testing the script gives me regular restarts -more regular than wget was failing previously.

./fritzdump.sh: Zeile 75: 11547 Exited ( wget --no-check-certificate -qO- $FRITZIP/cgi-bin/capture_notimeout?ifaceorminor=$IFACE\&snaplen=\&capture=Start\&sid=$SID & echo $! 1>&3 ) 3> pidwget
     11548 Segfault | ntopng -w 3001 -i -

Hmmm

[alanmilinovic]

I can confirm as well, it is not working.
This is what I get as well:
26/Feb/2020 11:08:01 [LuaEngine.cpp:12167] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/minute.lua][/var/lib/ntopng/plugins/ts_schemas/score/min.lua:1: unexpected symbol near '<\135>']

[emanuele-f]

Please open a separate issue for this and report the output of ntopng --version. Please upgrade ntopng to the latest version before opening.

[emanuele-f]

Closing for inactivity. Please reopen if necessary.