Nfdump 1.7.5 incompatibilities

[falz]

Recent install, building a new server with nfsen-ng instead of nfsen.

All flows are collecting properly, graphs / rrd's updating. When I issue any query, i get

 something went wrong. firstSeen,duration,proto,srcAddr,srcPort,dstAddr,dstPort,packets,bytes,flows

and in some queries i get that as well as this, which is what looks like because $output is empty (the above issue? unsure)

Warning: Undefined array key 0 in /opt/nfsen-ng/nfsen-ng-0.3.1/backend/processor/Nfdump.php on line 175

I was running the HEAD/master version, and then switched to the last release 0.3.1 and get the same issue. I've turned on debugging which gives the nfdump command in the apache web logs - i can run it no probem as the www-data (apache) user, so permissions are ok. the values after "something went wrong" re the .csv headers - they change depending on the query, but i never see results.

Guessing this is something wrong compatibility wise - my system is I followed the INSTALL.md instructions (which were pretty great)

• Ubuntu 20.4.x
• PHP 8.1
• nfdump 1.7.5 (which, btw, is only two weeks old)

Here's sample data output from cli, in case the output from nfdump is somehow the problem

firstSeen,duration,proto,srcAddr,srcPort,dstAddr,dstPort,packets,bytes,flows
2024-11-08 12:00:22.272,0.000,6,10.150.47.2,45708,34.199.107.131,443,4096,212992,1
2024-11-08 12:00:03.328,31.744,6,10.150.47.2,443,69.129.204.33,59649,12288,4648960,1
2024-11-08 12:01:55.456,10.240,6,10.150.47.2,443,69.129.204.33,59649,8192,425984,1
2024-11-08 12:03:14.560,0.000,6,10.150.47.2,443,69.129.204.33,59649,4096,1167360,1
2024-11-08 12:03:51.424,0.000,6,10.150.47.2,443,69.129.204.33,59649,4096,1818624,1
2024-11-08 12:04:30.848,102.912,6,10.150.47.2,443,69.129.204.33,59649,49152,54398976,1
2024-11-08 12:07:03.680,141.568,6,10.150.47.2,443,69.129.204.33,59649,61440,57442304,1
2024-11-08 12:09:57.248,145.664,6,10.150.47.2,443,69.129.204.33,59649,69632,54509568,1
2024-11-08 12:09:57.248,275.200,6,10.150.47.2,443,69.129.204.33,59649,53248,37867520,1

[falz]

Did a little more investigation here - i believe there are many issues, and perhaps the main one is the way nfsen 1.7.x outputs data.

Eyeballing processor/Nfdump.php does a lot of things with the output such as "remove the 3 summary lines at the end of the csv output" - these three lines aren't there in 1.7.5, but are in 1.6.x. haven't tested any other versions, will do more looking around next week but it seems to simply not like the output.

Strangely though, the "if last element contains a colon, it's not a csv" portion of code doesn't even work - it seems that that returns a non csv as a bunch of text, but still errors.

if I choose to download a csv file i get a nice error:

<br />
<b>Fatal error</b>:  Uncaught TypeError: fputcsv(): Argument #2 ($fields) must be of type array, string given in /opt/nfsen-ng/gitmaster/backend/api/Api.php:103
Stack trace:
#0 /opt/nfsen-ng/gitmaster/backend/api/Api.php(103): fputcsv()
#1 /opt/nfsen-ng/gitmaster/backend/index.php(12): mbolli\nfsen_ng\api\Api-&gt;__construct()
#2 {main}
  thrown in <b>/opt/nfsen-ng/gitmaster/backend/api/Api.php</b> on line <b>103</b><br />

[mbolli]

In recent nfdump versions there has been mentions of a json output. Maybe this would be better suited...

[falz]

I went from NFdump 1.7.5 -> 1.7.4 and things actually work now, so there's some incompatibility w/ 1.7.5. And indeed it says json output is supported since 1.6.17.

However, I also get "Unknown Extension ##" in the nfsen command output. This also comes through if i run the command from CLI as well. Guessing this is an Nfcapd/Nfdump issue, but perhaps they could be filtered out on the Nfsen-ng side?

[falz]

Additional note - the graphs for "ports" wasnt working either - and now it is.

[phaag]

I went from NFdump 1.7.5 -> 1.7.4 and things actually work now, so there's some incompatibility w/ 1.7.5. And indeed it says json output is supported since 1.6.17.

However, I also get "Unknown Extension ##" in the nfsen command output. This also comes through if i run the command from CLI as well. Guessing this is an Nfcapd/Nfdump issue, but perhaps they could be filtered out on the Nfsen-ng side?

You have incompatible version of the collector nfcapd and the cli tool nfdump. The collector is newer.

[falz]

Indeed as i had just downgraded. 1.7.4 -> 1.7.5

After waiting a day and doing a few tests on data since, downgrading, i am not getting this error.

So perhaps this ticket should be renamed to "nfsen-ng incompatible with nfdump 1.7.5 - want me to open a new issue for that and close this?

[mbolli]

No let's leave it in here, I renamed the issue.
Also thanks @phaag for chiming in.

[d4nys3k]

These commits seems to be cause. CVS header in nfdump changed and broke nfsen-ng...
phaag/nfdump@8b1416f
phaag/nfdump@23c7f4a

[falz]

^ Just confirming this is likely CSV (as in Comma Separated File) header, not CVS (as in RCS). Was confused for a moment.

[d4nys3k]

^ Just confirming this is likely CSV (as in Comma Separated File) header, not CVS (as in RCS). Was confused for a moment.

sure, it's my typo :) CSV header (first line) in nfdump output was changed from ts,td,sa,da,... to new form firstSeen,duration,srcAddr,dstAddr,... (shortened) with -o 'csv' output format. Tested just by calling similar nfdump command over command-line and looking to headers in output of v1.7.1 compared to v1.7.6. Then I looked for commits in nfdump probably causing such problem... and I decided that nfsen-ng seems to have problem on it's frontend with new output, as old headers in CSV are expected here (output from nfdump is correctly sent in json from the backend, first line is issued command, second CSV header and rest are data).

[dgeo]

Tried to adapt here with no success at this time…
https://github.com/dgeo/nfsen-ng/tree/testfix-nfsen175

It seems the main problem resides in backend not parsing correctly data

[dgeo]

In recent nfdump versions there has been mentions of a json output. Maybe this would be better suited...

This could simplify everything ! I just miss js-skills and time here :/