Unable to add pfsense as monitored snmp device

[fsamareanu]

Hello,

I'm trying to add pfsense as snmp device but I get this:

No answer from host 192.168.1.1: please check that SNMP is up and the community is set to the correct value.

Parameters used: community, v1

snmpwalk works when ran from the same host:

root@nuc:~# snmpwalk -c s3cr3t 192.168.1.1 -v1|head -1
iso.3.6.1.2.1.1.1.0 = STRING: "pfSense oinky-pfsense.local.lan 2.4.3-RELEASE pfSense FreeBSD 11.1-RELEASE-p7 amd64"

Any guidelines? Is probing a freebsd host even supported?

[emanuele-f]

Hi, what ntopng version are you using (ntopng --version)?

[fsamareanu]

Hello,

Latest dev on Ubuntu.

[fsamareanu]

3.3.180405

[emanuele-f]

Can you please post a screenshot of the page with the options you use to add your SNMP device?

[fsamareanu]

[emanuele-f]

What's the output of snmpwalk -v1 -c s3cr3t 192.168.1.1 1.3.6.1.2.1.1.5.0 ?

[fsamareanu]

root@nuc:~# snmpwalk -v1 -c s3cr3t 192.168.1.1 1.3.6.1.2.1.1.5.0
iso.3.6.1.2.1.1.5.0 = STRING: "oinky-pfsense.local.lan"

[emanuele-f]

What about this query? snmpwalk -v1 -c s3cr3t 192.168.1.1 1.3.6.1.2.1.17 . Please also post the output of tcpdump -ne udp port 169 while adding your SNMP device from the gui

Edit: sorry, port 161: tcpdump -ne udp port 161

[fsamareanu]

The query doesn't return anything. Tcpdump output below (asuming you wanted to intercept snmp traffic on port 161):

23:05:29.659509 94:c6:91:17:20:d0 > 00:0e:c4:d1:d2:4a, ethertype IPv4 (0x0800), length 94: 192.168.1.7.43372 > 192.168.1.1.161: C="s3cr3t" GetRequest(34) .1.3.6.1.2.1.1.5.0

On the same host i have a cacti instance monitoring pfsense just fine.

[fsamareanu]

Hello,

Here's both a snmpwalk and a request through GUI:

root@nuc:~# tcpdump -i eno1 -ne udp port 161 and host 192.168.1.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes

--Here i did a snmpwalk

09:31:35.192913 94:c6:91:17:20:d0 > 00:0e:c4:d1:d2:4a, ethertype IPv4 (0x0800), length 85: 192.168.1.7.48740 > 192.168.1.1.161: C="s3cr3t" GetNextRequest(28) .1.3.6.1.2.1.1.5.0
09:31:35.193268 00:0e:c4:d1:d2:4a > 94:c6:91:17:20:d0, ethertype IPv4 (0x0800), length 85: 192.168.1.1.161 > 192.168.1.7.48740: C="s3cr3t" GetResponse(28) .1.3.6.1.2.1.1.6.0=""
09:31:35.193329 94:c6:91:17:20:d0 > 00:0e:c4:d1:d2:4a, ethertype IPv4 (0x0800), length 85: 192.168.1.7.48740 > 192.168.1.1.161: C="s3cr3t" GetRequest(28) .1.3.6.1.2.1.1.5.0
09:31:35.193543 00:0e:c4:d1:d2:4a > 94:c6:91:17:20:d0, ethertype IPv4 (0x0800), length 108: 192.168.1.1.161 > 192.168.1.7.48740: C="s3cr3t" GetResponse(51) .1.3.6.1.2.1.1.5.0="oinky-pfsense.local.lan"

--Now from GUI

09:31:52.631137 94:c6:91:17:20:d0 > 00:0e:c4:d1:d2:4a, ethertype IPv4 (0x0800), length 94: 192.168.1.7.36671 > 192.168.1.1.161: C="s3cr3t" GetRequest(34) .1.3.6.1.2.1.1.5.0

I can also provide a pcap capture if required.

[fsamareanu]

Ok so i decided to call it quits and replaced the ancient standard snmp daemon running on pfsense with net-snmp package and this one works. Do you want to continue troubleshooting this (I have no issues doing tests if required)?

[emanuele-f]

It would be great if you could test it on the old deamon. So you don't get a "GetResponse" to the "GetRequest" on 1.3.6.1.2.1.1.5.0 when you add the device from the gui, but you get that response while you run snmpwalk manually?

[fsamareanu]

That seems to be the case, yes. If required I can provide remote access to the environment.

[emanuele-f]

Thank you. Can you produce two pcap files, one with the SNMP traffic generated by manual snmp walk and another with traffic while adding the device from the ntopng gui and send me at faranda@ntop.org ?

[fsamareanu]

Done.

[emanuele-f]

Thank you. I've analyzed the SNMP get-request packets and the only difference between them is the size of some fields, which are 1 byte size in snmpwalk requests while 4 bytes size in ntopng requests. As SNMP fields are correcly encoded in type,length,value by ntopng, I think there is probably a limitation on the ancient SNMP daemon you were using.

[fsamareanu]

I'll raise this with the pfsense folks. Although I have a workaround maybe it's a trivial fix.

[fsamareanu]

@emanuele-f I raised this under https://redmine.pfsense.org/issues/8460. Would you mind giving me your analysis so I can pass the message to pfsense developers?

[emanuele-f]

@fsamareanu sure. In SNMP some fields are encoded with TLV (Type Length Value), where the length specifies how many bytes to read on the next field. The daemon you are using seems to ignore the "Length" value from some SNMP fields (at least in version, error-status, error-index fields) and assume length 1. This works well with snmpwalk which uses a 1 byte values but it does not with ntopng which uses 4 bytes values.

[fsamareanu]

Hello,

This ended up being reported upstream to FreeBSD team at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227502 .

The comment I got there was this:

Eugene Grosbein freebsd_committer 2018-04-14 21:38:15 UTC
I've reproduced the problem building and running third-party/snmp/test.c from net/ntopng source tree. I run bsnmpd in debug mode:

/usr/sbin/bsnmpd -p /var/run/snmpd.pid -d -D dump,trace=0x30000000

Incoming SNMPv1 GetRequest as captured and decoded by tcpdump:

04:15:32.993260 IP (tos 0x0, ttl 62, id 21558, offset 0, flags [none], proto UDP (17), length 81)
X.X.X.X.46351 > X.X.X.X.Y: [udp sum ok] { SNMPv1 C="xxxxxxx" { GetRequest(34) R=1 .1.3.6.1.2.1.1.5.0 } }

bsnmpd fails to parse it producing errors:

snmpd[45132]: ASN.1: non-minimal integer at 00 00 00 00 04 07 72 65 77 6f 72 74 68 a0 22 02 04 00 00 00 01 02 04 00 00 00 00 02 04 00 00 00 00 30 0e 30 0c 06 08 2b 06 01 02 01 01 05 00 05 00
snmpd[45132]: SNMP: cannot decode version

ntopng uses bundled copy of library https://github.com/ejrh/snmp to encode SNMP data into packets and this library seems to produce incorrect DER/ASN.1 packets always encoding integers with 4 bytes per value. The library itself is pretty old, it was not updated for 6 years.

snmpwalk, on the other hand, produces correct requests and bsnmpd answers just fine.

It seems, net-snmpd tolerates such standard violation but bsnmpd does not. Please note that other modern software tend to stick to strict validation too.
For example, golang's library encoding/asn1 rejects such invalid "non-minimal integer encodings" since version 1.7: https://golang.org/doc/go1.7

[fsamareanu]

--- Comment #3 from Eugene Grosbein eugen@freebsd.org ---
(In reply to Florin Samareanu from comment #2)

The problem is more complex. Indeed, "In SNMP some fields are encoded with TLV
(Type Length Value), where the length specifies how many bytes to read on the
next field" - that's true. But ASN.1/DER encoding standard states that if
integer value is small enought to be fit in single byte, it MUST be encoded
with single byte and using 4 bytes is not allowed for such case by this
standard.

snmpwalk can send small or large integers and it encodes them just right using
noted "minimal integer" encoding: 1 or more bytes correspongingly. bsnmpd
parses both of small and large TLV fileds just fine.

On the other hand, ejrh/snmp library used by ntopng ALWAYS encodes integers
using 4 bytes, even for small values (R=1 in the test above). That is standard
violation that does not pass bsnmpd's strict validation, but pass relaxed
validation of net-snmpd.

[emanuele-f]

@fsamareanu thank you for your feedback, we are reviewing the issue

[simonemainardi]

thanks for pointing this issue out.

I was able to reproduce using FreeBSD 11.

root@osboxes:/usr/home/osboxes # /usr/sbin/bsnmpd -p /var/run/snmpd.pid -d -D dump,trace=0x30000000

snmpd[877]: ASN.1: non-minimal integer at 00 00 00 00 04 06 70 75 62 6c 69 63 a0 22 02 04 5b 11 55 b2 02 04 00 00 00 00 02 04 00 00 00 00 30 0e 30 0c 06 08 2b 06 01 02 01 01 05 00 05 00
snmpd[877]: SNMP: cannot decode version




snmpd[877]: ASN.1: non-minimal integer at 00 00 00 00 04 06 70 75 62 6c 69 63 a0 22 02 04 5b 11 5d 82 02 04 00 00 00 00 02 04 00 00 00 00 30 0e 30 0c 06 08 2b 06 01 02 01 01 05 00 05 00
snmpd[877]: SNMP: cannot decode version





snmpd[877]: ASN.1: zero-length integer at 04 06 70 75 62 6c 69 63 a0 1a 02 04 5b 11 68 3e 02 00 02 00 30 0e 30 0c 06 08 2b 06 01 02 01 01 05 00 05 00
snmpd[877]: SNMP: cannot decode version

The snmp library was poorly encoding all the integers using 4-bytes. I've fixed the issue accordingly.