metric gathering slower than snmpbulkwalk

[Andor]

with just snmpbulkwalk:

[andor@hostname ~]$ time snmpbulkwalk -v 2c -c community $target 1.3.6.1.2.1.2 > /dev/null

real    0m8.667s
user    0m0.066s
sys     0m0.026s
[andor@hostname ~]$ time snmpbulkwalk -v 2c -c community $target 1.3.6.1.2.1.31.1.1 > /dev/null

real    0m9.468s
user    0m0.072s
sys     0m0.016s
[andor@hostname ~]$ time snmpbulkwalk -v 2c -c community $target 1.3.6.1.2.1.1.3 > /dev/null

real    0m0.025s
user    0m0.015s
sys     0m0.005s

sum is ~18sec

with snmp_exporter with default config:

$ time curl localhost:9116/snmp?target=$target > metrics
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  363k  100  363k    0     0  14797      0  0:00:25  0:00:25 --:--:-- 76211

real    0m25.171s
user    0m0.004s
sys     0m0.016s

[brian-brazil]

Try -Nr 25 on the snmpbulkwalk. I smell hardware that is slower with increased batch sizes.

[Andor]

Looks like my snmp-tools have no options starting with N.

[brian-brazil]

Might be -Cr

[Andor]

Nope, looks it work even faster:

[andor@hostname ~]$ time snmpbulkwalk -C r25 -v 2c -c community $target 1.3.6.1.2.1.2 > /dev/null

real    0m8.356s
user    0m0.052s
sys     0m0.011s
[andor@hostname ~]$ time snmpbulkwalk -C r25 -v 2c -c community $target 1.3.6.1.2.1.31.1.1 > /dev/null

real    0m7.783s
user    0m0.049s
sys     0m0.009s

[brian-brazil]

Hmm, those should be the same speed. Is anything odd apparent from a Tcpdump?

[RichiH]

Shot in the dark, but might one be going for the A and the other for the AAAA record? A pcap of each variant would help indeed. Richard Sent by mobile; excuse my brevity.

[truedays]

Just started looking at Prometheus snmp_exporter, but here's my theory..

net-snmp's snmpbulkwalk defaults are:
retry (-r) 5
max-repetitions (-Cr) 10
timeout (-t) 1

While collector.go has
snmp.Retries = 3
snmp.MaxRepetitions = 25
snmp.Timeout = time.Second * 60

timeout is the time between the initial get-request and the corresponding get-response until it sends another duplicate get-request (UDP-style handshake).

I think you'll get comparable times if you more closely mirror the SNMP arguments used by collector.go

COMMUNITY='public'
TARGET='127.0.0.1'
time { for i in 1.3.6.1.2.1.{2,31.1.1,1.3}; do snmpbulkwalk -v2c -c ${COMMUNITY} -r3 -Cr25 -t60 ${TARGET} ${i}& done; wait; }

[brian-brazil]

I don't think that explanation does it, as we'd be seeing times of over a minute if that were the case.

[RichiH]

Andrew, could you provide the pcap? Richard Sent by mobile; excuse my brevity

[Andor]

Sorry, but i can't share pcap's.
Sorry, but it was my mistake, compared with different targets :D