Flows aggregate field causes segmentation fault

[thetooth]

I'm currently setting up a router using the vyos distro and have encountered a problem a few others seem to be running into, that is flow accounting not working on certain hardware. I did some digging and tracked the issue down to this project, the configuration is generated by some tooling behind the scenes and results in this:

# Genereated from VyOS configuration
daemonize: true
promisc: false
pidfile: /var/run/uacctd.pid
uacctd_group: 2
uacctd_nl_size: 2097152
snaplen: 128
aggregate: in_iface,src_mac,dst_mac,vlan,src_host,dst_host,src_port,dst_port,proto,tos,flows
plugin_pipe_size: 10485760
plugin_buffer_size: 10485
imt_path: /tmp/uacctd.pipe
imt_mem_pools_number: 169
plugins: memory

The uacctd process crashes out immediately with this configuration, either in core or plugin/memory. Disabling the IMT plugin allows other flow exports to work.

After some tests with different or default buffer sizes (in case these were completely wrong), I found the issue seems to be in the aggregate line, specifically the use of the flows counter.

Crashes

aggregate: in_iface,src_mac,dst_mac,vlan,src_host,dst_host,src_port,dst_port,proto,tos,flows

Works

aggregate: in_iface,src_mac,dst_mac,vlan,src_host,dst_host,src_port,dst_port,proto,tos

Not sure what the deal would be or if there is some reason this counter cannot be used with the IMT plugin but it would be nice to have this work, the reason being is the vyos guys are unlikely to want to change the config since most of there customers are using it purely for netflow export (I assume flows aggregate is used for that?), I however think the pmacct output is much more readable than some flashy graph that only updates once every 15 minutes. If no one can reproduce I can try and get a stack trace but might take me some time to setup.

Version
1.7.6-git [RELEASE]
1.7.7-git [ef37a41]

[paololucente]

Hi Jeff ( @thetooth ),

Thanks for reporting this. In case of crashes, yes, can you please follow instructions here? https://github.com/pmacct/pmacct/blob/1.7.7/QUICKSTART#L2876-#L2896 . It would be of great help. You can paste output of gdb or valgrind here or, if you prefer, by unicast email.

Paolo

[thetooth]

That's fairly impossible, in this distro they have stripped all debugging symbols and any means of installing debug tooling, I did manage to get a core but there is nothing useful.

Here is the only info I can get out of the system:

[ 1309.865917] uacctd[4035]: segfault at 6 ip 00007f1172f14b46 sp 00007ffe7c683de8 error 4 in libc-2.31.so[7f1172dd7000+14b000]
[ 1309.865944] Code: 08 73 27 80 fa 04 73 33 80 fa 01 77 3b 72 05 0f b6 0e 88 0f c3 c5 fa 6f 06 c5 fa 6f 4c 16 f0 c5 fa 7f 07 c5 fa 7f 4c 17 f0 c3 <48> 8b 4c 16 f8 48 8b 36 48 89 4c 17 f8 48 89 37 c3 8b 4c 16 fc 8b

#0  0x00007f1172f14b46 in ?? ()
#1  0x0000561ae684eff9 in ?? ()
#2  0x0000561ae684f243 in ?? ()
#3  0x0000561ae685a720 in ?? ()
#4  0x0000561ae685b795 in ?? ()
#5  0x0000561ae680ac59 in ?? ()
#6  0x00007f1173301267 in ?? ()
#7  0x00007ffe7c6842a0 in ?? ()
#8  0x32501673220b4900 in ?? ()
#9  0x000000000000009c in ?? ()
#10 0x00007f117243373f in ?? ()
#11 0x00007f1170e35024 in ?? ()
#12 0x0000000000000000 in ?? ()

With ASLR off (i think?):

#0  0x00007ffff7759b46 in ?? ()
#1  0x00005555555a8ff9 in ?? ()
#2  0x00005555555a9243 in ?? ()
#3  0x00005555555b4720 in ?? ()
#4  0x00005555555b5795 in ?? ()
#5  0x0000555555564c59 in ?? ()
#6  0x00007ffff7b46267 in ?? ()
#7  0x00007fffffff2100 in ?? ()
#8  0x0e0ff25f4721f600 in ?? ()
#9  0x000000000000009c in ?? ()
#10 0x00007ffff6c7873f in ?? ()
#11 0x00007ffff567a024 in ?? ()
#12 0x0000000000000000 in ?? ()

[paololucente]

Hi Jeff,

Yeah, unfortunately as you hinted that is not of much help. I will try to reproduce the issue in my dev environment. Failing that, we'll have to invent something.

Paolo

[paololucente]

Hi Jeff,

I am sorry to feedback that i have tried to reproduce the issue with the exact same config above (and played a bit around it) but, at least in my dev environment, i am unable to reproduce it. I'd appreciate further input / support from your side in order to troubleshoot this further.

Paolo

[thetooth]

Thanks for giving it a go, I will try to reproduce it on my side with a debug binary tomorrow. I am extremely bad at using C/C++ tooling so it may take some time, perhaps I will be able to statically link the entire libc runtime and up, if it still dies then it should give a good indication as to where it's going wrong. But I do believe it's still related to the flows param as I've had this process up for 7 days now by just omitting it.