-
Notifications
You must be signed in to change notification settings - Fork 212
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Segfault in _check_interface_entry_for_updates #107
Comments
With which Net-SNMP version has this been observed and on which platform has this been observed (Linux, FreeBSD, ...)? |
Sorry, it's a Gentoo Linux system running Net-SNMP 5.8. |
I took a close look at the IF-MIB implementation and I don't see how the rowreq_ctx pointer could be NULL other than as the result of memory corruption. As one can see in _add_new_interface(), CONTAINER_INSERT() is only called if rowreq_ctx != NULL. How about trying to reproduce this issue with snmpd running under Valgrind? |
The pointer isn't NULL, it's just the |
I think that the missing Anyway, it would be appreciated if the same workload could be applied to the latest version of the v5.8 branch that is run under Valgrind. BTW, two memory corruption fixes have been checked in earlier today on that branch. I'm not sure however whether these fixes are related to what has been reported above. |
We've encountered the same issue with the same crash dump at work. I probably wouldn't even have noticed if our IT department hadn't pinged me. We keep restarting the daemon, but it then crashes again at the same location some hours later. This is on an Ubuntu 20.04 machine running Net-SNMP 5.8 Wild guess: The machine in question is a CI machine where we keep spooling up and discarding Docker containers throughout the days. Possibly a race condition when Docker adds and / or removes network interfaces? |
Net-SNMP v5.8 is no longer supported. Is this reproducible with Net-SNMP v5.9? |
Docker seems to be a likely culprit - our affected machine is running it as well. 5.9 is still affected by this. |
I've installed Net-SNMP v5.9 on our machine. No crashes so far, I'll let you know if that changes. |
It crashed again. |
Any update on fix for this ? |
This is happening to me as well in 5.9, but only on vms running docker containers which are spun up/down regularly, similar to @sgarcialaguna-mms comment. |
@po5857, @singh4jitendra and @sgarcialaguna-mms , what made you decide that you ran into the same crash as the original reporter of this bug? Please provide more information about the crashes that you observed. Which Net-SNMP version are you using? Has Net-SNMP been compiled from source or has a binary package been installed that was provided by a Linux distributor? In the latter case, which Linux distribution are you using? Did the crash happen while snmpd was running or during shutdown? Is there anything unusual about your setup, e.g. a large number of disks or a large number of network interfaces? Is snmpd running inside Docker or not? |
@bvanassche, because it's crashing at the same exact place as the original reporter:
and more info from gdb:
In my case I'm on Fedora 34, which has net-snmp-5.9-9 rpm. Snmpd is not running inside the docker containers - only on the host. It crashes regularly - today's crashes for instance: Jul 24 02:40 I've had to tweak the systemd service file to add "Restart=on-failure" to avoid constant manual restarts. I just compiled a new rpm with optflags -O0 so I can avoid optimized out vars. Aside from that, let me know if there's something else I can do to assist in tracking this down. |
Disabling optimizations led to a different backtrace unfortunately.
It is now crashing in netsnmp_access_interface_entry_free here:
old_stats is referencing junk:
I can only guess that the change in optimization level moved the (apparent) race condition around. |
Or maybe memory corruption is involved. Does the list of network interfaces on your setup change after snmpd has been started ( |
Yes, docker creates a "veth" interface for each container. Containers are spinning up/down all the time, thus the list is constantly changing. One hackish workaround for this would be to allow users to ignore specific interfaces via config file. For instance, if we could ignore "veth*" (which we do not care about monitoring anyway) I suspect the crashing would stop.
|
I guess "include_ifmib_iface_prefix eth" would be the hackish workaround I'm looking for. |
Well, "include_ifmib_iface_prefix eth" did not help...it still crashes. |
I was asking for this information because I wanted to try to reproduce the issue. So far without success unfortunately. Can you run snmpd under Valgrind and reproduce the issue? Running snmpd under Valgrind is easier from the command line than from a service file and can be done e.g. as follows: |
So far 3 of these:
|
So far I haven't been able to reproduce this issue. But I took considerable time to verify the source code of the ifTable implementation. I fixed multiple bugs but I'm not sure that the reported segmentation fault has been fixed by the changes I made. Would it be possible to run the latest version of the v5.9.1 branch on your setup under Valgrind? The latest version of the v5.9.1 branch can be downloaded from https://github.com/net-snmp/net-snmp/archive/refs/heads/V5-9-patches.zip. |
I've been running this code for a few hours. So far got the below output:
|
I enabled some various debug options, and got this around your newly added assert:
And here's another
|
Thank you for having provided the entire snmpd output and not only the Valgrind complaints. The assertion failure has been fixed yesterday on the v5.9 and master branches. I had not yet reported this because I'm not sure whether the changes I made yesterday fix the Valgrind complaint. Anyway, if you have the time to retest the latest version of the v5.9 branch under Valgrind, that would be welcome. |
Thanks, I'm running that version now. Will advise if it complains. Completely unrelated: I've been dragging along a patch to add a feature we use for a few years, and I'd like to get it merged upstream. What's the best way to submit it for review? Mailing list? |
Please submit a pull request to https://github.com/net-snmp/net-snmp/. If that would be inconvenient, please create a patch with |
Thanks, will do. In other news, valgrind is still complaining, including some new traces I haven't seen before. It's a long list, so attached as a file. |
I'm wondering whether the reported behavior could have been caused by a bug in the Net-SNMP binary array implementation (libsnmp/container_binary_array.c). I have fixed several small bugs in that code and also added two consistency checks. Please retest the latest version of the v5.9 branch under Valgrind. If Valgrind would report a memory error, please share the entire snmpd log up to and including the first Valgrind complaint. |
I ran this new version for the past 3 hours or so. Attached is all the valgrind output. |
Disable the consistency checks added by the previous commit since the test these were intended for has completed. See also #107 .
So far I've been running this on 2 systems for the past 8 hours and no valgrind complaints. Will leave it running over the weekend and let you know, thanks! |
Two things:
The statfs error is for similar reasons as the interfaces - docker is adding/removing mounts regularly. |
Been running for a week now and aside from the occasional "error on subcontainer 'interface container' insert (-1)" error, it all looks good. Thanks! |
Thanks for the help with testing and also for confirming that this issue has been solved. Let's close this bug report. |
Hi all,
for a while I've had problems with snmpd crashing irregularly with a segfault. After finally catching a core dump and analyzing it I've found the following:
snmpd crashes at mibgroup/if-mib/ifTable/ifTable_data_access.c:317. The
rowreq_ctx->data
object looks like this:So there is no
ifLastChange
member. The system has an assortment of VLAN interfaces but nothing my other machines don't have.Is this something anyone has seen before?
Regards,
Dennis
The text was updated successfully, but these errors were encountered: