Crash in config2help during build #458
Comments
|
The most recent commit to that file was 4 and 1/2 years ago, and we've had 10 releases since then. The failure is that your libc's strndup() returned NULL. I have two questions about this, both of which are "how?" The first "how?" is that on an allyesconfig build "toybox help | wc -c" says 154k total help text. For comparison, when I run my devuan x86-64's "sleep" command /proc/$PID/smaps_rollup says 1592 kB referenced. If running this binary exhausts memory, you couldn't have compiled it. (Did you dynamically set some crazy-low ulimit that's an order of magnitude lower than what the compiler is allowed to consume? Some sort of container environment bug, maybe?) The second "how?" is that running out of physical memory has only ever returned NULL from libc allocation functions on NOMMU Linux. With an MMU, Linux libc suballocates from the heap and extends it by allocating more virtual address space, then the physical memory is dynamically allocated later by the page fault handler during a write access to the COW mapping of the zero page (or the modern eqvivalent that's more SMP friendly and avoids the reference count wrapping issue big iron hit, but conceptually the same). The OOM killer picks a process to SIGKILL at this time if it can't satisfy the allocation, usually the one that asked for more memory but there's a bucket of terrible heuristics that gets stirred from time to time to make it bigger. Recognition of "we actually don't have enough memory" happpens long after the allocation of virtual space has returned, unless you twiddle some "make the system extremely unstable" knobs ("disable overcommit") under /proc which basically ask programs to speculatively fail when they otherwise would have succeeded. (I.E. it never makes MORE stuff work, it only ever makes LESS stuff work. It changes the Linux programming API.) Toybox proper always checks those return values (xmalloc() and friends) because it supports running on NOMMU systems that can arbitrarily return NULL due to memory fragmentation and similar, and because commands exiting with failure happens all the time, ala "echo > /dev/full" and so on. But we don't support building on nommu systems, which is why I didn't bother to check for a NULL return from allocations in build tools processing known inputs: if it does fail, all I can do is exit() because the system is completely horked. I could change the line to exit with a more graceful error message, but your build would still break. It's theoretically possible that the failing call to strndup() has some other reason for returning NULL? Maybe it got dodgy arguments (a negative length maybe?) and I'm curious what its inputs were when it returned NULL. It's also possible that the heap was corrupted at some point before then and the malloc failed chasing pointers, except A) glibc dies with an assert() when that happens (rather than allocation functions randomly returning NULL), B) you ran it under valgrind which should catch any out of bounds write even to elsewhere in the heap if it's not cleanly within an allocation boundary. When scripts/make.sh calls mkflags it snapshots its input with tee, producing generated/flags.raw for later analysis, because flag processing is tricksy and occasionally non-obvious and has gone wrong a few times over the years (albeit when adding new commands, not when building a release version on someone else's system). But I never bothered doing that with config2help.c because in the 9 years since I replaced the python build script with a C implementation it's literally never come up. Still, if you wanted to add a call to tee into the pipeline on scripts/make.sh line 266 and send me the input going to config2help, I can try to reproduce your issue here? |
|
Ah, after going and reading the gentoo bug and then coming back and rereading this one, it's not strndup() returning NULL, it's strndup() being CALLED on a null. My bad, I misread, and that's different. Hmmm... The strndup(usage) is gated by if (catch->help && (that = keyword("usage:", catch->help->data))) and although keyword("usage:") can return NULL we don't enter the block when it does. The pointer "throw" is initialized to 0 and re-initialized that way each outer loop, and inside the block if (!throw) usage = that; precedes the strndup(). The only way we get a nonsense usage value is by looping back around with throw already set, which should never happen but apparently is? (This is vestigial code, it's for collating the help text from sub-options like busybox has a zillion of, which toybox started eliminating years ago and has almost none of left, ala commits 16c0ba5 and 1aaef2d and so on. This part of the code glued together multiple usage: lines to produce one usage: line listing all the command line options of both. 2/3 of what this does is obsolete now and I plan to remove it at some point, that's part of the reason it hasn't gotten a lot of attention in forever...) Hmmm... I think for what you're seeing to trigger, you'd have to have a command with no usage: line with a sub-option that DID have a usage: line, and then it would try to glue the additional usage line to nothing. Which should never happen because it's broken input (the help text in the config entries is wrong), but the code should produce a way more graceful error message if so? Your build is still going to break but it should do it with an "impossible input" error message saying which command it was processing at the time so you can track down what help text change made it go weird. (It's also theoretically possible that your .config could trigger a previously untested combo, providing impossible inputs that way, but that's still a bug in the input if so. The config dependencies shouldn't allow that...) |
|
Interestingly, 0.8.9 now breaks for me too when I try it again, which is consistent with what you're saying ofc. I guess newer glibc triggers it. |
|
Could be new libc, new compiler, a change in the runtime environment... I still want to fix it. It's just awkward without having locally reproduced it. Rather than try to fix the old help text collating tool nobody's looked at in years, I've shuffled the TODO list around a bit and am looking at removing the remaining command subfeatures that need collating, or at least eliminating the separate help text so this tool doesn't need to stitch them together anymore. That said, there are several of them: MKNOD_Z, ID_Z, MKDIR_Z, and MKFIFO_Z add the -Z options for selinux, and SORT_FLOAT adds the -g option. Possibly there's some OTHER kind of signaling I can do here? (There's also some other stuff like PASSWD_SAD and WGET_LIBTLS that don't really NEED to add help text, it adds an automatic capability that doesn't really need to be separately documented. The first is the terrible password checking heuristic and the second is https as well as http support.) Gimme a couple days to try to clear a couple other pending todo items and think about this one a bit. Poke me monday if I haven't fixed this by then? |
|
ping as requested (although I'm in no hurry, honest) |
I originally reported this downstream in Gentoo at https://bugs.gentoo.org/914589.
If I jam HOSTCC="gcc -ggdb3", I get this when running
generated/unstripped/config2help Config.in .config:Possibly slightly more useful from valgrind:
From asan/ubsan (probably the most helpful, as you get the naughtiness immediately rather than when it tries to read later on):
The text was updated successfully, but these errors were encountered: