generated/help.h:427:29: warning: missing terminating " character

[vt-alt]

In ALT Linux we have package rebuilds and occasionally it reports build errors for toybox. (It builds with make -j8).
I think this is related to generated/help.h generated incorrectly. I manually didn't catch this so can't see what is there.

Error examples:

• https://git.altlinux.org/beehive/logs/Sisyphus/x86_64/archive/2023/1231/error/toybox-0.8.10-alt1

musl-gcc -I/etc/sysconfig/kernel/include -pipe -frecord-gcc-switches -Wall -g -O2 -flto=auto -Wall -Wundef -Werror=implicit-function-declaration -Wno-char-subscripts -Wno-pointer-sign -funsigned-char -I . -Os -ffunction-sections -fdata-sections -fno-asynchronous-unwind-tables -fno-strict-aliasing -c toys/lsb/pidof.c -o generated/obj/pidof.o
In file included from main.c:74:
generated/help.h:427:29: warning: missing terminating " character
  427 | #define HELP_mke2fs_journal "usage: �.
      |                             ^
generated/help.h:428:6: error: 'E' undeclared here (not in a function)
  428 | ДЄs [-E stride=###] [-O option[,option]] [-L label] [-M path] [-o string] [-j] [-J size=###,device=XXX]\n\n-E stride= Set RAID stripe size (in blocks)\n-J         Journal options\n-L         Volume label\n-M         Path to mount point\n-O [opts]  Specify fewer ext2 option flags (for old kernels)\n-j         Create journal (ext3)\n-o         Created by\n           size: Number of blocks (1024-102400)\n           device: Specify an external journal\n           All of these are on by default (as appropriate)\n   none         Clear default options (all but journaling)\n   dir_index    Use htree indexes for large directories\n   filetype     Store file type info in directory entry\n   has_journal  Set by -j\n   journal_dev  Set by -J device=XXX\n   sparse_super Don't allocate huge numbers of redundant superblocks"
      |      ^
generated/help.h:428:7: error: expected ']' before 'stride'

• https://git.altlinux.org/beehive/logs/Sisyphus/x86_64/archive/2024/0108/error/toybox-0.8.10-alt1

musl-gcc -I/etc/sysconfig/kernel/include -pipe -frecord-gcc-switches -Wall -g -O2 -flto=auto -Wall -Wundef -Werror=implicit-function-declaration -Wno-char-subscripts -Wno-pointer-sign -funsigned-char -I . -Os -ffunction-sections -fdata-sections -fno-asynchronous-unwind-tables -fno-strict-aliasing -c toys/lsb/gzip.c -o generated/obj/gzip.o
In file included from main.c:74:
generated/help.h:427:29: warning: missing terminating " character
  427 | #define HELP_mke2fs_journal "usage:
      |                             ^
generated/help.h:428:1: error: stray '\275' in program
  428 | <bd> <83><94>s [-E stride=###] [-O option[,option]] [-L label] [-M path] [-o string] [-j] [-J size=###,device=XXX]\n\n-E stride= Set RAID stripe size (in blocks)\n-J         Journal options\n-L         Volume label\n-M         Path to mount point\n-O [opts]  Specify fewer ext2 option flags (for old kernels)\n-j         Create journal (ext3)\n-o         Created by\n           size: Number of blocks (1024-102400)\n           device: Specify an external journal\n           All of these are on by default (as appropriate)\n   none         Clear default options (all but journaling)\n   dir_index    Use htree indexes for large directories\n   filetype     Store file type info in directory entry\n   has_journal  Set by -j\n   journal_dev  Set by -J device=XXX\n   sparse_super Don't allocate huge numbers of redundant superblocks"
      | ^~~~
generated/help.h:428:3: error: stray '\203' in program

[landley]

Sigh, this may be another manifestation of #458 (they couldn't get me the breaking input files either. I think I know what's going wrong but it really seems like it would be an impossible input unless you patched the source...)

I'm trying to redo the help plumbing so it doesn't need config2help anymore, but had an attack of life and I'm 3 months behind getting the October toybox release out. I'll bump this up in priority now that a second user's seen it. (Fixing what's there without a test case would be a bit of stab in the dark, but I can make an attempt if I can't get the proper fix in soonish.)

[vt-alt]

Well, I can easily reproduce this, since it's just probabilistic, running builds in cycle. Now for the error:

In file included from main.c:74:
generated/help.h:427:29: warning: missing terminating " character
  427 | #define HELP_mke2fs_journal "usage: <F9>
      |                             ^
generated/help.h:428:2: error: stray '\242' in program
  428 | <U+0197><a2>s [-E stride=###] [-O option[,option]] [-L label] [-M path] [-o string] [-j] [-J size=###,device=XXX]\n\n-E stride= Set RAID stripe size (in blocks)\n-J         Journal options\n-L         Volume label\n-M         Path to mount point\n-O [opts]  Specify fewer ext2 option flags (for old kernels)\n-j         Create journal (ext3)\n-o         Created by\n           size: Number of blocks (1024-102400)\n           device: Specify an external journal\n           All of these are on by default (as appropriate)\n   none         Clear default options (all but journaling)\n   dir_index    Use htree indexes for large directories\n   filetype     Store file type info in directory entry\n   has_journal  Set by -j\n   journal_dev  Set by -J device=XXX\n   sparse_super Don't allocate huge numbers of redundant superblocks"
      |         ^~~~
generated/help.h:428:1: error: unknown type name '\U00000197'
  428 | Ɨ<A2>s [-E stride=###] [-O option[,option]] [-L label] [-M path] [-o string] [-j] [-J size=###,device=XXX]\n\n-E stride= Set RAID stripe size (in blocks)\n-J         Journal options\n-L         Volume label\n-M         Path to mount point\n-O [opts]  Specify fewer ext2 option flags (for old kernels)\n-j         Create journal (ext3)\n-o         Created by\n           size: Number of blocks (1024-102400)\n           device: Specify an external journal\n           All of these are on by default (as appropriate)\n   none         Clear default options (all but journaling)\n   dir_index    Use htree indexes for large directories\n   filetype     Store file type info in directory entry\n   has_journal  Set by -j\n   journal_dev  Set by -J device=XXX\n   sparse_super Don't allocate huge numbers of redundant superblocks"
      | ^
generated/help.h:428:7: error: 'E' undeclared here (not in a function)

builder@x86_64:~/RPM/BUILD/toybox-0.8.10$ cat -v -n generated/help.h | grep -w -1 428
   427  #define HELP_mke2fs_journal "usage: M-y^MM-FM-^WM-"s [-E stride=###] [-O option[,option]] [-L label] [-M path] [-o string] [-j] [-J size=###,device=XXX]\n\n-E stride= Set RAID stripe size (in blocks)\n-J         Journal options\n-L         Volume label\n-M         Path to mount point\n-O [opts]  Specify fewer ext2 option flags (for old kernels)\n-j         Create journal (ext3)\n-o         Created by\n           size: Number of blocks (1024-102400)\n           device: Specify an external journal\n           All of these are on by default (as appropriate)\n   none         Clear default options (all but journaling)\n   dir_index    Use htree indexes for large directories\n   filetype     Store file type info in directory entry\n   has_journal  Set by -j\n   journal_dev  Set by -J device=XXX\n   sparse_super Don't allocate huge numbers of redundant superblocks"
   428
   429  #define HELP_mke2fs "usage: mke2fs [-Fnq] [-b ###] [-N|i ###] [-m ###] device\n\nCreate an ext2 filesystem on a block device or filesystem image.\n\n-F         Force to run on a mounted device\n-n         Don't write to device\n-q         Quiet (no output)\n-b size    Block size (1024, 2048, or 4096)\n-N inodes  Allocate this many inodes\n-i bytes   Allocate one inode for every XXX bytes of device\n-m percent Reserve this percent of filesystem space for root user"

Looks like garbage after usage: and before [-E stride, and it's random:

builder@x86_64:~/RPM/BUILD/toybox-0.8.10$ generated/unstripped/config2help Config.in .config | cat -n | grep -w 427 | xxd | head -5
00000000: 2020 2034 3237 0923 6465 6669 6e65 2048     427.#define H
00000010: 454c 505f 6d6b 6532 6673 5f6a 6f75 726e  ELP_mke2fs_journ
00000020: 616c 2022 7573 6167 653a 20da b88c a254  al "usage: ....T
00000030: 7320 5b2d 4520 7374 7269 6465 3d23 2323  s [-E stride=###
00000040: 5d20 5b2d 4f20 6f70 7469 6f6e 5b2c 6f70  ] [-O option[,op
builder@x86_64:~/RPM/BUILD/toybox-0.8.10$ generated/unstripped/config2help Config.in .config | cat -n | grep -w 427 | xxd | head -5
00000000: 2020 2034 3237 0923 6465 6669 6e65 2048     427.#define H
00000010: 454c 505f 6d6b 6532 6673 5f6a 6f75 726e  ELP_mke2fs_journ
00000020: 616c 2022 7573 6167 653a 20f0 9fe3 ca39  al "usage: ....9
00000030: 7320 5b2d 4520 7374 7269 6465 3d23 2323  s [-E stride=###
00000040: 5d20 5b2d 4f20 6f70 7469 6f6e 5b2c 6f70  ] [-O option[,op
builder@x86_64:~/RPM/BUILD/toybox-0.8.10$ generated/unstripped/config2help Config.in .config | cat -n | grep -w 427 | xxd | head -5
00000000: 2020 2034 3237 0923 6465 6669 6e65 2048     427.#define H
00000010: 454c 505f 6d6b 6532 6673 5f6a 6f75 726e  ELP_mke2fs_journ
00000020: 616c 2022 7573 6167 653a 2003 1b34 7a97  al "usage: ..4z.
00000030: 7320 5b2d 4520 7374 7269 6465 3d23 2323  s [-E stride=###
00000040: 5d20 5b2d 4f20 6f70 7469 6f6e 5b2c 6f70  ] [-O option[,op
builder@x86_64:~/RPM/BUILD/toybox-0.8.10$ generated/unstripped/config2help Config.in .config | cat -n | grep -w 427 | xxd | head -5
00000000: 2020 2034 3237 0923 6465 6669 6e65 2048     427.#define H
00000010: 454c 505f 6d6b 6532 6673 5f6a 6f75 726e  ELP_mke2fs_journ
00000020: 616c 2022 7573 6167 653a 2095 3aa1 57b0  al "usage: .:.W.
00000030: 7320 5b2d 4520 7374 7269 6465 3d23 2323  s [-E stride=###
00000040: 5d20 5b2d 4f20 6f70 7469 6f6e 5b2c 6f70  ] [-O option[,op
builder@x86_64:~/RPM/BUILD/toybox-0.8.10$ generated/unstripped/config2help Config.in .config | cat -n | grep -w 427 | xxd | head -5
00000000: 2020 2034 3237 0923 6465 6669 6e65 2048     427.#define H
00000010: 454c 505f 6d6b 6532 6673 5f6a 6f75 726e  ELP_mke2fs_journ
00000020: 616c 2022 7573 6167 653a 20bb 4ed8 3a0e  al "usage: .N.:.
00000030: 7320 5b2d 4520 7374 7269 6465 3d23 2323  s [-E stride=###
00000040: 5d20 5b2d 4f20 6f70 7469 6f6e 5b2c 6f70  ] [-O option[,op

Run under valgrind:

builder@x86_64:~/RPM/BUILD/toybox-0.8.10$ gcc  -g -O2 scripts/config2help.c -o generated/unstripped/config2help
builder@x86_64:~/RPM/BUILD/toybox-0.8.10$ valgrind generated/unstripped/config2help Config.in .config > a
==189753== Memcheck, a memory error detector
==189753== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==189753== Using Valgrind-3.17.0 and LibVEX; rerun with -h for copyright info
==189753== Command: generated/unstripped/config2help Config.in .config
==189753==
--189753-- WARNING: unhandled amd64-linux syscall: 334
--189753-- You may be able to write your own handler.
--189753-- Read the file README_MISSING_SYSCALL_OR_IOCTL.
--189753-- Nevertheless we consider this a bug.  Please report
--189753-- it at http://valgrind.org/support/bug_reports.html.
==189753== Invalid read of size 1
==189753==    at 0x4847229: strnlen (vg_replace_strmem.c:436)
==189753==    by 0x48F74BB: strndup (in /lib64/libc.so.6)
==189753==    by 0x1095DF: main (config2help.c:365)
==189753==  Address 0x4adba8b is 11 bytes inside a block of size 120 free'd
==189753==    at 0x4843F4C: free (vg_replace_malloc.c:755)
==189753==    by 0x1098EB: main (config2help.c:460)
==189753==  Block was alloc'd at
==189753==    at 0x484179B: malloc (vg_replace_malloc.c:380)
==189753==    by 0x48CD35D: getdelim (in /lib64/libc.so.6)
==189753==    by 0x10A78C: parse (config2help.c:241)
==189753==    by 0x10A979: parse (config2help.c:255)
==189753==    by 0x10920F: main (config2help.c:314)
==189753==
==189753== Invalid read of size 1
==189753==    at 0x484BF80: __memcpy_chk (vg_replace_strmem.c:1617)
==189753==    by 0x1095DF: main (config2help.c:365)
==189753==  Address 0x4adba90 is 16 bytes inside a block of size 120 free'd
==189753==    at 0x4843F4C: free (vg_replace_malloc.c:755)
==189753==    by 0x1098EB: main (config2help.c:460)
==189753==  Block was alloc'd at
==189753==    at 0x484179B: malloc (vg_replace_malloc.c:380)
==189753==    by 0x48CD35D: getdelim (in /lib64/libc.so.6)
==189753==    by 0x10A78C: parse (config2help.c:241)
==189753==    by 0x10A979: parse (config2help.c:255)
==189753==    by 0x10920F: main (config2help.c:314)
==189753==
==189753== Invalid read of size 1
==189753==    at 0x484BF8D: __memcpy_chk (vg_replace_strmem.c:1617)
==189753==    by 0x1095DF: main (config2help.c:365)
==189753==  Address 0x4adba8e is 14 bytes inside a block of size 120 free'd
==189753==    at 0x4843F4C: free (vg_replace_malloc.c:755)
==189753==    by 0x1098EB: main (config2help.c:460)
==189753==  Block was alloc'd at
==189753==    at 0x484179B: malloc (vg_replace_malloc.c:380)
==189753==    by 0x48CD35D: getdelim (in /lib64/libc.so.6)
==189753==    by 0x10A78C: parse (config2help.c:241)
==189753==    by 0x10A979: parse (config2help.c:255)
==189753==    by 0x10920F: main (config2help.c:314)
==189753==

(First valgrind run was without -g and not very informative about source line numbers, This output above is regenerated after recompile after I already found problematic line with ASan.) Anyway, it looked like a memory corruption and I tried ASan:

builder@x86_64:~/RPM/BUILD/toybox-0.8.10$ gcc -fsanitize=address  -g -O2 scripts/config2help.c -o generated/unstripped/config2help
builder@x86_64:~/RPM/BUILD/toybox-0.8.10$ generated/unstripped/config2help Config.in .config > a
=================================================================
==188909==ERROR: AddressSanitizer: heap-use-after-free on address 0x60c000071a4b at pc 0x7feba45ee44c bp 0x7ffe01e46640 sp 0x7ffe01e45de8
READ of size 6 at 0x60c000071a4b thread T0
    #0 0x7feba45ee44b  (/usr/lib64/libasan.so.8+0x6e44b) (BuildId: a74d5a7d37a69f840ee795974830b5763483896c)
    #1 0x561868e8fdbc in main scripts/config2help.c:365
    #2 0x7feba43c0c8b  (/lib64/libc.so.6+0x27c8b) (BuildId: 8d90d7392295faafc24f1c706c43e4dede4a998b)
    #3 0x7feba43c0d44 in __libc_start_main (/lib64/libc.so.6+0x27d44) (BuildId: 8d90d7392295faafc24f1c706c43e4dede4a998b)
    #4 0x561868e918b0 in _start (/usr/src/RPM/BUILD/toybox-0.8.10/generated/unstripped/config2help+0x48b0) (BuildId: cadd746ba218d161aae375c0fd7843f6e499d997)

0x60c000071a4b is located 11 bytes inside of 120-byte region [0x60c000071a40,0x60c000071ab8)
freed by thread T0 here:
    #0 0x7feba465d090  (/usr/lib64/libasan.so.8+0xdd090) (BuildId: a74d5a7d37a69f840ee795974830b5763483896c)
    #1 0x561868e90c7c in main scripts/config2help.c:460

previously allocated by thread T0 here:
    #0 0x7feba465e54f in __interceptor_malloc (/usr/lib64/libasan.so.8+0xde54f) (BuildId: a74d5a7d37a69f840ee795974830b5763483896c)
    #1 0x7feba441035d in __getdelim (/lib64/libc.so.6+0x7735d) (BuildId: 8d90d7392295faafc24f1c706c43e4dede4a998b)

After it remove that free at config2help.c:460:

        //free(tusage);

Now it does not produce the difference on line 427:

builder@x86_64:~/RPM/BUILD/toybox-0.8.10$ gcc  -g -O2 scripts/config2help.c -o generated/unstripped/config2help
builder@x86_64:~/RPM/BUILD/toybox-0.8.10$ generated/unstripped/config2help Config.in .config | cat -n | grep -w 427 | xxd | head -5
00000000: 2020 2034 3237 0923 6465 6669 6e65 2048     427.#define H
00000010: 454c 505f 6d6b 6532 6673 2022 7573 6167  ELP_mke2fs "usag
00000020: 653a 206d 6b65 3266 7320 5b2d 4520 7374  e: mke2fs [-E st
00000030: 7269 6465 3d23 2323 5d20 5b2d 4f20 6f70  ride=###] [-O op
00000040: 7469 6f6e 5b2c 6f70 7469 6f6e 5d5d 205b  tion[,option]] [
builder@x86_64:~/RPM/BUILD/toybox-0.8.10$ generated/unstripped/config2help Config.in .config | cat -n | grep -w 427 | xxd | head -5
00000000: 2020 2034 3237 0923 6465 6669 6e65 2048     427.#define H
00000010: 454c 505f 6d6b 6532 6673 2022 7573 6167  ELP_mke2fs "usag
00000020: 653a 206d 6b65 3266 7320 5b2d 4520 7374  e: mke2fs [-E st
00000030: 7269 6465 3d23 2323 5d20 5b2d 4f20 6f70  ride=###] [-O op
00000040: 7469 6f6e 5b2c 6f70 7469 6f6e 5d5d 205b  tion[,option]] [

Looks not corrupted anymore.

[vt-alt]

Just to confirm this is not fixed in 0.8.11.
Build log https://git.altlinux.org/beehive/logs/Sisyphus/x86_64/archive/2024/0428/error/toybox-0.8.11-alt1

[Dev-Voyager23]

@vt-alt Does your fix of removing free work?
Can someone help?
Thanks!