clang UBSAN issue in 'check_gzip_header()'

[gvanem]

I did this

This is not an issue with libcurl per-se. But it seems clang + UBSAN is sensitive to a syntax
like enum { a, b, c } func (args) ... See below.

But trying to build libcurl with clang-cl and UBSAN ("Undefined Behaviour AddressSanitizer"), caused this internal error in
clang while compiling the check_gzip_header()function. Clang report:

clang-cl @clang-cl.args -Fo./objects/x64/content_encoding.obj content_encoding.c
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, 
preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: f:\\ProgramFiler\\LLVM-17.0\\win64\\bin\\clang-cl.exe @clang-cl.args -Fo./objects/x64/content_encoding.obj content_encoding.c
1.      <eof> parser at end of file
2.      Per-file LLVM IR generation
3.      content_encoding.c:372:3: Generating code for declaration 'check_gzip_header'
Exception Code: 0xC0000005
 #0 0x00007ff6c0c7dfc7 (F:\ProgramFiler\LLVM-17.0\win64\bin\clang-cl.exe+0x155dfc7)
 #1 0x00007ff6c223f961 (F:\ProgramFiler\LLVM-17.0\win64\bin\clang-cl.exe+0x2b1f961)
...
#33 0x00007ff6c3cbf130 (F:\ProgramFiler\LLVM-17.0\win64\bin\clang-cl.exe+0x459f130)
#34 0x00007ff9bca77344 (C:\Windows\System32\KERNEL32.DLL+0x17344)
#35 0x00007ff9bcc226b1 (C:\Windows\SYSTEM32\ntdll.dll+0x526b1)
clang-cl: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 17.0.1
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: f:\ProgramFiler\LLVM-17.0\win64\bin
clang-cl: note: diagnostic msg:
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-cl: note: diagnostic msg: c:\temp\content_encoding-6786a6.c
clang-cl: note: diagnostic msg: c:\temp\content_encoding-6786a6.sh
clang-cl: note: diagnostic msg:

********************

Attached is c:\temp\content_encoding-6786a6.sh: content_encoding-6786a6.sh.txt

Trying to understand why, I modified the file into:

--- a/lib/content_encoding.c 2023-11-13 11:47:41
+++ b/lib/content_encoding.c 2024-01-02 10:33:55
@@ -365,11 +365,13 @@

 #ifdef OLD_ZLIB_SUPPORT
 /* Skip over the gzip header */
-static enum {
+typedef enum {
   GZIP_OK,
   GZIP_BAD,
   GZIP_UNDERFLOW
-} check_gzip_header(unsigned char const *data, ssize_t len, ssize_t *headerlen)
+} gzip_status;
+
+static gzip_status check_gzip_header(unsigned char const *data, ssize_t len, ssize_t *headerlen)
 {
   int method, flags;
   const ssize_t totallen = len;

all is well. The same is commenting out #define OLD_ZLIB_SUPPORT 1.

Not sure which of flags triggered this internal-error. But these were the ASAN/UBSAN flags I used:

-fsanitize=address
-fsanitize-recover=address
-fsanitize=undefined

I expected the following

A successful compilation of content_encoding.c.

curl/libcurl version

Latest from git master

operating system

Win-10 22H2.

[gvanem]

-fsanitize=address
-fsanitize-recover=address
-fsanitize=undefined

Quoting from https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html#ubsan-checks:

-fsanitize=function: Indirect call of a function through a function pointer of the wrong type.

Hence, adding -fno-sanitize=function to my CFLAGS, there is no internal-error either.

[thesamesam]

@gvanem Did you end up filing a Clang bug for this?

[gvanem]

@thesamesam There are approx 5000 issues there. So I don't feel like it.

[thesamesam]

That's fine, although it doesn't really work that way wrt total count of open bugs. Some of them are very old and imported from Bugzilla, some are feature requests, etc. Different things are handled by different people.

[gvanem]

Ok then. I'll try to cook up a minimal example that demonstrates this UBSAN bug.

[gvanem]

I made an issue here.