http2.c: ASSERT: len >= stream->upload_blocked_len

[bagder]

I did this

OSS-Fuzz triggered this assert when fuzzing HTTP. It happens on http.c:2080.

Here's a stack trace

+----------------------------------------Release Build Stacktrace----------------------------------------+
--
  | Command: /mnt/scratch0/clusterfuzz/resources/platform/linux/unshare -c -n /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http -rss_limit_mb=2560 -timeout=60 -runs=100 /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases/crash-8ae050ae9e02b07a06ae273720218a22cc08117a
  | Time ran: 0.10861730575561523
  |  
  | INFO: Running with entropic power schedule (0xFF, 100).
  | INFO: Seed: 2532200055
  | INFO: Loaded 1 modules   (104786 inline 8-bit counters): 104786 [0x8ea4f00, 0x8ebe852),
  | INFO: Loaded 1 PC tables (104786 PCs): 104786 [0x8ebe854,0x8f8b2e4),
  | /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http: Running 1 inputs 100 time(s) each.
  | Running: /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases/crash-8ae050ae9e02b07a06ae273720218a22cc08117a
  | curl_fuzzer_http: http2.c:2080: ssize_t cf_h2_send(struct Curl_cfilter *, struct Curl_easy *, const void *, size_t, CURLcode *): Assertion `len >= stream->upload_blocked_len' failed.
  | AddressSanitizer:DEADLYSIGNAL
  | =================================================================
  | ==613==ERROR: AddressSanitizer: ABRT on unknown address 0x00000265 (pc 0xf7f05509 bp 0xffd0fa1c sp 0xffd0fa00 T0)
  | SCARINESS: 10 (signal)
  | #0 0xf7f05509 in linux-gate.so.1
  | #1 0xf7be7275 in raise
  | #2 0xf7bcf3f6 in abort
  | #3 0xf7bcf2ba in libc.so.6
  | #4 0xf7bdefde in __assert_fail
  | #5 0x83660bc in cf_h2_send curl/lib/http2.c:0
  | #6 0x82ed819 in Curl_conn_send curl/lib/cfilters.c:199:12
  | #7 0x8280195 in Curl_write curl/lib/sendf.c:175:19
  | #8 0x82997f7 in readwrite_upload curl/lib/transfer.c:978:14
  | #9 0x82997f7 in Curl_readwrite curl/lib/transfer.c:1127:14
  | #10 0x826bbaa in multi_runsingle curl/lib/multi.c:2459:16
  | #11 0x8268086 in curl_multi_perform curl/lib/multi.c:2756:16
  | #12 0x822a4ee in fuzz_handle_transfer(fuzz_data*) curl_fuzzer/curl_fuzzer.cc:341:3
  | #13 0x8229589 in LLVMFuzzerTestOneInput curl_fuzzer/curl_fuzzer.cc:97:3
  | #14 0x80ea00e in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned int) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:611:15
  | #15 0x80d4f6e in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned int) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:324:6
  | #16 0x80dab70 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned int)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:860:9
  | #17 0x8104757 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
  | #18 0xf7bd0ed4 in __libc_start_main
  | #19 0x80cc0b5 in _start
  |  
  | AddressSanitizer can not provide additional info.
  | SUMMARY: AddressSanitizer: ABRT (linux-gate.so.1+0x509) (BuildId: f2f1c233874dcb1eda613150567b5b9a3270795b)
  | ==613==ABORTING
  |  
  |  
  | +----------------------------------------Release Build Unsymbolized Stacktrace (diff)----------------------------------------+
  |  
  | ==613==ERROR: AddressSanitizer: ABRT on unknown address 0x00000265 (pc 0xf7f05509 bp 0xffd0fa1c sp 0xffd0fa00 T0)
  | SCARINESS: 10 (signal)
  | #0 0xf7f05509  (linux-gate.so.1+0x509) (BuildId: f2f1c233874dcb1eda613150567b5b9a3270795b)
  | #1 0xf7be7275  (/lib32/libc.so.6+0x31275) (BuildId: 8c11d7b4ac6d685f0bba1cf2506a80f64d314582)
  | #2 0xf7bcf3f6  (/lib32/libc.so.6+0x193f6) (BuildId: 8c11d7b4ac6d685f0bba1cf2506a80f64d314582)
  | #3 0xf7bcf2ba  (/lib32/libc.so.6+0x192ba) (BuildId: 8c11d7b4ac6d685f0bba1cf2506a80f64d314582)
  | #4 0xf7bdefde  (/lib32/libc.so.6+0x28fde) (BuildId: 8c11d7b4ac6d685f0bba1cf2506a80f64d314582)
  | #5 0x83660bc  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x83660bc)
  | #6 0x82ed819  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x82ed819)
  | #7 0x8280195  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x8280195)
  | #8 0x82997f7  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x82997f7)
  | #9 0x826bbaa  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x826bbaa)
  | #10 0x8268086  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x8268086)
  | #11 0x822a4ee  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x822a4ee)
  | #12 0x8229589  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x8229589)
  | #13 0x80ea00e  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x80ea00e)
  | #14 0x80d4f6e  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x80d4f6e)
  | #15 0x80dab70  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x80dab70)
  | #16 0x8104757  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x8104757)
  | #17 0xf7bd0ed4  (/lib32/libc.so.6+0x1aed4) (BuildId: 8c11d7b4ac6d685f0bba1cf2506a80f64d314582)
  | #18 0x80cc0b5  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x80cc0b5)

+----------------------------------------Release Build Stacktrace----------------------------------------+
	Command: /mnt/scratch0/clusterfuzz/resources/platform/linux/unshare -c -n /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http -rss_limit_mb=2560 -timeout=60 -runs=100 /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases/crash-8ae050ae9e02b07a06ae273720218a22cc08117a
	Time ran: 0.10861730575561523
	
	INFO: Running with entropic power schedule (0xFF, 100).
	INFO: Seed: 2532200055
	INFO: Loaded 1 modules   (104786 inline 8-bit counters): 104786 [0x8ea4f00, 0x8ebe852),
	INFO: Loaded 1 PC tables (104786 PCs): 104786 [0x8ebe854,0x8f8b2e4),
	/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http: Running 1 inputs 100 time(s) each.
	Running: /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases/crash-8ae050ae9e02b07a06ae273720218a22cc08117a
	curl_fuzzer_http: http2.c:2080: ssize_t cf_h2_send(struct Curl_cfilter *, struct Curl_easy *, const void *, size_t, CURLcode *): Assertion `len >= stream->upload_blocked_len' failed.
	AddressSanitizer:DEADLYSIGNAL
	=================================================================
	==613==ERROR: AddressSanitizer: ABRT on unknown address 0x00000265 (pc 0xf7f05509 bp 0xffd0fa1c sp 0xffd0fa00 T0)
	SCARINESS: 10 (signal)
	    #0 0xf7f05509 in linux-gate.so.1
	    #1 0xf7be7275 in raise
	    #2 0xf7bcf3f6 in abort
	    #3 0xf7bcf2ba in libc.so.6
	    #4 0xf7bdefde in __assert_fail
	    #5 0x83660bc in cf_h2_send [curl/lib/http2.c:0](https://github.com/curl/curl/blob/bc642cb333052347a1ddf8247ff4f8e2132bf7c6/lib/http2.c#L0)
	    #6 0x82ed819 in Curl_conn_send [curl/lib/cfilters.c:199](https://github.com/curl/curl/blob/bc642cb333052347a1ddf8247ff4f8e2132bf7c6/lib/cfilters.c#L199):12
	    #7 0x8280195 in Curl_write [curl/lib/sendf.c:175](https://github.com/curl/curl/blob/bc642cb333052347a1ddf8247ff4f8e2132bf7c6/lib/sendf.c#L175):19
	    #8 0x82997f7 in readwrite_upload [curl/lib/transfer.c:978](https://github.com/curl/curl/blob/bc642cb333052347a1ddf8247ff4f8e2132bf7c6/lib/transfer.c#L978):14
	    #9 0x82997f7 in Curl_readwrite [curl/lib/transfer.c:1127](https://github.com/curl/curl/blob/bc642cb333052347a1ddf8247ff4f8e2132bf7c6/lib/transfer.c#L1127):14
	    #10 0x826bbaa in multi_runsingle [curl/lib/multi.c:2459](https://github.com/curl/curl/blob/bc642cb333052347a1ddf8247ff4f8e2132bf7c6/lib/multi.c#L2459):16
	    #11 0x8268086 in curl_multi_perform [curl/lib/multi.c:2756](https://github.com/curl/curl/blob/bc642cb333052347a1ddf8247ff4f8e2132bf7c6/lib/multi.c#L2756):16
	    #12 0x822a4ee in fuzz_handle_transfer(fuzz_data*) [curl_fuzzer/curl_fuzzer.cc:341](https://github.com/curl/curl-fuzzer/blob/5f97dbd82035f7a28aeb2005de0cfcaedd69aae2/curl_fuzzer.cc#L341):3
	    #13 0x8229589 in LLVMFuzzerTestOneInput [curl_fuzzer/curl_fuzzer.cc:97](https://github.com/curl/curl-fuzzer/blob/5f97dbd82035f7a28aeb2005de0cfcaedd69aae2/curl_fuzzer.cc#L97):3
	    #14 0x80ea00e in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned int) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:611:15
	    #15 0x80d4f6e in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned int) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:324:6
	    #16 0x80dab70 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned int)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:860:9
	    #17 0x8104757 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
	    #18 0xf7bd0ed4 in __libc_start_main
	    #19 0x80cc0b5 in _start
	
	AddressSanitizer can not provide additional info.
	SUMMARY: AddressSanitizer: ABRT (linux-gate.so.1+0x509) (BuildId: f2f1c233874dcb1eda613150567b5b9a3270795b)
	==613==ABORTING
	
	
	+----------------------------------------Release Build Unsymbolized Stacktrace (diff)----------------------------------------+
	
	==613==ERROR: AddressSanitizer: ABRT on unknown address 0x00000265 (pc 0xf7f05509 bp 0xffd0fa1c sp 0xffd0fa00 T0)
	SCARINESS: 10 (signal)
	    #0 0xf7f05509  (linux-gate.so.1+0x509) (BuildId: f2f1c233874dcb1eda613150567b5b9a3270795b)
	    #1 0xf7be7275  (/lib32/libc.so.6+0x31275) (BuildId: 8c11d7b4ac6d685f0bba1cf2506a80f64d314582)
	    #2 0xf7bcf3f6  (/lib32/libc.so.6+0x193f6) (BuildId: 8c11d7b4ac6d685f0bba1cf2506a80f64d314582)
	    #3 0xf7bcf2ba  (/lib32/libc.so.6+0x192ba) (BuildId: 8c11d7b4ac6d685f0bba1cf2506a80f64d314582)
	    #4 0xf7bdefde  (/lib32/libc.so.6+0x28fde) (BuildId: 8c11d7b4ac6d685f0bba1cf2506a80f64d314582)
	    #5 0x83660bc  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x83660bc)
	    #6 0x82ed819  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x82ed819)
	    #7 0x8280195  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x8280195)
	    #8 0x82997f7  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x82997f7)
	    #9 0x826bbaa  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x826bbaa)
	    #10 0x8268086  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x8268086)
	    #11 0x822a4ee  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x822a4ee)
	    #12 0x8229589  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x8229589)
	    #13 0x80ea00e  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x80ea00e)
	    #14 0x80d4f6e  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x80d4f6e)
	    #15 0x80dab70  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x80dab70)
	    #16 0x8104757  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x8104757)
	    #17 0xf7bd0ed4  (/lib32/libc.so.6+0x1aed4) (BuildId: 8c11d7b4ac6d685f0bba1cf2506a80f64d314582)
	    #18 0x80cc0b5  (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_curl_d1a7f12cc2e5055727a9c66d5eca203f3c8f5a6c/revisions/curl_fuzzer_http+0x80cc0b5)


### I expected the following

No assert to trigger

### curl/libcurl version

git master (curl-8_2_0-24-gbc642cb33)

### operating system

Linux

[bagder]

@icing this assert seems to have code handling the case in run-time just following it, so I'm not sure about the importance of this trigger. What do you think?

[icing]

@bagder when writing this, I assumed that subsequent calls would satisfy the assertion. But, given the complexity of the transfer handling, I was not 100% sure.

So, my question would be: is the situation triggered by OSSFuzz also part of "normal" behaviour that we just do not see regularly? How does it trigger?

[cmeister2]

Architecture is i386. Test case it fails on is like this:

TLVHeader(type='Server response 9' (25), length=13, data=b"")
TLVHeader(type='CURLOPT_HTTP_VERSION' (40), length=4, data=b'')
TLVHeader(type='Server response 2' (18), length=0, data=b'')
TLVHeader(type='CURLOPT_URL' (1), length=34, data=b'')
TLVHeader(type='CURLOPT_USERAGENT' (45), length=34, data=b'')
TLVHeader(type='Server response 1' (17), length=39, data=b'')
TLVHeader(type='Server banner (sent on connection)' (2), length=225, data=b'')
TLVHeader(type='CURLOPT_USERPWD' (44), length=65534, data=b'')

Weirdest thing is probably a 65KB user+password setting...

[icing]

Interesting, will try to make a test case with this.

[cmeister2]

I'll try and repro locally too.

[cmeister2]

I have gdb working now so I can offer more details about the stack trace if necessary.

#5  0x0837d095 in cf_h2_send (cf=0xf2e00914, data=0xf2002a84, buf=0xf308c804, len=65536, err=0xffffb5f0)
    at /src/curl/lib/http2.c:2080
2080          DEBUGASSERT(len >= stream->upload_blocked_len);
(gdb) p len
$1 = 65536
(gdb) p stream->upload_blocked_len
$2 = 81853

[icing]

Is this on the current master? Because the 65536 looks suspiciously like a bug I fixed in c76df46.

[cmeister2]

No, it wouldn't be. If it's fixed it's likely to roll through to OSSFuzz in the next day or so.

[cmeister2]

@icing I've verified this still fails on latest master.

[icing]

@cmeister2 thanks for testing. Hmm, looking at Curl_buffer_send() which should be the one invoking this, I have no trust in this function. @bagder, I think we need to do some reviewing of this thing. I wonder what it does if writing is only partial twice. This http->backup thing looks suspect.

[cmeister2]

@icing if it helps I can write down step by step repro instructions