Memory leak in CURLOPT_XOAUTH2_BEARER

[Tachi107]

I did this

Given the following code:

#include <curl/curl.h>

int main(void) {
  curl_global_init(CURL_GLOBAL_ALL);

  CURL* curl = curl_easy_init();

  curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_BEARER);
  curl_easy_setopt(curl, CURLOPT_XOAUTH2_BEARER, "c4e448d652a961fda0ab64f882c8c161d5985f805d45d80c9ddca108f8e2fde3");
  curl_easy_setopt(curl, CURLOPT_HTTPGET, 1L);
  curl_easy_setopt(curl, CURLOPT_URL, "https://andrea.pappacoda.it");

  for (int i = 0; i < 5; i++) {
    curl_easy_perform(curl);
  }

  curl_easy_cleanup(curl);

  curl_global_cleanup();
}

AddressSanitizer reports a memory leak:

$ cc -g -fsanitize=address main.c $(pkg-config --cflags --libs libcurl) -o asan && ./asan
=================================================================
==41730==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 260 byte(s) in 4 object(s) allocated from:
    #0 0x7f52f54d97a7 in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cpp:454
    #1 0x7f52f54423cd  (/lib/x86_64-linux-gnu/libcurl.so.4+0x673cd)

SUMMARY: AddressSanitizer: 260 byte(s) leaked in 4 allocation(s).

and valgrind does too:

$ cc -g main.c $(pkg-config --cflags --libs libcurl) -o valgrind && valgrind --leak-check=full ./valgrind
==41878== 
==41878== HEAP SUMMARY:
==41878==     in use at exit: 3,710 bytes in 12 blocks
==41878==   total heap usage: 32,937 allocs, 32,925 frees, 3,397,085 bytes allocated
==41878== 
==41878== 260 bytes in 4 blocks are definitely lost in loss record 5 of 8
==41878==    at 0x483F7B5: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==41878==    by 0x499331A: strdup (strdup.c:42)
==41878==    by 0x48CB3CD: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.8.0)
==41878==    by 0x48AB9B7: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.8.0)
==41878==    by 0x48AC81D: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.8.0)
==41878==    by 0x4884AE2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.8.0)
==41878==    by 0x1092FB: main (main.c:15)
==41878== 
==41878== LEAK SUMMARY:
==41878==    definitely lost: 260 bytes in 4 blocks
==41878==    indirectly lost: 0 bytes in 0 blocks
==41878==      possibly lost: 0 bytes in 0 blocks
==41878==    still reachable: 3,450 bytes in 8 blocks
==41878==         suppressed: 0 bytes in 0 blocks
==41878== Reachable blocks (those to which a pointer was found) are not shown.
==41878== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==41878== 
==41878== For lists of detected and suppressed errors, rerun with: -s
==41878== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

The size of the leak depends on the value of CURLOPT_XOAUTH2_BEARER; in this example it is 260 because the token is 65 characters long (including '\0') times 4.

I expected the following

Repeatedly performing HTTP requests with the same handle when a bearer token is set with the CURLOPT_XOAUTH2_BEARER option should not leak the token on each request.

curl/libcurl version

curl 7.83.0 (x86_64-pc-linux-gnu) libcurl/7.83.0 OpenSSL/1.1.1o zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.10.0 nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.11
Release-Date: 2022-04-27
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

operating system

Linux debian 5.17.0-1-amd64 #1 SMP PREEMPT Debian 5.17.3-1 (2022-04-18) x86_64 GNU/Linux

[bagder]

I can reproduce with a new test case. PR pending.

[rlorigro]

Hi @bagder,

I am using AddressSanitizer on htslib, which also (as I understand it) relies on libcurl for fetching portions of remote GCS files with OAuth tokens, and I see a large leak from inside libcurl:

=================================================================
==28900==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 13996032 byte(s) in 2 object(s) allocated from:
    #0 0x7f5ceeab4c38 in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:164
    #1 0x7f5cef430401  (/lib/x86_64-linux-gnu/libcurl.so.4+0x1e401)

SUMMARY: AddressSanitizer: 13996032 byte(s) leaked in 2 allocation(s).

I am on ubuntu 22.04. Do you know if this is simply a version issue? Or somehow curl is misused here?

[dfandrich]

That log doesn't help narrow the problem. Can you provide a log from a libcurl with debug symbols? Note that we test libcurl for memory leaks on every submit, so leaks are much more likely to be a result of an application misusing the API rather than from any problem with libcurl itself. If you can provide the source code for a standalone program that shows a leak happening in libcurl, we'd be interested.

[rlorigro]

Thanks for the quick response. Here is a slightly more verbose output from valgrind:

==33413== 
==33413== HEAP SUMMARY:
==33413==     in use at exit: 5,611,520 bytes in 1 blocks
==33413==   total heap usage: 451,403 allocs, 451,402 frees, 69,411,210 bytes allocated
==33413== 
==33413== 5,611,520 bytes in 1 blocks are possibly lost in loss record 1 of 1
==33413==    at 0x484DCD3: realloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==33413==    by 0x48C0401: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.7.0)
==33413==    by 0x48EC5B7: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.7.0)
==33413==    by 0x4906B00: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.7.0)
==33413==    by 0x48E8193: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.7.0)
==33413==    by 0x48EB3BD: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.7.0)
==33413==    by 0x1AB591: wait_perform (hfile_libcurl.c:733)
==33413==    by 0x1ACCC7: restart_from_position (hfile_libcurl.c:1059)
==33413==    by 0x1AE063: libcurl_seek (hfile_libcurl.c:977)
==33413==    by 0x19F70D: hseek (hfile.c:469)
==33413==    by 0x18E630: bgzf_check_EOF_common (bgzf.c:1536)
==33413==    by 0x1900C7: bgzf_check_EOF (bgzf.c:2149)
==33413== 
==33413== LEAK SUMMARY:
==33413==    definitely lost: 0 bytes in 0 blocks
==33413==    indirectly lost: 0 bytes in 0 blocks
==33413==      possibly lost: 5,611,520 bytes in 1 blocks
==33413==    still reachable: 0 bytes in 0 blocks
==33413==         suppressed: 0 bytes in 0 blocks
==33413== 
==33413== For lists of detected and suppressed errors, rerun with: -s
==33413== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

Let me know if you need something more/different.

I have a test executable here which I am using to get the above output:
https://github.com/rlorigro/sv_merge/blob/main/src/test/test_hts_gcp.cpp

The project makes use of CMake to download and install the htslib dependency automatically, so with some basic libs, it should be possible to use the conventional build steps to get this running:

cd sv_merge
mkdir build
cd build
cmake ..
make -j [n_threads]

But I understand if this is too much to expect. I can try to take it up with the htslib authors if you think the issue is more likely on their end (or perhaps on my end).

Thanks

[bagder]

==33413== 5,611,520 bytes in 1 blocks are possibly lost in loss record 1 of 1
==33413== at 0x484DCD3: realloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)

5.6 megabytes in a single realloc done by libcurl? That looks highly unlikely.

My guess is on the write callback installed by the application.

[jmarshall]

FYI @rlorigro's problem turns out to have been #10971.

[bagder]

aaaaah, thanks for the update