CURLOPT_READFUNCTION not called following empty POST

[RanBarLavie]

Problem Description

My application uses libcurl to send two POST requests, one after the other, over the same connection.

1. The 1st one is a POST request with no content (Content-Length: 0).
2. The 2nd is a "chunked" Transfer-Encoding POST request, which supplies the content with the CURLOPT_READFUNCTION callback.

The 1st request goes through and receives a response, as expected.
For some reason, the 2nd request only sends out the header. It seems libcurl never calls the callback function to consume the content.

Reproduce

The problem can be reproduced like this:

#include <curl/curl.h>
#include <assert.h>

static size_t read_callback(char* buffer, size_t size, size_t nitems, void* userdata);

int main(int argc, char* argv[])
{
  CURL* curl = curl_easy_init();
  assert(curl != NULL);
  const char* url = argc > 1 ? argv[1] : "http://localhost:8080/";
  curl_easy_setopt(curl, CURLOPT_URL, url);
  curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

  /* 1st request is a POST with no content */
  curl_easy_setopt(curl, CURLOPT_POST, 1L);
  curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, 0L);
  curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "");
  CURLcode result = curl_easy_perform(curl);
  assert(result == CURLE_OK);

  /* 2nd request is a POST with "chunked" transfer encoding */
  curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_callback);
  curl_easy_setopt(curl, CURLOPT_POSTFIELDS, NULL);
  struct curl_slist* slist = curl_slist_append(NULL, "Transfer-Encoding: chunked");
  curl_easy_setopt(curl, CURLOPT_HTTPHEADER, slist);
  result = curl_easy_perform(curl);
  assert(result == CURLE_OK);

  curl_slist_free_all(slist);
  curl_easy_cleanup(curl);

  return 0;
}

static size_t read_callback(char* buffer, size_t size, size_t nitems, void* userdata)
{
  static int done = 0;
  if (done) {
    return 0;
  }
  done = 1;
  assert(size * nitems >= 2);
  buffer[0] = 'H';
  buffer[1] = 'i';
  return 2;
}

Output:

./curltest http://httpbin.org/post
* About to connect() to httpbin.org port 80 (#0)
*   Trying 34.205.150.168...
* Connected to httpbin.org (34.205.150.168) port 80 (#0)
> POST /post HTTP/1.1
Host: httpbin.org
Accept: */*
Content-Length: 0
Content-Type: application/x-www-form-urlencoded

< HTTP/1.1 200 OK
< (...response truncated for conciseness...)
* Connection #0 to host httpbin.org left intact
* Found bundle for host httpbin.org: 0x1e56ab0
* Re-using existing connection! (#0) with host httpbin.org
* Connected to httpbin.org (34.205.150.168) port 80 (#0)
> POST /post HTTP/1.1
Host: httpbin.org
Accept: */*
Transfer-Encoding: chunked
Content-Type: application/x-www-form-urlencoded

And that's where program execution hangs, instead of sending out the message body.

Debug:
reproduce.txt

Workaround

Interestingly, the problem is eliminated by changing the 1st request to send some content. Even 1 byte does the trick, like this for example:

  curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, 1L);
  curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "X");

The fact that the problem (in the 2nd request) is fixed just by setting some content in the body of the 1st request makes this look like a bug.

This is the output with the workaround in place:
workaround.txt

Version

libcurl version: 7.87.0

[curl -V output]

# curl -V
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.87.0 OpenSSL/1.0.2k-fips zlib/1.2.7 nghttp2/1.31.1
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS Debug TrackMemory IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets

Operating System

# uname -a
Linux blade-162-115 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Any insight will be appreciated. I will be happy to provide more information as needed.
Thank you!

[dfandrich]

This looks like expected behaviour to me. You set CURLOPT_POSTFIELDSIZE=0 for the first request but you don't change it for the second request, so it sends 0 for that one, too.

[RanBarLavie]

Thanks @dfandrich for looking into this and replying so fast.
The reason CURLOPT_POSTFIELDSIZE remains 0 in the 2nd request is that it's a "chunked" transfer encoding request, which does not know the content length in advance.
I realize that setting CURLOPT_POSTFIELDSIZE=2 in the 2nd request would eliminate the problem in the test app. But it's not clear what a "real-world" application can do in this case, if it doesn't know the payload size in advance.

[dfandrich]

I believe you need to set CURLOPT_POSTFIELDSIZE to -1 to get that behaviour.

[RanBarLavie]

Yes! That solves it.
Now I see that -1 is the default value for CURLOPT_POSTFIELDSIZE, so it all makes sense.

The documentation is a bit confusing here - CURLOPT_POSTFIELDSIZE only states that -1 will cause libcurl to strlen the data, and not mentioning its impact on the read callback.

So, I'll do a bit more testing and then close this issue.
Thank you @dfandrich !