HTTP POST with Negotiate fails

I did this

On Windows I want to perform a POST (or another custom) request (CURLOPT_CUSTOMREQUEST) with some data specified with CURLOPT_UPLOAD/CURLOPT_INFILESIZE_LARGE/CURLOPT_READFUNCTION/CURLOPT_SEEKFUNCTION by using negotiate authentication (CURLOPT_HTTPAUTH CURLAUTH_NEGOTIATE, CURLOPT_USERNAME (empty for Windows credentials), CURLOPT_PASSWORD (empty for Windows credentials)), but the authentication fails. Performing a GET request with the same setup succeeds.

Explicitly suppressing the "Expect: 100-continue" header seems to get the request working, but this is not an acceptable solution.

Using CURLOPT_POSTFIELDS instead of CURLOPT_UPLOAD, CURLOPT_READFUNCTION, CURLOPT_SEEKFUNCTION also works. But this is not the intended way to upload large amounts of data.
Using CURLOPT_POSTFIELDS with explicitly defining the "Expect: 100-continue" header fails.

Using CURLAUTH_NTLM instead of CURLAUTH_NEGOTIATE with the same cURL configuration works (also with the "Expect: 100-continue" mechanism). But this is less secure.

Connecting to an IIS with all authentication methods disabled but "Windows Authentication"

I expected the following

Negotiate authentication works also for POST/PUT-requests with CURLOPT_UPLOAD/CURLOPT_READFUNCTION. Handles with a GET request which performed a negotiate challenge can be reused to avoid challenge requests on every new request re-using a CURL handle.

curl/libcurl version

Used cURL Library: Version 7.52.1 64 Bit on Windows built with
-DBUILDING_LIBCURL -DHTTP_ONLY _DWITHOUT_MM_LIB -DUSE_SSLEAY -DUSE_OPENSSL -DCURL_WANTS_CA_BUNDLE_ENV -DENABLE_IPV6 -DUSE_WINDOWS_SSPI -DHAVE_STRUCT_POLLFD

operating system

Client: Tested on Windows 7 and Windows 2008 R2

connection close after negotiate challenge?

In cURL source code I found the following which seems to kill the connection after each negotiation. Why is this? Shouldn't the connection be kept after a (successful) negotiation for re-use?

If I comment out the "streamclose" part and I perform a GET request with negotiate authentication and reuse the same CURL handle for a following POST request with CURLOPT_UPLOAD the POST request succeeds as the authenticated connection will not be killed after the GET. Of course I cannot use this experimental trial as a workaround. I just wanted to find out how the connection handling of cURL works.

CURLcode Curl_http_done(struct connectdata *conn,
                        CURLcode status, bool premature)
{
  struct Curl_easy *data = conn->data;
  struct HTTP *http = data->req.protop;

  infof(data, "Curl_http_done: called premature == %d\n", premature);

  Curl_unencode_cleanup(conn);

#ifdef USE_SPNEGO
  if(data->state.proxyneg.state == GSS_AUTHSENT ||
     data->state.negotiate.state == GSS_AUTHSENT) {
    /* add forbid re-use if http-code != 401/407 as a WA only needed for
     * 401/407 that signal auth failure (empty) otherwise state will be RECV
     * with current code.
     * Do not close CONNECT_ONLY connections. */
    if((data->req.httpcode != 401) && (data->req.httpcode != 407) && // <-- uncommenting this
       !data->set.connect_only) // <-- uncommenting this
      streamclose(conn, "Negotiate transfer completed"); // <-- uncommenting this
    Curl_cleanup_negotiate(data);
  }
#endif
...

code to reproduce the problem:

#define CURL_STATICLIB
#include <curl/curl.h>

//
// Select one of the test cases:
//

// #1 Negotiate upload with CURLOPT_UPLOAD (which I expect to be working)

#define TEST_NEGOTIATE
#define TEST_UPLOAD

// ------------------------------------------------

// #2 Negotiate upload with CURLOPT_POSTFIELDS (works)
/*
#define TEST_NEGOTIATE
#define TEST_POSTFIELDS
*/
// ------------------------------------------------

// #3 NTLM upload (works)
/*
#define TEST_NTLM
#define TEST_UPLOAD
*/
// ------------------------------------------------

// #4 Negotiate GET and UPLOAD with same handle
// (after successful GET the connection is killed, PUT fails)
/*
#define TEST_NEGOTIATE
#define TEST_GET_WITH_UPLOAD
*/
// ------------------------------------------------


static int CURLdebugfunction(CURL *, curl_infotype infotype, char *str, size_t len, void *)
{
  const char *infostring = "CURLINFO_???";
  switch (infotype) {
  #define ENTRY(NAME) case NAME: infostring = TEXT(#NAME); break;
    ENTRY(CURLINFO_TEXT)
      ENTRY(CURLINFO_HEADER_IN)
      ENTRY(CURLINFO_HEADER_OUT)
      ENTRY(CURLINFO_DATA_IN)
      ENTRY(CURLINFO_DATA_OUT)
    #undef ENTRY
  }
  printf("CURLDEBUGINFO %s: %*.*s\n",
    (const char *)infostring, (int)len, (int)len, (const char *)str);
  return 0;
}
size_t CURLheaderfunction(void *ptr, size_t size, size_t nmemb, void *stream)
{
  char *hdr = (char *)malloc(size*nmemb + 1);
  if (hdr) {
    memcpy(hdr, ptr, size*nmemb);
    hdr[size*nmemb] = '\0';
    printf("RECEIVED HEADER: %s", (const char *)hdr);
    free(hdr);
  }
  else {
    printf("HEADER: OUT OF MEMORY\n");
  }
  return size*nmemb;
}
static size_t CURLreadpos = 0;
size_t CURLreadfunction(void *ptr, size_t size, size_t nmemb, void *stream)
{
  size_t read = 0;
  for (; CURLreadpos < 10 && CURLreadpos < size * nmemb; CURLreadpos++) {
    ((char *)ptr)[CURLreadpos] = 'a';
    read++;
  }
  printf("READ REQUEST DATA: %llu -> %llu\n", size*nmemb, (UINT64)read);
  return read;
}
int CURLseekfunction(void *ptr, curl_off_t offset, int origin)
{
  printf("SEEK: %lld, %d\n", (INT64)offset, origin);
  if (origin == SEEK_SET) {
    CURLreadpos = offset;
    return CURL_SEEKFUNC_OK;
  }
  printf("SEEK: NOT IMPL\n");
  return CURL_SEEKFUNC_CANTSEEK;
}
int main(void)
{
  CURL *curl;
  CURLcode res;
  curl_global_init(CURL_GLOBAL_ALL);
  curl = curl_easy_init();
  if (curl) {
    curl_easy_setopt(curl, CURLOPT_URL, "http://url-to-server-with-windows-auth-goes-here");
    curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, CURLheaderfunction);
    curl_easy_setopt(curl, CURLOPT_READFUNCTION, CURLreadfunction);
    curl_easy_setopt(curl, CURLOPT_SEEKFUNCTION, CURLseekfunction);
    curl_easy_setopt(curl, CURLOPT_DEBUGFUNCTION, CURLdebugfunction);
    curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

  #if defined(TEST_NEGOTIATE)
    curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NEGOTIATE);
    curl_easy_setopt(curl, CURLOPT_USERNAME, "");
    curl_easy_setopt(curl, CURLOPT_PASSWORD, "");
  #endif
  #if defined(TEST_NTLM)
    curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NTLM);
    curl_easy_setopt(curl, CURLOPT_USERNAME, "<NTLM-User>");
    curl_easy_setopt(curl, CURLOPT_PASSWORD, "<NTLM-Password>");
  #endif

  #if defined(TEST_UPLOAD)
      curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
      curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
      curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, 10);
  #endif
  #if defined(TEST_POSTFIELDS)
    curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "aaaaaaaaaa");
  #endif
  #if defined(TEST_GET_WITH_UPLOAD)
    curl_easy_setopt(curl, CURLOPT_HTTPGET, 1L);
  #endif

    res = curl_easy_perform(curl);

  #if defined(TEST_GET_WITH_UPLOAD)
    curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
    curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
    curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, 10);
    res = curl_easy_perform(curl);
  #endif

    if (res != CURLE_OK) {
      printf("curl_easy_perform() failed: %s\n",
        curl_easy_strerror(res));
    }
    else {
      long http_code = 0;
      curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code);
      printf("CURL STATUS: %ld\n", http_code);
    }
    curl_easy_cleanup(curl);
  }
  curl_global_cleanup();
  return 0;
}

test program output

Output test case # 1 (Negotiate upload)
#1_negotiate_upload_failed.txt

Output test case # 2 (Negotiate postfields)
#2_negotiate_postfields_succeeded.txt

Output test case # 3 (NTLM upload)
#3_ntlm_upload_succeeded.txt

Output test case # 4 (Negotiate upload with same handle after GET)
#4_negotiate_upload_after_successful_get_failed.txt

Can anybody reproduce this?
Do you need some additional information?

It happens to our system also. Unless the "Expect: ..." is suppressed it doesn't work. Due to lack of time, setting this header is our official workaround. Although not ideal as curl has to re-send the data after auth.

For me suppressing the Expect-Header works if directly connected to the HTTP server for small requests, but if going through a proxy (CCProxy 8.0 on Windows in my test), the first request which is a POST request in my case succeeds, but the second request (on the same cURL handle) which is a GET request does not get any response and hangs/times out. Could be that this is a CCProxy issue, but I have also seen that problem without proxy after some more requests have already been performed with cURL, maybe depending on the state of the cURL's connection cache.
Suppressing the Expect-Header does also not work for me if the request body is more than 32kb (or randomly already at 16kb). After providing that data via the cURL read function, the server seems to stop the request and and sends a 401 somewhere in the middle of the challenge handshake and cURL aborts the challenge/request. The cURL seek function seems to be called properly in that case.

I'd expect Negotiate POST to be not optimal in case of multi-round negotiation (usually NTLM inside), because there is no special handling like the 'authneg' logic for NTLM, see more in this mail thread:
https://curl.haxx.se/mail/lib-2016-11/0196.html

However, it sounds like some flows (like with Expect) behave better than others so perhaps it could be fixed up to at least work.

No, I can't get it working with uploading big files. cURL calls the read function multiple times (for each challenge request) twice, in combination with the seek function and and after that the server responds always with 401; the full content is never transferred to the server.
What can I do to upload big files with Negotiate?

#define CURL_STATICLIB
#include <curl/curl.h>

// Test case #1
#define TEST_NEGOTIATE
#define UPLOAD_SIZE (1 * 1024 * 1024)
#define TEST_SUPPRESS_EXPECT_CONTINUE

// Test case #2
/*
#define TEST_NTLM
#define UPLOAD_SIZE (1 * 1024 * 1024)
*/

// Test case #3
/*
#define TEST_NEGOTIATE
#define UPLOAD_SIZE 10
#define TEST_SUPPRESS_EXPECT_CONTINUE
*/

static int CURLdebugfunction(CURL *, curl_infotype infotype, char *str, size_t len, void *)
{
  const char *infostring = "CURLINFO_???";
  switch (infotype) {
  #define ENTRY(NAME) case NAME: infostring = TEXT(#NAME); break;
    ENTRY(CURLINFO_TEXT)
      ENTRY(CURLINFO_HEADER_IN)
      ENTRY(CURLINFO_HEADER_OUT)
      ENTRY(CURLINFO_DATA_IN)
      ENTRY(CURLINFO_DATA_OUT)
    #undef ENTRY
  }
  printf("CURLDEBUGINFO %s: %*.*s\n",
    (const char *)infostring, (int)len, (int)len, (const char *)str);
  return 0;
}
size_t CURLheaderfunction(void *ptr, size_t size, size_t nmemb, void *stream)
{
  char *hdr = (char *)malloc(size*nmemb + 1);
  if (hdr) {
    memcpy(hdr, ptr, size*nmemb);
    hdr[size*nmemb] = '\0';
    printf("RECEIVED HEADER: %s", (const char *)hdr);
    free(hdr);
  }
  else {
    printf("HEADER: OUT OF MEMORY\n");
  }
  return size*nmemb;
}
static size_t CURLreadpos = 0;
size_t CURLreadfunction(void *ptr, size_t size, size_t nmemb, void *stream)
{
  size_t read = 0;
  for (; CURLreadpos < UPLOAD_SIZE && read < size * nmemb;) {
    ((char *)ptr)[read] = 'a';
    read++;
    CURLreadpos++;
  }
  printf("READ REQUEST DATA: %llu -> %llu\n", size*nmemb, (UINT64)read);
  return read;
}
int CURLseekfunction(void *ptr, curl_off_t offset, int origin)
{
  printf("SEEK: %lld, %d\n", (INT64)offset, origin);
  if (origin == SEEK_SET) {
    CURLreadpos = offset;
    return CURL_SEEKFUNC_OK;
  }
  printf("SEEK: NOT IMPL\n");
  return CURL_SEEKFUNC_CANTSEEK;
}
int main(void)
{
  CURL *curl;
  CURLcode res;
  curl_global_init(CURL_GLOBAL_ALL);
  curl = curl_easy_init();
  if (curl) {
    curl_easy_setopt(curl, CURLOPT_URL, "http://url-to-server-with-windows-auth-goes-here");
    curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, CURLheaderfunction);
    curl_easy_setopt(curl, CURLOPT_READFUNCTION, CURLreadfunction);
    curl_easy_setopt(curl, CURLOPT_SEEKFUNCTION, CURLseekfunction);
    curl_easy_setopt(curl, CURLOPT_DEBUGFUNCTION, CURLdebugfunction);
    curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

  #if defined(TEST_NEGOTIATE)
    curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NEGOTIATE);
  #endif
  #if defined(TEST_NTLM)
    curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NTLM);
  #endif
    curl_easy_setopt(curl, CURLOPT_USERNAME, "");
    curl_easy_setopt(curl, CURLOPT_PASSWORD, "");

    curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
    curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
    curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, UPLOAD_SIZE);

  #if defined(TEST_SUPPRESS_EXPECT_CONTINUE)
    struct curl_slist *headers = NULL;
    headers = curl_slist_append(headers, "Expect:");
    curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
  #endif

    res = curl_easy_perform(curl);

    if (res != CURLE_OK) {
      printf("curl_easy_perform() failed: %s\n",
        curl_easy_strerror(res));
    }
    else {
      long http_code = 0;
      curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code);
      printf("CURL STATUS: %ld\n", http_code);
    }
    curl_easy_cleanup(curl);
  }
  curl_global_cleanup();
  return 0;
}

test program output

Negotiate with suppressing expect/continue and uploading 1 MiB
#1_negotiate_suppress_expect_1mib_failed.txt

NTLM with uploading 1 MiB (not suppressing expect/continue)
#2_ntlm_1mib_succeeded.txt

Negotiate with suppressing expect/continue and uploading 10 bytes
#3_negotiate_supporess_expect_10bytes_succeeded.txt

We have the same problem in our environment with negotiate, is there any possability to get it fixed?

I have created a fix which solves the following problems with CURLAUTH_NEGOTIATE:

Immediately initialize negotiate challenge on the first challenge request (This was not done so far)
Do not send content while negotiating (Content-Length must be zero in this phase)
Fix Expect: 100-continue handling
Enable reusing easy handles to avoid re-challlenging every request
Enable restarting negotiate authentication on a already authenticated connection

I changed the following:

Added GSS_AUTHDONE state to negotiatedata which represents a connection with completed negotiation
Added GSS_AUTHSUCC state to negotiatedata which represents a fully authenticated negotiated connection
Moved and adjusted authentication state handling from output_auth_headers into Curl_output_negotiate and Curl_auth_spnego_cleanup (All other challenge based auth methods do this)
Commented out the code which closed the connection after a negotiated request
Add read stream rewind logic analogical to the NTLM stream rewind handling.
Moved negotiatedata members from UrlState to connectdata
Consider authproblem state for CURLAUTH_NEGOTIATE
Consider reuse_forbid for CURLAUTH_NEGOTIATE
Consider negotiate authentication restart on a already fully authenticated connection

I have tested this code (only on Windows!) by simultaneously downloading and uploading small and big amounts of data and reusing already created connections and either directly connecting to a server or doing negotiate authentication through a proxy server. I have not tested proxy negotiate authentication.
Regards,
Dominik

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

PR is still active

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

The PR is still active.

Navigation Menu

Repository navigation

HTTP POST with Negotiate fails #1261

I did this

I expected the following

curl/libcurl version

operating system

connection close after negotiate challenge?

code to reproduce the problem:

test program output

test program output

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

HTTP POST with Negotiate fails #1261

Description

I did this

I expected the following

curl/libcurl version

operating system

connection close after negotiate challenge?

code to reproduce the problem:

test program output

Activity

dhoelzl commented on Feb 22, 2017

jzakrzewski commented on Feb 22, 2017

dhoelzl commented on Feb 24, 2017

iboukris commented on Feb 24, 2017

dhoelzl commented on Feb 27, 2017

test program output

Nitrael86 commented on Jul 17, 2017

dhoelzl commented on Oct 9, 2017

stale commented on Apr 15, 2018

dhoelzl commented on Apr 16, 2018

stale commented on Mar 12, 2019

MarcelRaad commented on Mar 12, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions