Payload dropped from POST when using proxy with NTLM authentication

I did this

Performed a POST request to an API that I am using (https://api.emarsys.net/api/v2/email/getlaunchesofemail, documentation here) using NTLM authentication for the corporate proxy that I am behind. The JSON payload is however dropped from the request as is evidenced by the Content-Length: 0 header.

(This issue is related to the downstream use of the R curl package, see jeroen/curl#146.)

$ curl --proxy http://<proxy_url>:<proxy_port> \
	--proxy-ntlm \
	--proxy-user : \
	--verbose \
	--include \
	--header "Content-Type: application/json" \
	--header 'X-WSSE: UsernameToken <token>' \
	--data '{"emailId":"1234"}' \
	https://api.emarsys.net/api/v2/email/getlaunchesofemail

* timeout on name lookup is not supported
*   Trying <proxy_ip>...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to <proxy_url> (<proxy_ip>) port <proxy_port> (#0)
* Establish HTTP proxy tunnel to api.emarsys.net:443
* Proxy auth using NTLM with user ''
> CONNECT api.emarsys.net:443 HTTP/1.1
> Host: api.emarsys.net:443
> Proxy-Authorization: NTLM <token>
> User-Agent: curl/7.53.0
> Proxy-Connection: Keep-Alive
>
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [109 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [5413 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: OU=Domain Control Validated; OU=EssentialSSL Wildcard; CN=*.emarsys.net
*  start date: Nov 20 00:00:00 2015 GMT
*  expire date: Dec 16 23:59:59 2018 GMT
*  subjectAltName: host "api.emarsys.net" matched cert's "*.emarsys.net"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
} [5 bytes data]
> POST /api/v2/email/getlaunchesofemail HTTP/1.1
> Host: api.emarsys.net
> User-Agent: curl/7.53.0
> Accept: */*
> Content-Type: application/json
> X-WSSE: UsernameToken <token>
> Content-Length: 0
>
{ [5 bytes data]
< HTTP/1.1 400 Bad Request
< Server: openresty/1.11.2.2
< Date: Wed, 28 Mar 2018 11:33:56 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 59
< Connection: keep-alive
< X-Ratelimit-Limit: 1000
< X-Ratelimit-Reset: 1522236840
< X-Ratelimit-Remaining: 999
< Strict-Transport-Security: max-age=15552000; includeSubDomains; preload
< X-Suite-Response-Time: 19
< Vary: Accept-Encoding
< Cache-Control: max-age=0, no-cache, no-store, must-revalidate
< Pragma: no-cache
<
{ [59 bytes data]
100    59  100    59    0     0     34      0  0:00:01  0:00:01 --:--:--    35HTTP/1.1 200 Connection established

HTTP/1.1 400 Bad Request
Server: openresty/1.11.2.2
Date: Wed, 28 Mar 2018 11:33:56 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 59
Connection: keep-alive
X-Ratelimit-Limit: 1000
X-Ratelimit-Reset: 1522236840
X-Ratelimit-Remaining: 999
Strict-Transport-Security: max-age=15552000; includeSubDomains; preload
X-Suite-Response-Time: 19
Vary: Accept-Encoding
Cache-Control: max-age=0, no-cache, no-store, must-revalidate
Pragma: no-cache

{"replyCode":6004,"replyText":"Invalid email ID","data":""}
* Connection #0 to host <proxy_url> left intact

I expected the following

A POST request to https://api.emarsys.net/api/v2/email/getlaunchesofemail with a non-zero Content-Length header and the following JSON object as the payload:

{
   "emailId":"1234"
}

curl/libcurl #version

curl 7.53.0 (x86_64-w64-mingw32) libcurl/7.53.0 OpenSSL/1.0.2k zlib/1.2.11 libssh2/1.8.0 nghttp2/1.19.0 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smtp smtps telnet tftp
Features: IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL libz TLS-SRP HTTP2 HTTPS-proxy Metalink

operating system

Microsoft Windows 7 Professional
6.1.7601 Service Pack 1, build 7601

Where did the " in X-WSSE: UsernameToken <token>" come from?
Probably some shell SNAFU in action. In such cases, it's handy to use the --libcurl - option to see what CURLOPT_x curl is using. With this post-test.bat:

@echo off
setlocal
set DATA=%TEMP%\post-data
echo {"emailId":"1234"} > %DATA%
set header1="Content-Type: application/json"
set header2="X-WSSE: UsernameToken whatever01234"
curl.exe %* --header %header1% --header %header2% --data @%DATA% --libcurl - https://api.emarsys.net/api/v2/email/getlaunchesofemail

curl seems to send what was intended (except for the proxy stuff):

 slist1 = curl_slist_append(slist1, "Content-Type: application/json");
 slist1 = curl_slist_append(slist1, "X-WSSE: UsernameToken whatever01234");
 ...
 curl_easy_setopt(hnd, CURLOPT_HTTPHEADER, slist1);
 curl_easy_setopt(hnd, CURLOPT_POSTFIELDS, "{\"emailId\":\"1234\"}");
 curl_easy_setopt(hnd, CURLOPT_POSTFIELDSIZE_LARGE, (curl_off_t)18);

I see no Content-length: 0 with the above.

Here's what I believe happens:

This asks for NTLM auth with the proxy, but as can be seen in the verbose output there's no NTLM negotiation happening so I suspect that curl doesn't correctly note the "auth phase done" there and when it is time to send the POST request, it thinks it is still in a "multi-pass authentication phase" during which it skips sending the request body for data-saving purposes.

Work-around: try --proxy-anyauth instead of --proxy-ntlm, it should make it work for NTLM-proxies but even for this case without auth.

@gvanem

@gvanem The quote in X-WSSE: UsernameToken <token>" was part of the token, which I forgot to remove...

The generated libcurl code using the --libcurl option is:

/********* Sample code generated by the curl command line tool **********
 * All curl_easy_setopt() options are documented at:
 * https://curl.haxx.se/libcurl/c/curl_easy_setopt.html
 ************************************************************************/
#include <curl/curl.h>

int main(int argc, char *argv[])
{
  CURLcode ret;
  CURL *hnd;
  struct curl_slist *slist1;

  slist1 = NULL;
  slist1 = curl_slist_append(slist1, "Content-Type: application/json");
  slist1 = curl_slist_append(slist1, "X-WSSE: UsernameToken <token>");

  hnd = curl_easy_init();
  curl_easy_setopt(hnd, CURLOPT_URL, "https://api.emarsys.net/api/v2/email/getlaunchesofemail");
  curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
  curl_easy_setopt(hnd, CURLOPT_PROXY, "http://<proxy_url>:<proxy_port>");
  curl_easy_setopt(hnd, CURLOPT_PROXYUSERPWD, ":");
  curl_easy_setopt(hnd, CURLOPT_PROXYAUTH, (long)CURLAUTH_NTLM);
  curl_easy_setopt(hnd, CURLOPT_POSTFIELDS, "{\"emailId\":\"1234\"} ");
  curl_easy_setopt(hnd, CURLOPT_POSTFIELDSIZE_LARGE, (curl_off_t)19);
  curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.53.0");
  curl_easy_setopt(hnd, CURLOPT_HTTPHEADER, slist1);
  curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L);
  curl_easy_setopt(hnd, CURLOPT_HTTP_VERSION, (long)CURL_HTTP_VERSION_2TLS);
  curl_easy_setopt(hnd, CURLOPT_SSH_KNOWNHOSTS, "C:\\Users\\<username>\\AppData\\Roaming/_ssh/known_hosts");
  curl_easy_setopt(hnd, CURLOPT_VERBOSE, 1L);
  curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L);

  /* Here is a list of options the curl code used that cannot get generated
     as source easily. You may select to either not use them or implement
     them yourself.

  CURLOPT_WRITEDATA set to a objectpointer
  CURLOPT_INTERLEAVEDATA set to a objectpointer
  CURLOPT_WRITEFUNCTION set to a functionpointer
  CURLOPT_READDATA set to a objectpointer
  CURLOPT_READFUNCTION set to a functionpointer
  CURLOPT_SEEKDATA set to a objectpointer
  CURLOPT_SEEKFUNCTION set to a functionpointer
  CURLOPT_ERRORBUFFER set to a objectpointer
  CURLOPT_STDERR set to a objectpointer
  CURLOPT_DEBUGFUNCTION set to a functionpointer
  CURLOPT_DEBUGDATA set to a objectpointer
  CURLOPT_HEADERFUNCTION set to a functionpointer
  CURLOPT_HEADERDATA set to a objectpointer

  */

  ret = curl_easy_perform(hnd);

  curl_easy_cleanup(hnd);
  hnd = NULL;
  curl_slist_free_all(slist1);
  slist1 = NULL;

  return (int)ret;
}
/**** End of sample code ****/

So interestingly the CURLOPT_POSTFIELDS option does contain the correct payload, but curl still drops it from the actual request.

@bagder yes, --proxy-anyauth works, but I think in my case it is resorting to Negotiate authentication, so I cannot confirm this completely.

So interestingly the CURLOPT_POSTFIELDS option does contain the correct payload,

That wasn't clear from your 1st post. So maybe curl --verbose isn't verbose enough?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

There's an active PR for this now.

Navigation Menu

Repository navigation

Payload dropped from POST when using proxy with NTLM authentication #2431

I did this

I expected the following

curl/libcurl #version

operating system

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Payload dropped from POST when using proxy with NTLM authentication #2431

Description

I did this

I expected the following

curl/libcurl #version

operating system

Activity

gvanem commented on Mar 28, 2018

bagder commented on Mar 28, 2018

matthijsvanderloos commented on Mar 28, 2018

gvanem commented on Mar 28, 2018

stale commented on Mar 12, 2019

MarcelRaad commented on Mar 12, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions