Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Payload dropped from POST when using proxy with NTLM authentication #2431

Closed
matthijsvanderloos opened this issue Mar 28, 2018 · 6 comments

Comments

Projects
None yet
4 participants
@matthijsvanderloos
Copy link

commented Mar 28, 2018

I did this

Performed a POST request to an API that I am using (https://api.emarsys.net/api/v2/email/getlaunchesofemail, documentation here) using NTLM authentication for the corporate proxy that I am behind. The JSON payload is however dropped from the request as is evidenced by the Content-Length: 0 header.

(This issue is related to the downstream use of the R curl package, see jeroen/curl#146.)

$ curl --proxy http://<proxy_url>:<proxy_port> \
	--proxy-ntlm \
	--proxy-user : \
	--verbose \
	--include \
	--header "Content-Type: application/json" \
	--header 'X-WSSE: UsernameToken <token>' \
	--data '{"emailId":"1234"}' \
	https://api.emarsys.net/api/v2/email/getlaunchesofemail

* timeout on name lookup is not supported
*   Trying <proxy_ip>...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to <proxy_url> (<proxy_ip>) port <proxy_port> (#0)
* Establish HTTP proxy tunnel to api.emarsys.net:443
* Proxy auth using NTLM with user ''
> CONNECT api.emarsys.net:443 HTTP/1.1
> Host: api.emarsys.net:443
> Proxy-Authorization: NTLM <token>
> User-Agent: curl/7.53.0
> Proxy-Connection: Keep-Alive
>
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [109 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [5413 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: OU=Domain Control Validated; OU=EssentialSSL Wildcard; CN=*.emarsys.net
*  start date: Nov 20 00:00:00 2015 GMT
*  expire date: Dec 16 23:59:59 2018 GMT
*  subjectAltName: host "api.emarsys.net" matched cert's "*.emarsys.net"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
} [5 bytes data]
> POST /api/v2/email/getlaunchesofemail HTTP/1.1
> Host: api.emarsys.net
> User-Agent: curl/7.53.0
> Accept: */*
> Content-Type: application/json
> X-WSSE: UsernameToken <token>
> Content-Length: 0
>
{ [5 bytes data]
< HTTP/1.1 400 Bad Request
< Server: openresty/1.11.2.2
< Date: Wed, 28 Mar 2018 11:33:56 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 59
< Connection: keep-alive
< X-Ratelimit-Limit: 1000
< X-Ratelimit-Reset: 1522236840
< X-Ratelimit-Remaining: 999
< Strict-Transport-Security: max-age=15552000; includeSubDomains; preload
< X-Suite-Response-Time: 19
< Vary: Accept-Encoding
< Cache-Control: max-age=0, no-cache, no-store, must-revalidate
< Pragma: no-cache
<
{ [59 bytes data]
100    59  100    59    0     0     34      0  0:00:01  0:00:01 --:--:--    35HTTP/1.1 200 Connection established

HTTP/1.1 400 Bad Request
Server: openresty/1.11.2.2
Date: Wed, 28 Mar 2018 11:33:56 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 59
Connection: keep-alive
X-Ratelimit-Limit: 1000
X-Ratelimit-Reset: 1522236840
X-Ratelimit-Remaining: 999
Strict-Transport-Security: max-age=15552000; includeSubDomains; preload
X-Suite-Response-Time: 19
Vary: Accept-Encoding
Cache-Control: max-age=0, no-cache, no-store, must-revalidate
Pragma: no-cache

{"replyCode":6004,"replyText":"Invalid email ID","data":""}
* Connection #0 to host <proxy_url> left intact

I expected the following

A POST request to https://api.emarsys.net/api/v2/email/getlaunchesofemail with a non-zero Content-Length header and the following JSON object as the payload:

{
   "emailId":"1234"
}

curl/libcurl #version

curl 7.53.0 (x86_64-w64-mingw32) libcurl/7.53.0 OpenSSL/1.0.2k zlib/1.2.11 libssh2/1.8.0 nghttp2/1.19.0 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smtp smtps telnet tftp
Features: IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL libz TLS-SRP HTTP2 HTTPS-proxy Metalink

operating system

Microsoft Windows 7 Professional
6.1.7601 Service Pack 1, build 7601

@gvanem

This comment has been minimized.

Copy link
Member

commented Mar 28, 2018

Where did the " in X-WSSE: UsernameToken <token>" come from?
Probably some shell SNAFU in action. In such cases, it's handy to use the --libcurl - option to see what CURLOPT_x curl is using. With this post-test.bat:

@echo off
setlocal
set DATA=%TEMP%\post-data
echo {"emailId":"1234"} > %DATA%
set header1="Content-Type: application/json"
set header2="X-WSSE: UsernameToken whatever01234"
curl.exe %* --header %header1% --header %header2% --data @%DATA% --libcurl - https://api.emarsys.net/api/v2/email/getlaunchesofemail

curl seems to send what was intended (except for the proxy stuff):

 slist1 = curl_slist_append(slist1, "Content-Type: application/json");
 slist1 = curl_slist_append(slist1, "X-WSSE: UsernameToken whatever01234");
 ...
 curl_easy_setopt(hnd, CURLOPT_HTTPHEADER, slist1);
 curl_easy_setopt(hnd, CURLOPT_POSTFIELDS, "{\"emailId\":\"1234\"}");
 curl_easy_setopt(hnd, CURLOPT_POSTFIELDSIZE_LARGE, (curl_off_t)18);

I see no Content-length: 0 with the above.

@bagder

This comment has been minimized.

Copy link
Member

commented Mar 28, 2018

Here's what I believe happens:

This asks for NTLM auth with the proxy, but as can be seen in the verbose output there's no NTLM negotiation happening so I suspect that curl doesn't correctly note the "auth phase done" there and when it is time to send the POST request, it thinks it is still in a "multi-pass authentication phase" during which it skips sending the request body for data-saving purposes.

Work-around: try --proxy-anyauth instead of --proxy-ntlm, it should make it work for NTLM-proxies but even for this case without auth.

@matthijsvanderloos

This comment has been minimized.

Copy link
Author

commented Mar 28, 2018

@gvanem The quote in X-WSSE: UsernameToken <token>" was part of the token, which I forgot to remove...

The generated libcurl code using the --libcurl option is:

/********* Sample code generated by the curl command line tool **********
 * All curl_easy_setopt() options are documented at:
 * https://curl.haxx.se/libcurl/c/curl_easy_setopt.html
 ************************************************************************/
#include <curl/curl.h>

int main(int argc, char *argv[])
{
  CURLcode ret;
  CURL *hnd;
  struct curl_slist *slist1;

  slist1 = NULL;
  slist1 = curl_slist_append(slist1, "Content-Type: application/json");
  slist1 = curl_slist_append(slist1, "X-WSSE: UsernameToken <token>");

  hnd = curl_easy_init();
  curl_easy_setopt(hnd, CURLOPT_URL, "https://api.emarsys.net/api/v2/email/getlaunchesofemail");
  curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
  curl_easy_setopt(hnd, CURLOPT_PROXY, "http://<proxy_url>:<proxy_port>");
  curl_easy_setopt(hnd, CURLOPT_PROXYUSERPWD, ":");
  curl_easy_setopt(hnd, CURLOPT_PROXYAUTH, (long)CURLAUTH_NTLM);
  curl_easy_setopt(hnd, CURLOPT_POSTFIELDS, "{\"emailId\":\"1234\"} ");
  curl_easy_setopt(hnd, CURLOPT_POSTFIELDSIZE_LARGE, (curl_off_t)19);
  curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.53.0");
  curl_easy_setopt(hnd, CURLOPT_HTTPHEADER, slist1);
  curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L);
  curl_easy_setopt(hnd, CURLOPT_HTTP_VERSION, (long)CURL_HTTP_VERSION_2TLS);
  curl_easy_setopt(hnd, CURLOPT_SSH_KNOWNHOSTS, "C:\\Users\\<username>\\AppData\\Roaming/_ssh/known_hosts");
  curl_easy_setopt(hnd, CURLOPT_VERBOSE, 1L);
  curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L);

  /* Here is a list of options the curl code used that cannot get generated
     as source easily. You may select to either not use them or implement
     them yourself.

  CURLOPT_WRITEDATA set to a objectpointer
  CURLOPT_INTERLEAVEDATA set to a objectpointer
  CURLOPT_WRITEFUNCTION set to a functionpointer
  CURLOPT_READDATA set to a objectpointer
  CURLOPT_READFUNCTION set to a functionpointer
  CURLOPT_SEEKDATA set to a objectpointer
  CURLOPT_SEEKFUNCTION set to a functionpointer
  CURLOPT_ERRORBUFFER set to a objectpointer
  CURLOPT_STDERR set to a objectpointer
  CURLOPT_DEBUGFUNCTION set to a functionpointer
  CURLOPT_DEBUGDATA set to a objectpointer
  CURLOPT_HEADERFUNCTION set to a functionpointer
  CURLOPT_HEADERDATA set to a objectpointer

  */

  ret = curl_easy_perform(hnd);

  curl_easy_cleanup(hnd);
  hnd = NULL;
  curl_slist_free_all(slist1);
  slist1 = NULL;

  return (int)ret;
}
/**** End of sample code ****/

So interestingly the CURLOPT_POSTFIELDS option does contain the correct payload, but curl still drops it from the actual request.

@bagder yes, --proxy-anyauth works, but I think in my case it is resorting to Negotiate authentication, so I cannot confirm this completely.

@gvanem

This comment has been minimized.

Copy link
Member

commented Mar 28, 2018

So interestingly the CURLOPT_POSTFIELDS option does contain the correct payload,

That wasn't clear from your 1st post. So maybe curl --verbose isn't verbose enough?

@stale

This comment has been minimized.

Copy link

commented Mar 12, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Mar 12, 2019

@MarcelRaad

This comment has been minimized.

Copy link
Member

commented Mar 12, 2019

There's an active PR for this now.

@stale stale bot removed the stale label Mar 12, 2019

@bagder bagder closed this in dd8a19f Mar 13, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.