New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unsupported Request Method and Protocol behind a proxy #2018

Closed
mazong1123 opened this Issue Oct 26, 2017 · 6 comments

Comments

Projects
None yet
2 participants
@mazong1123
Copy link

mazong1123 commented Oct 26, 2017

Background

  1. I am behind a proxy. And 127.0.0.1 is not bypassed.
  2. curl -V show me that sftp is supported.

I did this

curl -u sftptest:passwd sftp://127.0.0.1:22/home/sftptest/

I expected the following

The sftp connection should be successful. However, I received:

...
<title>ERROR: The requested URL could not be retrieved</title>
...
<p><b>Unsupported Request Method and Protocol</b></p>

When I open the verbose option I got following message:

*   Trying 172.17.10.80...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to (nil) (172.17.18.84) port 8080 (#0)
* Server auth using Basic with user 'sftptest'
> GET sftp://sftptest:password@127.0.0.1/home/sftptest HTTP/1.1
> Host: 127.0.0.1:22
> Authorization: Basic c2Z0cHRlc3Q6MXEydzNlNHI=
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.52.1 OpenSSL/1.1.0e libssh2/1.7.0_DEV
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 400 Bad Request
< Server: nps/2.3.1
< Mime-Version: 1.0
< Date: Wed, 25 Oct 2017 10:03:23 GMT
< Content-Type: text/html
< Content-Length: 4205
< X-Squid-Error: ERR_INVALID_URL 0
< Vary: Accept-Language
< Content-Language: en
< X-Cache: MISS from netentsec-nps-172.17.18.84
< Connection: close
< 
{ [data not shown]
* Curl_http_done: called premature == 0

100  4205  100  4205    0     0  29450      0 --:--:-- --:--:-- --:--:-- 29822
* Closing connection 0

So curl first connected to my proxy, then use http request to connect sftp://xxx. I'd expect the request protocol never been changed even behind a proxy.

If I bypass 127.0.0.1, it simply works. So I'm pretty much sure that curl will change original sftp request to http request after connected to a proxy, which sounds like a bug.

@bagder

This comment has been minimized.

Copy link
Member

bagder commented Oct 26, 2017

User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.52.1

That's a "fun" combination of a very old command line tool using a more recent libcurl. Might lead to unexpected support and non-support of some features.

What you see is the (old) documented behavior of you asking to use a HTTP proxy so curl converts the request to HTTP-like and tries that. You can tunnel through the HTTP proxy with -p, --proxytunnel to make curl do what I believe you want. Just be aware that very few HTTP proxies are setup to allow tunneling to port 22.

Since 7.55.0, curl will enable tunneling automatically when you try to use SFTP over an HTTP proxy.

@mazong1123

This comment has been minimized.

Copy link

mazong1123 commented Oct 26, 2017

Even if rebuild culr from current master branch code on Ubuntu 16.04 (a newer machine) I still seeing it is using HTTP other than SFTP if the proxy is enabled:

* STATE: INIT => CONNECT handle 0x1a42ec8; line 1425 (connection #-5000)
* Added connection 0. The cache now contains 1 members
*   Trying 172.17.18.84...
* TCP_NODELAY set
* STATE: CONNECT => WAITCONNECT handle 0x1a42ec8; line 1477 (connection #0)
* Connected to 172.17.18.84 (172.17.18.84) port 8080 (#0)
* STATE: WAITCONNECT => WAITPROXYCONNECT handle 0x1a42ec8; line 1594 (connection #0)
* Marked for [keep alive]: HTTP default
* allocate connect buffer!
* Establish HTTP proxy tunnel to 127.0.0.1:22
* Server auth using Basic with user 'sftptest'
> CONNECT 127.0.0.1:22 HTTP/1.1
> Host: 127.0.0.1:22
> User-Agent: curl/7.57.0-DEV
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 503 Service Unavailable
< Server: nps/2.3.1
< Mime-Version: 1.0
< Date: Thu, 26 Oct 2017 07:09:25 GMT
< Content-Type: text/html
< Content-Length: 4069
< X-Squid-Error: ERR_CONNECT_FAIL 111
< Vary: Accept-Language
< Content-Language: en

A correct SFTP connection can be established if 127.0.0.1 is bypassed:

* STATE: INIT => CONNECT handle 0x14b6ec8; line 1425 (connection #-5000)
* Added connection 0. The cache now contains 1 members
*   Trying 127.0.0.1...
* TCP_NODELAY set
* STATE: CONNECT => WAITCONNECT handle 0x14b6ec8; line 1477 (connection #0)
* Connected to 127.0.0.1 (127.0.0.1) port 22 (#0)
* STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x14b6ec8; line 1594 (connection #0)
* Marked for [keep alive]: SSH default
* SFTP 0x14bd180 state change from SSH_STOP to SSH_INIT
* SFTP 0x14bd180 state change from SSH_INIT to SSH_S_STARTUP
* STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x14b6ec8; line 1608 (connection #0)
* SFTP 0x14bd180 state change from SSH_S_STARTUP to SSH_HOSTKEY
* SSH MD5 fingerprint: 4cf03683e054a3398c91d76a16715e6b
* SSH host check: 2, key: <none>
* SFTP 0x14bd180 state change from SSH_HOSTKEY to SSH_SESSION_FREE
* Marked for [closure]: SSH session free
* SFTP 0x14bd180 state change from SSH_SESSION_FREE to SSH_STOP
* multi_done
* SSH DISCONNECT starts now
* SSH DISCONNECT is done
* Closing connection 0
* The cache now contains 0 members
* Expire cleared
curl: (51) SSL peer certificate or SSH remote key was not OK
@mazong1123

This comment has been minimized.

Copy link

mazong1123 commented Oct 26, 2017

I looked at the stacktrace and find these two options lead to two different Curl_handler:

With Proxy:

#0  Curl_conncontrol (conn=0x727b48, ctrl=0, reason=0x4c45c3 "HTTP default") at connect.c:1392
#1  0x000000000044804e in Curl_http_connect (conn=0x727b48, done=0x7fffffffd298) at http.c:1343
#2  0x000000000045b29d in Curl_protocol_connect (conn=0x727b48, protocol_done=0x7fffffffd298) at url.c:4232
#3  0x000000000042fcbd in multi_runsingle (multi=0x727348, now=..., data=0x721ec8) at multi.c:1605
#4  0x0000000000430f41 in curl_multi_perform (multi=0x727348, running_handles=0x7fffffffd450) at multi.c:2163
#5  0x0000000000428d80 in easy_transfer (multi=0x727348) at easy.c:708
#6  0x0000000000428fbf in easy_perform (data=0x721ec8, events=false) at easy.c:794
#7  0x0000000000429026 in curl_easy_perform (data=0x721ec8) at easy.c:813
#8  0x000000000041be47 in operate_do (global=0x7fffffffda30, config=0x703018) at tool_operate.c:1562
#9  0x000000000041d1d1 in operate (config=0x7fffffffda30, argc=5, argv=0x7fffffffdb88) at tool_operate.c:2066
#10 0x0000000000415ca5 in main (argc=5, argv=0x7fffffffdb88) at tool_main.c:261

Without proxy:

#0  Curl_conncontrol (conn=0x727b58, ctrl=0, reason=0x4c9228 "SSH default") at connect.c:1392
#1  0x0000000000479744 in ssh_connect (conn=0x727b58, done=0x7fffffffd238) at ssh.c:2915
#2  0x000000000045b29d in Curl_protocol_connect (conn=0x727b58, protocol_done=0x7fffffffd238) at url.c:4232
#3  0x000000000042fcbd in multi_runsingle (multi=0x727358, now=..., data=0x721ec8) at multi.c:1605
#4  0x0000000000430f41 in curl_multi_perform (multi=0x727358, running_handles=0x7fffffffd3f0) at multi.c:2163
#5  0x0000000000428d80 in easy_transfer (multi=0x727358) at easy.c:708
#6  0x0000000000428fbf in easy_perform (data=0x721ec8, events=false) at easy.c:794
#7  0x0000000000429026 in curl_easy_perform (data=0x721ec8) at easy.c:813
#8  0x000000000041be47 in operate_do (global=0x7fffffffd9d0, config=0x703018) at tool_operate.c:1562
#9  0x000000000041d1d1 in operate (config=0x7fffffffd9d0, argc=3, argv=0x7fffffffdb28) at tool_operate.c:2066
#10 0x0000000000415ca5 in main (argc=3, argv=0x7fffffffdb28) at tool_main.c:261

Sepcifically, conn->handler is different at this line

@bagder

This comment has been minimized.

Copy link
Member

bagder commented Oct 26, 2017

< HTTP/1.1 503 Service Unavailable

That's the proxy which doesn't allow you to tunnel through. 503 does seem to imply that it doesn't like CONNECT at all. This is a problem with your proxy, not with curl...

@mazong1123

This comment has been minimized.

Copy link

mazong1123 commented Oct 26, 2017

@bagder hmm... missed that :( . I'll test it tomorrow. Thanks for your help :)

@mazong1123

This comment has been minimized.

Copy link

mazong1123 commented Oct 27, 2017

Thanks, I just tried to change 127.0.0.1 to the ethernet ip address and it works.

* STATE: INIT => CONNECT handle 0x2238ec8; line 1425 (connection #-5000)
* Rebuilt URL to: sftp://sftptest@172.26.9.94/
* Added connection 0. The cache now contains 1 members
*   Trying 172.17.18.84...
* TCP_NODELAY set
* STATE: CONNECT => WAITCONNECT handle 0x2238ec8; line 1477 (connection #0)
* Connected to 172.17.18.84 (172.17.18.84) port 8080 (#0)
* STATE: WAITCONNECT => WAITPROXYCONNECT handle 0x2238ec8; line 1594 (connection #0)
* Marked for [keep alive]: HTTP default
* allocate connect buffer!
* Establish HTTP proxy tunnel to 172.26.9.94:22
* Server auth using Basic with user 'sftptest'
> CONNECT 172.26.9.94:22 HTTP/1.1
> Host: 172.26.9.94:22
> User-Agent: curl/7.57.0-DEV
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* STATE: WAITPROXYCONNECT => SENDPROTOCONNECT handle 0x2238ec8; line 1573 (connection #0)
* CONNECT phase completed!
* SFTP 0x223f170 state change from SSH_STOP to SSH_INIT
* SFTP 0x223f170 state change from SSH_INIT to SSH_S_STARTUP
* STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x2238ec8; line 1608 (connection #0)
* SFTP 0x223f170 state change from SSH_S_STARTUP to SSH_HOSTKEY
* SSH MD5 fingerprint: 4cf03683e054a3398c91d76a16715e6b
* SSH host check: 2, key: <none>
* SFTP 0x223f170 state change from SSH_HOSTKEY to SSH_SESSION_FREE
* Marked for [closure]: SSH session free
* SFTP 0x223f170 state change from SSH_SESSION_FREE to SSH_STOP
* multi_done
* SSH DISCONNECT starts now
* SSH DISCONNECT is done
* Closing connection 0
* The cache now contains 0 members

@mazong1123 mazong1123 closed this Oct 27, 2017

@lock lock bot locked as resolved and limited conversation to collaborators May 6, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.