[webapp remote-connection] - webapp:BrokenPipe error when trying to create TCP tunnel for SSH

[Primital]

Describe the bug
After starting a TCP tunnel and trying to ssh to localhost on the specified port, the az tunnel crashes immediately after a BrokenPipeError: [Errno 32] Broken pipe

To reproduce
I don't know the cause. On my system it crashes everytime. Please read my comments in additional context

Expected behavior
I expect the ssh connection to establish

Environment summary
Python3.7.2_2 installed via brew. az-cli installed via brew. Pyenv used for other projects. Installed versions using pyenv are 2.7.15 and 3.6.5

Additional context
I have tried reinstalling everything through brew. I tried brew doctor. I tried brew link python. I tried to reinstall python and then brew link python again. I tried to link brews python versions to pyenv paths. I exhausted all possible search terms on google.

Verbose output from crash

az webapp remote-connection create -g MYRESRCGROUP -n MYAPP -p 1337 --verbose
Port 1337 is open
Creating a socket on port: 1337
Setting socket options
Binding to socket on local address and port
Finished initialization
Status response message: FAILURE:2222:Unable to connect to WebApp
WARNING - Remote debugging may not be setup properly. Reponse content: FAILURE:2222:Unable to connect to WebApp
SSH is available { username: root, password: Docker! }
Start your favorite client and connect to port 1337

In another window i used ssh root@127.0.0.1 -p 1337

Websocket tracing enabled
--- request header ---
GET /AppServiceTunnel/Tunnel.ashx HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: MYDOMAIN.scm.azurewebsites.net
Origin: http://MYDOMAIN.scm.azurewebsites.net
Sec-WebSocket-Key: RQM/U3LlGAC45h3cBVfZ+g==
Sec-WebSocket-Version: 13
Authorization: Basic JGtvbW11bnBvcnRhbDpjN3hXaFBuckxxQnVkMktZTWpQRG9ObzhuelFpOGRTcjdscTVna1FaZ0tMclNjNHJ6Sm5xMGMwZGI0QU0=


-----------------------
--- response header ---
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Server: Kestrel
Sec-WebSocket-Accept: Hb1GVBHYowi3+S7XGOYT4aj54fo=
Set-Cookie: ARRAffinity=bf607f7282cfa28719bacd3fbabae13c629c7e590d103982a12542d52319894c;Path=/;HttpOnly;Domain=MYDOMAIN.scm.azurewebsites.net
Connection: Upgrade
Access-Control-Allow-Origin: http://MYDOMAIN.scm.azurewebsites.net
Access-Control-Expose-Headers: ETag,Content-Length,Content-Encoding,Date
Date: Wed, 20 Mar 2019 13:45:24 GMT
-----------------------
Websocket, connected status: True
Got debugger connection... index: 1
Waiting for debugger data, index: 1
Received debugger data, nbytes: 21, index: 1
Sending to websocket, response data: bytearray(b'SSH-2.0-OpenSSH_7.9\r\n'), index: 1
send: b'\x82\x95\x15D\xef\xe7F\x17\xa7\xca\'j\xdf\xcaZ4\x8a\x89F\x17\xa7\xb8"j\xd6\xea\x1f'
Waiting for websocket data, connection status: True, index: 1
Both debugger and websocket threads started...
Successfully connected to local server..
Received frame: fin=1 opcode=8 data=b'\x03\xf3Connection failure. Connection refused 172.19.0.3:2222'
send: b'\x88\x82[\xba\xcf\x9cXR'
Traceback (most recent call last):
  File "/Users/myname/.azure/cliextensions/webapp/azext_webapp/tunnel.py", line 146, in _listen_to_web_socket
    data = ws_socket.recv()
  File "/Users/myname/.azure/cliextensions/webapp/azext_webapp/tunnel.py", line 32, in recv
    data = super(TunnelWebSocket, self).recv()
  File "/usr/local/Cellar/azure-cli/2.0.60/libexec/lib/python3.7/site-packages/websocket/_core.py", line 313, in recv
    opcode, data = self.recv_data()
  File "/usr/local/Cellar/azure-cli/2.0.60/libexec/lib/python3.7/site-packages/websocket/_core.py", line 330, in recv_data
    opcode, frame = self.recv_data_frame(control_frame)
  File "/usr/local/Cellar/azure-cli/2.0.60/libexec/lib/python3.7/site-packages/websocket/_core.py", line 357, in recv_data_frame
    self.send_close()
  File "/usr/local/Cellar/azure-cli/2.0.60/libexec/lib/python3.7/site-packages/websocket/_core.py", line 390, in send_close
    self.send(struct.pack('!H', status) + reason, ABNF.OPCODE_CLOSE)
  File "/usr/local/Cellar/azure-cli/2.0.60/libexec/lib/python3.7/site-packages/websocket/_core.py", line 253, in send
    return self.send_frame(frame)
  File "/usr/local/Cellar/azure-cli/2.0.60/libexec/lib/python3.7/site-packages/websocket/_core.py", line 278, in send_frame
    l = self._send(data)
  File "/usr/local/Cellar/azure-cli/2.0.60/libexec/lib/python3.7/site-packages/websocket/_core.py", line 448, in _send
    return send(self.sock, data)
  File "/usr/local/Cellar/azure-cli/2.0.60/libexec/lib/python3.7/site-packages/websocket/_socket.py", line 151, in send
    return _send()
  File "/usr/local/Cellar/azure-cli/2.0.60/libexec/lib/python3.7/site-packages/websocket/_socket.py", line 136, in _send
    return sock.send(data)
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 984, in send
    return self._sslobj.write(data)
ssl.SSLError: [SSL: BAD_LENGTH] bad length (_ssl.c:2337)
Traceback (most recent call last):
  File "/Users/myname/.azure/cliextensions/webapp/azext_webapp/tunnel.py", line 175, in _listen_to_client
    ws_socket.send_binary(responseData)
  File "/usr/local/Cellar/azure-cli/2.0.60/libexec/lib/python3.7/site-packages/websocket/_core.py", line 284, in send_binary
    return self.send(payload, ABNF.OPCODE_BINARY)
  File "/usr/local/Cellar/azure-cli/2.0.60/libexec/lib/python3.7/site-packages/websocket/_core.py", line 253, in send
    return self.send_frame(frame)
  File "/usr/local/Cellar/azure-cli/2.0.60/libexec/lib/python3.7/site-packages/websocket/_core.py", line 278, in send_frame
    l = self._send(data)
  File "/usr/local/Cellar/azure-cli/2.0.60/libexec/lib/python3.7/site-packages/websocket/_core.py", line 448, in _send
    return send(self.sock, data)
  File "/usr/local/Cellar/azure-cli/2.0.60/libexec/lib/python3.7/site-packages/websocket/_socket.py", line 151, in send
    return _send()
  File "/usr/local/Cellar/azure-cli/2.0.60/libexec/lib/python3.7/site-packages/websocket/_socket.py", line 136, in _send
    return sock.send(data)
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 984, in send
    return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe
Both debugger and websocket threads stopped...
Stopped local server..

[patricklee2]

@Primital can you open https://${app-name}.scm.azurewebsites.net/AppServiceTunnel/Tunnel.ashx?GetStatus in a browser?
what is the name of the app?, can you confirm that the webapp and scm sites are running?
what is the result of az webapp ssh?

[panchagnula]

Closing this due to inactivity.

[ghost]

Hi. Sorry to revive an old issue, but I'm having this same problem.

When I go to <webapp>.scm.azurewebsites.net/AppServiceTunnel/Tunnel.ashx?GetStatus I see the following:

FAILURE:2222:Unable to connect to WebApp

Response when I run az webapp create-remote-connection --subscription <...> --resource-group <...> -n <...> -p <...> --verbose :

Status response message: FAILURE:2222:Unable to connect to WebApp
WARNING - Remote debugging may not be setup properly. Reponse content: FAILURE:2222:Unable to connect
to WebApp

And the response from az webapp ssh:

This command is in preview. It may be changed/removed in a future release.
Connection is not ready yet, please wait
.
.
...
.
Timeout Error, Unable to establish a connection

[gguerrero]

Same issue here:

az webapp ssh --name <myApp> --resource-group <myResourceGroup> --verbose
This command is in preview. It may be changed/removed in a future release.
No port defined, creating on random free port
Creating a socket on port: 0
Setting socket options
Binding to socket on local address and port
Auto-selecting port: 50693
Finished initialization
Status response message: FAILURE:2222:Unable to connect to WebApp
WARNING - Remote debugging may not be setup properly. Reponse content: FAILURE:2222:Unable to connect to WebApp
Connection is not ready yet, please wait
.
.
.
Timeout Error, Unable to establish a connection

Connecting properly to the webapp as I can do any az webapp show or any other command from my shell.
Is there any missing configuration in the webapp side?

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @AzureAppServiceCLI, @antcp.

[coderintherye]

In case it is helpful I get the same error messaging:
Creating a socket on port: 2222
Setting socket options
Binding to socket on local address and port
Finished initialization
Status response message: FAILURE:2222:Unable to connect to WebApp

after an app instance goes unhealthy, restarting the app instance does not resolve the problem, but going to webssh via the browser does, after that the remote connection works fine. The browser access seems to indicate it is creating a new Docker container...perhaps that somehow resolves the issue.

[terrynguyen255]

I'm experiencing same issue:

++Rcv raw: b'\x88\x16\x03\xf3Connection failure. '
++Rcv decoded: fin=1 opcode=8 data=b'\x03\xf3Connection failure. '
++Sent raw: b'\x88\x82\x88`NN\x8b\x88'
++Sent decoded: fin=1 opcode=8 data=b'\x03\xe8'
Received websocket data: 
Received websocket data: , index: 1
Client disconnected!, index: 1
Both debugger and websocket threads stopped...
Stopped local server..
Command ran in 16.841 seconds (init: 0.113, invoke: 16.728)

My status page shows SUCCESS:2222
https://APP_NAME.scm.azurewebsites.net/AppServiceTunnel/Tunnel.ashx?GetStatus