Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: reconnecting to Listen'ed and forwarded ports hangs connection #22269

Closed
zebrig opened this issue Oct 14, 2017 · 9 comments

Comments

Projects
None yet
4 participants
@zebrig
Copy link

commented Oct 14, 2017

What version of Go are you using (go version)?

go1.9.1 windows/amd64

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\User\go
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config

What did you do?

Trying to make simple port forwarding

  1. Make an ssh connection (Dial), set to Listen a port on the remote and forward (io.Copy) it to any service (Dial'ed to local vnc, or any other)
  2. connect to this port on the remote (or simply probe the open port by telnet) and close the connection
  3. try to connect again - the connection fails

sometimes on server side we have CLOSE_WAIT on this connection.
(if we do the same thing by putty or something, it comes to TIME_WAIT)

What did you expect to see?

Working port forward on second connect

What did you see instead?

One of three:

  1. all ssh sessions totally hung, cannot open new sessions
  2. existing sessions hung, can open new
  3. only this port foward hung, other things are working
    which is giong to happen seems to depend on traffic activity and direction on disconnect.

@bradfitz bradfitz changed the title golang.org\x\crypto\ssh reconnecting to Listen'ed ports hang connection x/crypto/ssh: reconnecting to Listen'ed ports hang connection Oct 14, 2017

@gopherbot gopherbot added this to the Unreleased milestone Oct 14, 2017

@alexbrainman

This comment has been minimized.

Copy link
Member

commented Oct 16, 2017

@zebrig you need to show your code. Just describing your code is not good enough. Your code might have some errors. Please show small program that demonstrates your problem.

Thank you

Alex

@zebrig

This comment has been minimized.

Copy link
Author

commented Oct 16, 2017

@alexbrainman Here is the sample code.

package main

import (
	"io"
	"log"
	"net"
	"time"

	"golang.org/x/crypto/ssh"
)

var (
	username         = "root"
	serverAddrString = "yourserver:22"
	knownHostKey     = "|1|xxx= ecdsa-sha2-nistp256 xxx="
	pkey             = []byte(`-----BEGIN RSA PRIVATE KEY-----
xxx==
-----END RSA PRIVATE KEY-----
`)
)

func main() {
//Connection settings
	signer, err := ssh.ParsePrivateKey(pkey)
	if err != nil {
		log.Fatalln("unable to parse private key: %v", err)
	}

	hostKey, _, _, _, err := ssh.ParseAuthorizedKey([]byte(knownHostKey))
	if err != nil {
		log.Fatalln("error parsing: %v", err)
	}

	config := &ssh.ClientConfig{
		User: username,
		Auth: []ssh.AuthMethod{
			ssh.PublicKeys(signer),
		},
		HostKeyCallback: ssh.FixedHostKey(hostKey),
	}

// Here we connect to a local port. 
//5900 is used for testing, because it has a welcome-banner. 445, 3389 work the same.
	localListener, err := net.Dial("tcp", "127.0.0.1:5900")
	if err != nil {
		log.Fatalln("net.Dial failed: %v", err)
	}
	defer localListener.Close()

//Connecting to server
	client, err := ssh.Dial("tcp", serverAddrString, config)

//Here we open a port to listen on the remote server's loopback interface. 
	sshListener, err := client.Listen("tcp", "127.0.0.1:12345")
	if err != nil {
		log.Fatalln("unable to register tcp forward: ", err)
	}
	defer sshListener.Close()

//Accepting connection on remote server port
	sshConn, err := sshListener.Accept()
	if err == nil {
		go func() {
// Portforwarding. Now I see it is wrapped by a goroutine, that is not really needed, but it doesn't change anything.
			go func() {
				_, err = io.Copy(localListener, sshConn)
				if err != nil {
					log.Fatalln("%T %v", err, err)
				}
			}()
			go func() {
				_, err = io.Copy(sshConn, localListener)
				if err != nil {
					log.Fatalln("%T %v", err, err)
				}
			}()
		}()
	} else {
		log.Fatalln("ssh.listen.Accept failed: %v", err)
	}
	for {
		time.Sleep(time.Second * 5)
	}
}

The code itself works.

To check the status of the port on the server I run this on the remote server:

#netstat -an | grep 12345
tcp        0      0 127.0.0.1:12345         0.0.0.0:*               LISTEN

Port is open, now I try to connect simply by telneting the port: (vnc was used, as it has a welcome-banner):

#telnet 127.0.0.1 12345
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
RFB 003.008
^]
telnet> quit
Connection closed.

#netstat -an | grep 12345
tcp        0      0 127.0.0.1:12345         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:12345         127.0.0.1:47682         CLOSE_WAIT
tcp        0      0 127.0.0.1:47682         127.0.0.1:12345         FIN_WAIT2

Trying to connect again: (port is open, but no welcome-banner)

#telnet 127.0.0.1 12345
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.

Or sometimes it does like this: (ssh connection either hung or breaks)

#telnet 127.0.0.1 12345
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

Telnet is used here for clarity of what is happening. Native client software behaves the same way, throwing out errors.

The golang program is run on the windows-machine, The above-mentioned commands are run on the ssh server

@zebrig

This comment has been minimized.

Copy link
Author

commented Oct 16, 2017

I have done some research.

The same behavior occurs if we do not open a port (sshListener, err := client.Listen("tcp", "127.0.0.1:12345")), but Listen to a unix socket.

When forwarding port with other ssh software (putty, plink), it works just fine.

If somebody would give me a hint, I will do a thorough testing and investigating in any needed environment to get more needed information. I already have Wireshark dumps of good connections (with putty) and bad connections (with the above mentioned code).

As far as I see, the problem is that the connection is not closed properly. The connection is left abandoned by the server too early. When the server is in CLOSE_WAIT state, the Listened port does not respond with the last needed FIN and in respond the client doesnt send the last ACK. So, the client hangs in FIN_WAIT_2 state and the server hangs in CLOSE_WAIT state.

As I see, the port is left half-open and that causes the ssh connection to hang. Because CLOSE_WAIT lasts until the death of ssh process, it prevents other connections to be served. While TIME_WAIT lasts for only 2 maximum segment lifetimes and is not blocking connections if the server uses SO_REUSEADDR (OpenSSH uses SO_REUSEADDR since 5.5 version as far as I remember, but not sure)

@fraenkel

This comment has been minimized.

Copy link
Contributor

commented Oct 16, 2017

@zebrig Your code above only allows a single connect via Accept. Did you expect it to continuously accept new connections? If so, you need to add a loop.

@zebrig

This comment has been minimized.

Copy link
Author

commented Oct 16, 2017

@fraenkel Thank you for a good idea. But unfortunately, still no luck, same behavior. I'm attaching a fixed code. The only difference is that the for-loop now wraps the Accept block to constantly accept new connections, as you have mentioned.

package main

import (
	"io"
	"log"
	"net"
	"time"

	"golang.org/x/crypto/ssh"
)

var (
	username         = "root"
	serverAddrString = "yourserver:22"
	knownHostKey     = "|1|xxx= ecdsa-sha2-nistp256 xxx="
	pkey             = []byte(`-----BEGIN RSA PRIVATE KEY-----
xxx==
-----END RSA PRIVATE KEY-----
`)
)func main() {
	//Connection settings
	signer, err := ssh.ParsePrivateKey(pkey)
	if err != nil {
		log.Fatalln("unable to parse private key: %v", err)
	}

	hostKey, _, _, _, err := ssh.ParseAuthorizedKey([]byte(knownHostKey))
	if err != nil {
		log.Fatalln("error parsing: %v", err)
	}

	config := &ssh.ClientConfig{
		User: username,
		Auth: []ssh.AuthMethod{
			ssh.PublicKeys(signer),
		},
		HostKeyCallback: ssh.FixedHostKey(hostKey),
	}

	// Here we connect to a local port.
	//5900 is used for testing, because it has a welcome-banner. 445, 3389 work the same.
	localListener, err := net.Dial("tcp", "127.0.0.1:5900")
	if err != nil {
		log.Fatalln("net.Dial failed: %v", err)
	}
	defer localListener.Close()

	//Connecting to server
	client, err := ssh.Dial("tcp", serverAddrString, config)

	//Here we open a port to listen on the remote server's loopback interface.
	sshListener, err := client.Listen("tcp", "127.0.0.1:12345")
	if err != nil {
		log.Fatalln("unable to register tcp forward: ", err)
	}
	defer sshListener.Close()

	//Accepting connection on remote server port
	for {
		sshConn, err := sshListener.Accept()
		fmt.Println("A new connection has arrived")
		if err == nil {
			go func() {
				// Portforwarding. Now I see it is wrapped by a goroutine, that is not really needed, but it doesn't change anything.
				go func() {
					_, err = io.Copy(localListener, sshConn)
					if err != nil {
						log.Fatalln("%T %v", err, err)
					}
				}()
				go func() {
					_, err = io.Copy(sshConn, localListener)
					if err != nil {
						log.Fatalln("%T %v", err, err)
					}
				}()
			}()
		} else {
			log.Fatalln("ssh.listen.Accept failed: %v", err)
		}
	}
}
@zebrig

This comment has been minimized.

Copy link
Author

commented Oct 17, 2017

I have also tried to create not a loop, but a sequence of Accept code blocks assigning sshListener.Accept() in every block to a new variable to avoid any possible problems of reusing variables in goroutines. So the code looked like this:

sshConn, err := sshListener.Accept()
.....
sshConn2, err := sshListener.Accept()
...
sshConn3, err := sshListener.Accept()

Thus, any new connection is assigned to a new variable.
But still it didn't work out. The problem stays the same.

@zebrig

This comment has been minimized.

Copy link
Author

commented Oct 17, 2017

I have made more investigation and found out that it is not a problem of ssh package. It is a problem of net.Listen!

I have modified the program not to make ssh connections, but to do port forwarding locally:

  1. First, it Dial's any open local service port.
  2. Then it starts to Listen port 12345.
  3. Then it makes io.Copy data between this two ports.

The first
telnet 127.0.0.1 12345
works, the second connects to the port, but no data is transmitted.
I will mention again, that telnet is used for clear testing. Native clients have the same errors in connecting.

I have compiled the same program for linux (Dialed 22 port - ssh), the behavior is the same.

package main

import (
	"io"
	"log"
	"net"
)

func main() {
	// Here we connect to a local port.
	//5900 is used for testing, because it has a welcome-banner. 445, 3389 work the same.
	localDialer, err := net.Dial("tcp", "127.0.0.1:5900")
	if err != nil {
		log.Fatalln("net.Dial failed: %v", err)
	}
	defer localDialer.Close()

	//Open a local port to Listen
	localListener, err := net.Listen("tcp", "127.0.0.1:12345")
	if err != nil {
		log.Fatalln("unable to register tcp forward: ", err)
	}
	defer localListener.Close()

	//Accepting connection on Listen'ed port to forward it to Dial'ed port
	for {
		localConn, err := localListener.Accept()

		if err == nil {
			go func() {
				_, err = io.Copy(localDialer, localConn)
				if err != nil {
					log.Fatalln("%T %v", err, err)
				}
			}()
			go func() {
				_, err = io.Copy(localConn, localDialer)
				if err != nil {
					log.Fatalln("%T %v", err, err)
				}
			}()

		} else {
			log.Fatalln("listen.Accept failed: %v", err)
		}
	}
}

@zebrig zebrig changed the title x/crypto/ssh: reconnecting to Listen'ed ports hang connection net: reconnecting to Listen'ed and forwarded ports hangs connection Oct 17, 2017

@fraenkel

This comment has been minimized.

Copy link
Contributor

commented Oct 17, 2017

@zebrig My quick glance says your localDialer doesn't work the way you think. You are reusing it across connections which is probably a bad idea.

You should be able to debug this situations but getting a goroutine dump and examining where all your routines are blocked. It is also a bit difficult when a self contained example is not provided.

@zebrig

This comment has been minimized.

Copy link
Author

commented Oct 17, 2017

@fraenkel Thank you very much for your comment.
It was a week of debugging, dumping and rewriting and a single post to save many more days or weeks of useless tries.

You are absolutely right. For each new Listen'ed net.Conn, there should be a new Dial'ed net.Conn.

I should have posted a question on stackoverflow or something before opening an issue.

I'm sorry for wasting your time and thank you very very much for your help.

@zebrig zebrig closed this Oct 17, 2017

@golang golang locked and limited conversation to collaborators Oct 17, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.