Unable to reuse same port after stopping server

[jcarlson]

In my rspec suite, I am starting/stopping the server between examples and setting the port to a known value each time.

The first spec works, but subsequent specs fail, citing the following error:
An error occurred in an after hook
NoMethodError: undefined method close' for nil:NilClass occurred at /home/ubuntu/Corndog/vendor/bundle/ruby/2.0.0/gems/ftpd-0.11.0/lib/ftpd/server.rb:53:instop'
...
Errno::EADDRINUSE: Address already in use - bind(2)

For each spec, I am creating a new server instance in the before block and starting the server. I am stopping the server in the after block.

From putzing around in the rails console, it seems like this works on OSX but not on Ubuntu.

As far as I can tell, when asking the server to stop, the associated thread does not stop listening to the bound port.

I think I can work around the issue by saving a handle to the Thread returned by server.start and then killing that thread manually when I stop my server, but it seems like maybe this is an issue with Ftpd?

[wconrad]

Jarrod, Thank you for the report, and for using ftpd. Your analysis sounds entirely plausible. Ftpd's own tests let each instance of the server pick a port at random, and so would not notice if the thread was not shutting down the socket properly or in a timely fashion. That it works in one OS but not another makes me wonder if it is a race condition. I will have a look at it. With luck, I can reproduce the problem (I run Debian, a close relative to Ubuntu). I might not be able to get to this until next weekend--busy weekend, and busy week ahead.

Another workaround you might try for now is to let the server pick the port, if you have that flexibility. You can call #bound_port on the server to find out which port it bound to, and then pass that to the code under test via a configuration file, environment variable, command-line argument, object attribute, or via the Deep Space Network. I am not suggesting this as a permanent fix, of course--ftpd should be made to function properly, and DSN bandwidth is too hard to get.

[wconrad]

Also, if it is a race condition, it may be that a short sleep after stopping the server may help (as a workaround). It it does help, that would be evidence supporting the "race condition" hypothesis.

[wconrad]

I may have it figured out. Here's what I think I know.

Under MRI 1.8 (the Ruby under which ftpd was first written), closing the server socket was as good as a shutdown: It caused the server thread's call to TCPServerSocket#accept to raise an error, which the thread used as its signal to end. Under MRI >= 1.9.3, #close does not cause the server thread's call to TCPServerSocket#accep to raise an error. The #accept hangs, and the thread is never killed.

ftpd no longer supports 1.8.7, but it does make me feel better knowing that this did work properly once upon a time.

The race condition (at least, the one I found) is this: If the first server's thread has not gotten around to calling #accept when the second server starts, the second server will succeed. If, however, the first server's thread has gotten around to calling #accept, then the second server will fail with Errno::EADDRINUSE. Apparently, the address is not really in use until #accept is called.

Luckily, it seems as though a reliable test for this bug can be made (race conditions can be nasty to test). If the test monkey-patches the server socket to force the first server's #accept to happen before the second server is created, then the bug should show itself every time.

The fix will be to call #shutdown on the server socket before calling #close.

[jcarlson]

Thanks for the quick response! I'll upgrade my gem version and see if that fixes the issue. Glad you found a root cause so easily. That's always satisfying, no?

[wconrad]

@jcarlson This was the most satisfying bug I've fixed in a long time. It was difficult to reproduce and quite puzzling at first. However, in the end, I understood the bug, got it fixed, and learned something about sockets that I didn't know before. Good times!

Now, I just hope I fixed the actual bug that's bugging you :)

[jcarlson]

Unfortunately, this doesn't fix my problem. Now, on OS X, when I call server.stop, it raises an error immediately:

[6] pry(main)> server.stop
Errno::ENOTCONN: Socket is not connected
from /Users/dev/.rvm/gems/ruby-2.0.0-p353@corndog/gems/ftpd-0.14.0/lib/ftpd/server.rb:53:in `shutdown'

This is better because it fails fast, but obviously still an issue.

I'm on Ruby 2.0.0p353 on OS X 10.8.5. Our CI server is on Ubuntu.

FWIW, the reason I don't use a random port number is that my client code reads settings from a YAML file in the config directory. So when I setup my test server, I also read from that YAML file and have the server start on the port that the YAML file specifies. I guess I could hack around until I can make the port a sort of shared read/write global that the server can update when the port is known... but I don't particularly love that path.

Here's a couple test helper methods that seem to get the trick done with v0.11.0:

def start
  @thread = @server.start
  @running = true
end

def stop
  if @running
    @server.stop
    @thread.exit
  end
  @running = false
end