Skip to content
This repository

Low socket limit #104

Closed
stakach opened this Issue March 06, 2012 · 49 comments

7 participants

Stephen von Takach Luis Lavena Roger Pack Jon Justin Baker Hiroshi Shirosaki cubiic
Stephen von Takach

Windows defines a default of 64 asynchronous sockets per application as defined by FD_SETSIZE
See: http://support.microsoft.com/kb/111855

This limits windows as a platform for running large scale applications such as eventmachine.

Could we please set FD_SETSIZE to a considerably large limit in the makefile, say 99999, to allow for larger scale applications. Currently it is quite useless.

Luis Lavena
Owner

Can you be more specific? Which version of Ruby are you talking about? For 1.8.x we used to increase FD_SIZE to 256. Ruby 1.9 does not suffer from that.

https://github.com/oneclick/rubyinstaller/blob/master/config/ruby_installer.rb#L18-31

If you found that Ruby 1.9.2 or 1.9.3 suffered from that, perhaps your report is better suited to Ruby itself:

http://bugs.ruby-lang.org

Please confirm.

Stephen von Takach

Ruby 1.9.3 suffers from it, only for asynchronous IO and in any libraries that use functions Ruby core 1.9 now avoids such as select(2).

So the FD_SIZE should still be set and much higher than 256 for any serious async application.

Stephen von Takach

Basically libraries such as eventmachine require it to be set otherwise they are limited to 64 sockets which is pretty meaningless and considering I've been doing production deployments using EM and thin on windows it is pretty scary.

Roger Pack
Collaborator
rdp commented March 06, 2012

for 1.8.x it's set to 256 isn't it?

Jon
Owner

@stakach:

1) any specific examples of "larger scale applications" on Windows other than your thin/EM deploys?

2) your "larger scale applications" don't require 64bit and aren't limited by our current 32bit build?

A bit off-topic, but I'm curious on if/how the following relate to what you're doing, and what you think of Greg's patch from 9 months ago?

http://bugs.ruby-lang.org/issues/4906

http://www.ruby-forum.com/topic/869239

Stephen von Takach

Yes, however 256 is way too low.

I've been hacking at Eventmachine for the last 2 days as I have application that is using ~150 sockets now and is intended to scale well beyond 1000 before the end of the year.

Right now it is 3am here and I'm configuring a Linux VM to NAT with the windows host so I don't hit the 64 limit. I would prefer not to be doing this again in a couple of months after I've reached the 256 limit...

There are 16383 ephemeral ports on windows (ie the probable max an application will ever use) can we use that?

Stephen von Takach

@jonforums
http://bugs.ruby-lang.org/issues/4906 === eventmachine/eventmachine@5f1b8d7

The 64 is the number of sockets, not 64bits.. I wish it was bits.

Stephen von Takach

Of course I'm not entirely sure of ruby internals, if there is a windows file descriptor wrapper for every socket (not just a ruby facade) then the limiting number of sockets would be Windows limit of 2048 file descriptors: http://msdn.microsoft.com/en-us/library/6e3b887c(v=vs.71).aspx

At least that limit is somewhat configurable at run time.

Luis Lavena
Owner

@stakach I can't find the issue reported by @rdp on Ruby-Core about FD_SETSIZE

I see the value of the request, however, please see this on MSDN:

http://msdn.microsoft.com/en-us/library/windows/desktop/ms739169(v=vs.85).aspx

The maximum number of sockets that a Windows Sockets application can use is not affected by the manifest constant FD_SETSIZE.

But, further reading:

This value defined in the Winsock2.h header file is used in constructing the FD_SET structures used with select function. The default value in Winsock2.h is 64. If an application is designed to be capable of working with more than 64 sockets using the select and WSAPoll functions

We can up that numbre to 32767, which seems to be the maximum supported by iMaxSockets

If you agree, I'll change 1.9 recipe so next build will handle that.

Jon
Owner

@stakach hehe..yeh, I know the 64 number is the number of sockets.

I asked about 64bits because some say they need 64bit (more than 4GB, etc) for "larger scale apps" as 32bit won't do. Seems like 32bit is fine for what you're doing, but I was interested in more details.

Stephen von Takach

@jonforums the larger scale applications we build are building automation systems: https://github.com/stakach/em_control
We're currently deploying our first big rollout (disastrous as we'd never scaled higher than 64 on windows) and we've got many more coming up. In fact this first 150 devices is only a small part of the rollout.

We also make a digital signage system that is entirely HTML5 (management + playback). Local deployments of that usually run on windows as we have to do powerpoint conversions and offloading to windows is more infrastructure + cost people don't want to see. Some of those deployments will be nearing 64 displays very soon.

Stephen von Takach

@luislavena 32767 sounds great!

Jon
Owner

@stakach how very cool...cloning your repo...OT, could https://github.com/adoxa/ansicon be helpful for your colorization needs?

Stephen von Takach

@jonforums you looking at my telnet hack? (I stopped using that)
Anyway it's not quite ready for public release, I want to package it up somewhat like rails and have a project generator.

Otherwise everyone will have to clone the repo :)

Jon
Owner

you looking at my telnet hack? (I stopped using that)

yes

...I want to package it up somewhat like rails and have a project generator.

just scanned your repo, but would a single distributable .exe containing your Ruby stuff be a deployment option for you? FWIW, Luis has adopted a clone of exerb project https://github.com/luislavena/exerb-mingw and he and another guy have made some updates for 1.9.3. I haven't had time to play with the mods yet, but exerb/1.8.7 has worked well for https://github.com/vertiginous/pik

Stephen von Takach

It could be and probably something we'd consider when we get this running in the cloud, for peoples local deployments.
I'll look into it further, both look like interesting projects!

Jon
Owner

Good luck...no more from me as it's now OT to your issue and Luis has accepted the mod for the next release :)

Please feel free to drop an ANN if/when it makes sense to http://groups.google.com/group/rubyinstaller since you're using RubyInstaller. And I'd also drop an ANN on http://www.ruby-forum.com/forum/ruby since many Rubyist's don't realize all the cool things being done with Ruby on Windows!

Stephen von Takach

Thanks mate, will do!

Roger Pack
Collaborator
rdp commented March 06, 2012

isn't the max number limited by msvcrt.dll? if so what's the max actually usable maybe we should use that number (of course, that number might vary based on whether you're on a server or standard windows box...)

Stephen von Takach

That's just the c run time. Technically socket limits are purely memory based, except for this particular case where we want to use this particular function for async io.

Windows imposes limits on IIS connections and network shares however they are all higher level.

Roger Pack
Collaborator
rdp commented March 06, 2012
Stephen von Takach

That limit does exist. I mention it in my 6th comment.
It relates to open files not sockets.

Roger Pack
Collaborator
rdp commented March 06, 2012

Interesting. (oh BTW ruby core in 1.9 uses select(2) if you call IO.select doesn't it?) Anyway, with Ruby sockets per se (like require 'socket'; TCPSocket.new) I think it creates "a windows file descriptor wrapper for every socket". I don't know if EM does or not. It'd be interesting to see what the theoretical limit "actually is" with EM or the like, given a high FD_SETSIZE. The only drawback to a large FD_SETSIZE is that the whole set is put on the stack, but since 1.9 has individual threads anyway now, it probably wouldn't matter if it were large I guess....

Stephen von Takach

Yeah it would be good to know, if I start getting near the 2048 boundary I'd be playing it cautious.
Also if it is file descriptors you have to manually increase the limit from 512 see stakach/eventmachine@7d55c9f

Luis Lavena luislavena closed this in 8b3469d March 06, 2012
Luis Lavena
Owner

@stakach I've committed a fix, and will be available in the next patchlevel release of Ruby 1.9.3.

In the meantime, if you want to test, you can build locally by cloning RubyInstaller repository and doing rake ruby19 to compile.

Really cool the em_control project. Will be interesting when you're done if you can do a formal announcement in RubyInstaller mailing list:

http://groups.google.com/group/rubyinstaller/

Cheers.

Stephen von Takach

libffi didn't compile properly. Giving it another go.
Awesome how everything is configured.

Stephen von Takach

It was a path with a space. All good now

Stephen von Takach

Ok the FD_SETSIZE size we set clashed with event machine (which defines a smaller number) causing a seg fault.
Reducing FD_SETSIZE to equal eventmachines 1024 worked.

So the next release of eventmachine and ruby installer must always match!

Previously eventmachine was higher than ruby and it created so much weirdness! I'm creating an issue to let the eventmachine guys know.

Stephen von Takach

Also, thanks everyone for your help! Couldn't have got this working without you!

Luis Lavena
Owner

@stakach please link the issue of EventMachine with this using oneclick/rubyinstaller#104 so we are aware.

Cheers.

Stephen von Takach

Ok, the eventmachine one is: eventmachine/eventmachine#303

Stephen von Takach stakach referenced this issue in eventmachine/eventmachine March 06, 2012
Open

Added windows IPv6 support #303

Stephen von Takach stakach referenced this issue in rubinius/rubinius March 07, 2012
Closed

Eventmachine Support on Windows #1600

Stephen von Takach

Hi @luislavena looks like Windows will never support more than 2048 (technically 2045 thanks to STD streams) and the limits on file handles.

So we might as well drop the FD_SETSIZE before your next release to 2048 to save memory - the change won't effect eventmachine

Luis Lavena
Owner
Stephen von Takach

It is a Microsoft limit on the standard c library: http://msdn.microsoft.com/en-us/library/6e3b887c%28vs.71%29.aspx
and a windows file descriptor is allocated for every socket: See the comment by @rdp

It's easy to test:

Server

require 'rubygems'
require 'eventmachine'

module EchoServer

    @@connected_clients = 0

def post_init
    @@connected_clients += 1
    p "there are now #{@@connected_clients} connected"
end
  def receive_data data
    send_data ">>>you sent: #{data}"
    close_connection if data =~ /quit/i
  end
end

EventMachine::run {
  EventMachine::start_server "127.0.0.1", 8081, EchoServer
}

Client

require 'rubygems'
require 'eventmachine'

class Echo < EventMachine::Connection
  def post_init
    @timer = EventMachine::PeriodicTimer.new(5) do
  send_data 'Hello'
end
 send_data 'Hello'
  end

  def receive_data(data)
    p 'recieved:' + data
  end

def unbind
    p ' connection totally closed'
    @timer.cancel
  end
end

EventMachine.run {
    (1..3000).each do |h|
        EventMachine.connect '127.0.0.1', 8081, Echo
    end
}

Luis Lavena
Owner
Luis Lavena
Owner

@stakach I haven't tested this example yet, but I don't think Ruby is using open methods on the sockets mapped to fd's, or it does?

Stephen von Takach

I have a feeling it is the file descriptor limit being hit.
Other wise someone would have had to have set WSAStartup's max sockets at 2045

Justin Baker
Collaborator

Running the example I get this error on client.rb at around 1000(+/-50) connections.

unable to create new socket: Too many open files (EventMachine::ConnectionError)

Here is the BasicSocket initialization

It looks like rb_update_max_fd is being called, but I have no idea what that actually does so it's just a hunch.

Hiroshi Shirosaki
Collaborator

Ruby associates windows socket to a C run-time file descriptor by _open_osfhandle().

https://github.com/ruby/ruby/blob/trunk/win32/win32.c#L3486
http://msdn.microsoft.com/en-us/library/bdts1c9x(v=vs.71).aspx

So that socket size seems limited to file descriptor max (2048).
If ruby did not use _open_osfhandle() for socket, socket size might not limited to 2048.

Luis Lavena
Owner

Thank you @shirosaki

So we conclude that having FD_SETSIZE of 16K is pointless.

Sometimes I wish Ruby did IO/socket implementation differently.

Anyhow, going to change the limit, again.

Thanks @stakach

Luis Lavena luislavena reopened this August 26, 2012
Luis Lavena luislavena closed this issue from a commit August 26, 2012
Luis Lavena Decreased file descriptors for 1.9.x to 2K
Ruby associates a file descriptor to each socket, so can't use more than
2048 even if we force FD_SETSIZE to a bigger value.

This closes #104 (again)
c5ed8e3
Luis Lavena luislavena closed this in c5ed8e3 August 26, 2012
cubiic

Hi Everyone, and sorry for the headache here, but I'm facing almost the same problem is my production server, I would really appreciate any help regarding this issue, I'm running my system with Ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux] , at OS Ubuntu 12.04.1 LTS and em-http-request (1.0.3).

I have a ruby processes that consume msgs from a RabbitMQ queue using amqp gem, something like this :

require "bundler/setup"
require "amqp"
require 'eventmachine'
require 'em-http'

AMQP.start(:host => $AMQP_URL) do |connection|
  @channel ||= AMQP::Channel.new(connection)
  @queue   ||= @channel.queue("results")

  puts " [*] Waiting for messages. "

  @queue.subscribe do |body|    
    http = EventMachine::HttpRequest.new(URL).post :body => body          
    http.callback { # do something   }        
    http.errback  { $LOG.error "[errorback] -> #{http.error}" }
  end
end

Now the URL is slow, and the queue has so much messages ( > 30K ), I got this error in the log:
[errorback] -> unable to create new socket: Too many open files

Any help would be highly appreciated since I've been trying all my time figuring out how to solve it but with no results at all.

Thanks in advance

Hiroshi Shirosaki
Collaborator

@cubiic have you tried to increase the open file descriptor maximum?

such as:
$ ulimit -n 100000
http://serverfault.com/questions/235356/open-file-descriptor-limits-conf-setting-isnt-read-by-ulimit-even-when-pam-limi

cubiic

@shirosaki I really appreciate your welling to help, but guide me more here since I'm not an expert with this, please check out what I've been trying to do :

https://gist.github.com/cubiic/f7c8b1ca3f56ac9fd73b

As you can see I've been trying to do that unsuccessfully,

Thanks in advance

Jon
Owner

@cubiic while interesting, this linux issue has nothing to do with rubyinstaller or ruby on windows.

Worse, the documentation for any fix will be buried in this thread and very difficult to for a linux user to find. That's a shame, and a waste of your and Shirosaki-san's efforts. The EM list, or the ruby-talk list (or even stackoverflow) are much better places to work/document the problem.

cubiic

Sorry man , but I though the problem is relevant based on the return error i'm getting, if you don't feel so I can just close it, but I found "unable to create new socket: Too many open files" error relevant by googling it

Jon
Owner

It's related, but irrelevant on this project's issue list.

But don't miss my main point: I don't want the doco for a fix getting "buried" here. Switch the conversation (with an fyi link back here if you like) to a more linux visible place, like ruby-talk so we linux users can better help/benefit.

Hiroshi Shirosaki
Collaborator

@cubiic $ ulimit -S -n 100000 would work for non-root user.
http://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user

If more guide is needed, please switch the place as Jon stated.

cubiic

Thanks a lot for the help here,

Alberto Ornaghi alor referenced this issue in eventmachine/eventmachine February 27, 2013
Closed

ruby 1.9.3-p194 on windows lead to segfault #319

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.