Can we get another beta? #333

Closed
luislavena opened this Issue May 24, 2012 · 54 comments

Comments

Projects
None yet
7 participants
Contributor

luislavena commented May 24, 2012

Hello guys,

Lot of folks are having problems with 1.0.0.beta.4, Ruby 1.9.3-p194 and the dreaded select() and FD_SETSIZE issue.

Also, any attempt to compile from source with gem install eventmachine --pre --platform=ruby results in errors.

I just tried to compile from master and everything worked.

Can we get another beta out so people can compile it if required?

Thank you! ❤️ ❤️ ❤️

@luislavena luislavena referenced this issue in oneclick/rubyinstaller May 24, 2012

Closed

Getting seg fault with 1.9.3p194 #116

Azolo commented May 24, 2012

Actually it's still broke, It's a Runtime error so it actually compiles fine. 👅

#319 still needs to resolved

Contributor

luislavena commented May 24, 2012

@Azolo problem is that beta.4 (the one on RubyGems) does not compile at all, with or without FD_SETSIZE 👅 👅

Azolo commented May 24, 2012

@luislavena touché, my mistake. But interestingly enough the mingw32 compiled gem is at beta.4.1.

PS C:\Users\Justin> gem install eventmachine --pre
Successfully installed eventmachine-1.0.0.beta.4.1-x86-mingw32
1 gem installed
Contributor

garbagecat commented May 24, 2012

I'll try to track down TMM and see if this can be done quickly. Thanks for reporting it, Luis.

On May 24, 2012, at 3:23 PM, Luis Lavena wrote:

Hello guys,

Lot of folks are having problems with 1.0.0.beta.4, Ruby 1.9.3-p194 and the dreaded select() and FD_SETSIZE issue.

Also, any attempt to compile from source with gem install eventmachine --pre --platform=ruby results in errors.

I just tried to compile from master and everything worked.

Can we get another beta out so people can compile it if required?

Thank you! ❤️ ❤️ ❤️


Reply to this email directly or view it on GitHub:
#333

algritz commented Jun 18, 2012

Any luck getting the new beta done?

Contributor

luislavena commented Jun 18, 2012

@tmm1 @raggi ping?

Contributor

garbagecat commented Jun 18, 2012

Yeah, that's the thing to do. If you can't reach him, let me know. I'm totally crazed this week.

On Jun 18, 2012, at 11:23 AM, Luis Lavena wrote:

@tmm1 @raggi ping?


Reply to this email directly or view it on GitHub:
#333 (comment)

algritz commented Jun 18, 2012

Thanks, I'll come back once in a while to check if there's anything new.

Contributor

tmm1 commented Jun 18, 2012

I just pushed eventmachine 1.0.0.rc.1 to rubygems.

stakach commented Jun 19, 2012

@tmm1 @luislavena It's still an issue as FD_SETSIZE needs to be set at compile time otherwise it defaults to 64 sockets which is even worse then before.

All pre-compiled gems should have an FD_SETSIZE matching that of the ruby implementation they are compiled for.

FD_SETSIZE can only be removed completely if compilation occurs during gem installation.
We should come to an agreement that all ruby implementations running on Windows use a set size of 32767

stakach commented Jun 19, 2012

That or it needs to be compiled with the latest version of rubyinstaller ruby installed?

Contributor

garbagecat commented Jun 19, 2012

I must admit that FD_SETSIZE is an artifact of my programming style from the days when it wasn't surprising for kernels to have very small ones, like 64 or 256. I'd love to get rid of it altogether.

There are some commands in the actual EM module that allow you to set the number of available descriptors programmatically if your platform supports epoll. If you start your run as root, you can set this limit as high as 100,000 and then drop privs. That's what I do in most of my EM programs.

On Jun 19, 2012, at 2:56 AM, Stephen von Takach wrote:

@tmm1 @luislavena It's still an issue as FD_SETSIZE needs to be set at compile time otherwise it defaults to 64 sockets which is even worse then before.

All pre-compiled gems should have an FD_SETSIZE matching that of the ruby implementation they are compiled for.

FD_SETSIZE can only be removed completely if compilation occurs during gem installation.
We should come to an agreement that all ruby implementations running on Windows use a set size of 32767


Reply to this email directly or view it on GitHub:
#333 (comment)

Contributor

luislavena commented Jun 19, 2012

@stakach latest RubyInstaller (1.9.3-p194) has FD_SETSIZE to 32767, so pre-compiled EventMachine should set to the same value.

The problem with that is that it will fail with previous versions of Ruby, like 1.8.7 (which has 256) or 1.9.2 (which has 64).

I can't go back in time and fix previous versions, but Ruby itself has been fixed so next release it will not matter how many FD_SETSIZE you set in your application, it will not segv due Ruby anymore.

So my proposal is remove the definition from the code and let extconf determine if FD_SETSIZE is required (by looking to previous $defs)

Thoughts?

Contributor

garbagecat commented Jun 19, 2012

No objection from me.

On Jun 19, 2012, at 9:02 AM, Luis Lavena wrote:

@stakach latest RubyInstaller (1.9.3-p194) has FD_SETSIZE to 32767, so pre-compiled EventMachine should set to the same value.

The problem with that is that it will fail with previous versions of Ruby, like 1.8.7 (which has 256) or 1.9.2 (which has 64).

I can't go back in time and fix previous versions, but Ruby itself has been fixed so next release it will not matter how many FD_SETSIZE you set in your application, it will not segv due Ruby anymore.

So my proposal is remove the definition from the code and let extconf determine if FD_SETSIZE is required (by looking to previous $defs)

Thoughts?


Reply to this email directly or view it on GitHub:
#333 (comment)

stakach commented Jun 19, 2012

What I'm trying to say is that the newly released eventmachine does not run on the latest version of RubyInstaller.
It segfaults as it was not compiled with FD_SETSIZE set to 32767.

As the next version of Ruby will run no matter what FD_SETSIZE is I vote we add it back to eventmachine, set to 32767 so it runs on the current version and supports more then 64 sockets when the next version of ruby is released.

Contributor

tmm1 commented Jun 19, 2012

I'm building these gems using rake-compiler on my Mac. AFAIK, there will be no FDSET_SIZE in $defs there.

How do I create a gem that works with 1.8.7, 1.9.2 and 1.9.3 on windows without segfaulting?

Contributor

luislavena commented Jun 19, 2012

@stakach that is why I mentioned on previous comment that FD_SETSIZE definition should be checked on extconf.rb and if not found, defined.

I'm seem unable to locally compile EventMachine to try out my theory, but checking against RbConfig::CONFIG["cppflags"] for FD_SETSIZE should do the trick.

stakach commented Jun 19, 2012

Maybe we don't have a pre-compiled version then? That would work on all platforms?

Contributor

luislavena commented Jun 19, 2012

@stakach a pre-compiled version is required since OpenSSL is not available for compilation on Windows (working on solve that issue)

@tmm1 can you check RbConfig::CONFIG["CPPFLAGS"]?

What about something like this?

if RbConfig::CONFIG["host_os"] =~ /mingw/
  found = RbConfig::CONFIG.values_at("CFLAGS", "CPPFLAGS").
    any? { |v| v.include?("FD_SETSIZE") }

  $defs << "-DFD_SETSIZE=32767" unless found
end

I just tested against RubyInstaller 1.8.7 and 1.9.3, and FD_SETSIZE is not defined if present in RbConfig.

Now, RubyInstaller 1.8.7 had FD_SETSIZE=256 and 1.9.3 now have 32767, so perhaps the size can be done using RUBY_VERSION accordingly.

When cross compiling RUBY_VERSION is set accordingly so should work.

Let me know otherwise.

stakach commented Jun 19, 2012

Setting setsize to 32767 won't segfault older versions of RubyInstaller.
The segfault only occurs if the setsize in Eventmachine is smaller then the setsize in Ruby

Contributor

luislavena commented Jun 19, 2012

@stakach sorry, don't remember the details, been a few months since I dealt with this issue at Ruby-Core.

Perhaps the check for FD_SETSIZE is enough and check for version is not required. Agree?

stakach commented Jun 19, 2012

Agreed :)

Contributor

tmm1 commented Jun 21, 2012

Can someone start a pull request?

Contributor

luislavena commented Jun 21, 2012

@tmm1 will give a try when I arrive home later today.

Contributor

luislavena commented Jun 21, 2012

@tmm1 see #337

😃

Contributor

luislavena commented Jun 23, 2012

@tmm1 ping?

😃

Contributor

luislavena commented Jun 26, 2012

@stakach 1.0.0.rc2 is out, however binary gem doesn't work with RubyInstaller 1.9.3-p194, only with TCS release.

AARGGH

@tmm1 can you tell me if the generated Makefile in tmp/i386-mingw32 contains -DFD_SETSIZE=32767 ?

I see no logic based on previous @stakach comment that it wouldn't segfault.

stakach commented Jun 26, 2012

@luislavena it runs on all the versions of ruby installer prior to p194.

So my assumption here is that it was not compiled with -DFD_SETSIZE=32767

Contributor

luislavena commented Jun 26, 2012

@stakach I'll get my ubuntu VM tomorrow and test cross-compilation, this entire issue is making me bald, so I guess @tmm1 head is already a lovely shinny one 😁

stakach commented Jun 26, 2012

My logic about the segfault only occuring if the setsize in Eventmachine is smaller then the setsize in Ruby is valid.

The easy solution would be just to re-add the define in project.h as 32767 - might not be the most pretty or technically valid however it will run on all versions of ruby installer and rubinius.

Contributor

tmm1 commented Jun 26, 2012

It's possible I didn't fully clean before doing the rc2 build. I am seeing the -DFDSET_SIZE as part of the gcc command.

I just did a full clean and re-build the rc2 gem. Can you see if it works any better: http://cl.ly/1G0a210F2T2K473z0X13

Azolo commented Jun 26, 2012

It doesn't for me =( Here's the build I created using @luislavena's gem-compiler gem: http://ge.tt/9Ct7AeJ?c

It has the gem that's compiled on my local machine plus the tmp folder that gem-compiler created. I don't really know if that will help any but, it's there.

stakach commented Jun 26, 2012

I'm still seeing the seg-fault with your re-build @tmm1.
If I compile the gem on windows without SSL support it works.

Contributor

luislavena commented Jun 26, 2012

@stakach I just checked on my mac. FD_SETSIZE gets defined for 1.8.7 and 1.9.3 Makefiles:

rubyeventmachine/1.8.7/Makfile

CPPFLAGS = -DWITHOUT_SSL -DBUILD_FOR_RUBY -DHAVE_RB_TRAP_IMMEDIATE -DHAVE_RBTRAP -DHAVE_RB_TIME_NEW -DOS_WIN32 -DFD_SETSIZE=32767 -DHAVE_WINDOWS_H -DHAVE_WINSOCK_H -DHAVE_MAKE_PAIR   

rubyeventmachine/1.9.3/Makfile:


CPPFLAGS = -DWITHOUT_SSL -DBUILD_FOR_RUBY -DHAVE_RB_THREAD_BLOCKING_REGION -DHAVE_TBR -DHAVE_RB_THREAD_CHECK_INTS -DHAVE_RB_TIME_NEW -DOS_WIN32 -DFD_SETSIZE=32767 -DHAVE_WINDOWS_H -DHAVE_WINSOCK_H -DHAVE_MAKE_PAIR  $(DEFS) $(cppflags)

Neither of my 1.8.7 or 1.9.3 cross-compiled Rubies had FD_SETSIZE

fastfilereader extension didn't receive this code change, so I don't know if this could be the cause.

I just setup the environment locally, built OpenSSL 1.0.0j (did some changes to extconf for it) and built a native gem.

Can you download and test it? (don't have my Windows machine with me now)

http://cl.ly/2u0s2f003s001j3N3P3Y

Contributor

luislavena commented Jun 26, 2012

OK guys,

Except for the dependency on libgcc_s_sjlj-1.dll (for which you need to include DevKit (running devkitvars.bat) my build did work under:

  • 1.8.7-p358
  • 1.9.2-p290
  • 1.9.3-p194
  • 1.9.3-p232 (TCS build)

So there must be something interfering with the build.

@stakach if you can test and confirm then...

@tmm1 I'm going to send another pull request with the modifications I made. If you can build another mingw32 gem and upload to CloudApp for us to download and test, will be much appreciated.

I'm working really hard on getting gem-compiler working to provide a better building functionality, but free time is something scarce lately.

Thank you.

Azolo commented Jun 26, 2012

@luislavena I'm getting a pretty awesome error with your new download.

https://gist.github.com/2999990

I don't know what is going on there, I feel like it's probably my fault though.

Contributor

luislavena commented Jun 26, 2012

@Azolo see previous comment, you need libgcc dll in the PATH for this to work, you need devkitvars, you should have gotten a popup indicating the missing DLL.

That is the problem with the mingw-w64 compiler I had in my Mac, need to go back to stable 4.5.4 to avoid that dependency issue.

stakach commented Jun 26, 2012

@luislavena works for me!

Azolo commented Jun 26, 2012

@luislavena Ahh, gotcha. I totally misinterpreted that statement.

Sorry about that, it's working now. 😃

algritz commented Jun 27, 2012

Working for me now, A big thanks for your hard work guys, its really appreciated !

Contributor

luislavena commented Jun 28, 2012

Hello folks

OK, last try, here is another build of the gem:

http://cl.ly/1z0p2R2H162k1j1b3l0C

This time it doesn't require adding the DevKit to the PATH, it should work out of the box with normal gem installation.

Please uninstall the previous version of the gem I provided before, install this and try. I tested on 1.8.7 and 1.9.3 without issues.

If this work for others (to reduce the changes of works for me), @tmm1 the patches I applied are here:

https://github.com/luislavena/eventmachine/compare/minor-tweaks-cross-compile

And I will send a pull request once @stakach, @Azolo and @algritz confirm this new version works.

The step to build the gem where:

  • Use Ruby 1.8.7 natively
  • Have Ruby 1.8.7-p358 and 1.9.3-p125 cross compiled Rubies installed.
  • Build openssl_libs from ruby-pg with rake openssl_libs OPENSSL_VERSION=1.0.0j
  • Move the build artifacts of openssl into rake-compiler: mv build/builds/openssl-1.0.0j/ ~/.rake-compiler/builds/
  • Build the extensions with rake cross compile OPENSSL_VERSION=1.0.0j RUBY_CC_VERSION=1.8.7:1.9.3
  • Build the gem: rake cross native gem

Thank you guys for your patience while I replied, I was out sick (still).

Cheers.

Azolo commented Jun 28, 2012

@luislavena No weirdness here.

Contributor

luislavena commented Jun 28, 2012

Thanks @Azolo once I get confirmation from @stakach that this new version works, will send the pull request.

stakach commented Jun 29, 2012

@luislavena clean environment, no devkit in the path - looks good! Epic work.

Contributor

tmm1 commented Jun 29, 2012

Pushed up 1.0.0.rc.3 gems to rubygems.

stakach commented Jun 29, 2012

@tmm1 Unfortunately I'm seeing an error on all RubyInstallers:

Unable to load the EventMachine C extension; To use the pure-ruby reactor, require 'em/pure_ruby'
LoadError: cannot load such file -- 1.9/rubyeventmachine

etc

Contributor

luislavena commented Jun 29, 2012

@tmm1 seems is missing 1.9 folder into the gem, perhaps RUBY_CC_VERSION is not set to 1.8.7:1.9.3 ?

Contributor

tmm1 commented Jun 29, 2012

I copy-pasted that command.. I think maybe I never installed a cross-compiled 1.9.3 locally, only 1.9.2.

Yep, oops:

$ rake-compiler cross-ruby
Updating .rake-compiler/config.yml
Found Ruby version 1.8.7 (.rake-compiler/ruby/ruby-1.8.7-p334/lib/ruby/1.8/i386-mingw32/rbconfig.rb)
Found Ruby version 1.9.2 (.rake-compiler/ruby/ruby-1.9.2-p180/lib/ruby/1.9.1/i386-mingw32/rbconfig.rb)
Contributor

luislavena commented Jun 29, 2012

@tmm1 ahh, this part:

Have Ruby 1.8.7-p358 and 1.9.3-p125 cross compiled Rubies installed.

Is what was missing :-P

Just changing RUBY_CC_VERSION to 1.8.7:1.9.2 should work.

Sorry for not being more clear, build against 1.9.2 or 1.9.3 will be the same, I'm always installing latest.

Contributor

tmm1 commented Jun 29, 2012

OK, try 1.0.0.rc.4.

Contributor

luislavena commented Jun 29, 2012

@tmm1 tested against echo server (from README)

  • Ruby 1.8.7-p358
  • Ruby 1.9.2-p290
  • Ruby 1.9.3-p194
  • Ruby 1.9.3-p231 (TCS build)

Works on all of them.

Thank you! ❤️ ❤️ ❤️

@luislavena luislavena closed this Jun 29, 2012

Contributor

tmm1 commented Jun 29, 2012

Excellent. Thanks for all the help everyone!

Contributor

luislavena commented Jun 29, 2012

Thanks to you @tmm1 for your patience!

ruilov commented Jun 29, 2012

I was running into this too, and wanted to thank everyone here for the hard work getting to the bottom of the issue. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment