No fat binary gems for windows #291

larskanis · 2013-10-29T21:12:32Z

ffi is currently not available as a fat binary gem for x64-mingw32. This is somewhat pity.

I've done some contributions regarding to ffi on windows and for cross build of binary gems in the past. I'm also using the ffi gem for several projects (most notably libusb). So, if I can help out in this area, please give me a note.

simi · 2013-10-30T18:47:08Z

I'm trying to build x86-mingw32. There were no x64-mingw32 gem binaries in history of ffi (according to http://rubygems.org/gems/ffi/versions/). Should I ping you for test when I'll have it prepared?

larskanis · 2013-10-30T19:55:12Z

Sorry, my PR description was somewhat short and not really helpful...

I've written the patches for x64-mingw32 cross build (in ffi and rake-compiler), but Wayne did never release a x64-mingw32 version. It used to work with a simple rake gem:win32 if all required tools are installed. The most easy way to do build fat binary gems is by using the rake-compiler-dev-box: https://github.com/tjschuck/rake-compiler-dev-box . The box has all compiler versions used for 1.8.7, 1.9.3 and 2.0.0 x86/x64 versions. It also cross builds matching ruby versions. Unfortunately there is an issue with the strip command in the dev-box, so you may better use this patch: tjschuck/rake-compiler-dev-box#4

Of course I can test the gems.

simi · 2013-10-30T19:57:22Z

I tried to cross compile ffi via rake-compiler-dev-box few minutes ago, but I'm getting Don't know how to build task 'cross'. There is missing something in Rakefile probably. I can dig deeper later (friday?).

larskanis · 2013-10-30T20:01:05Z

It's been a while that I cross built ffi the last time, maybe something has changed. I'll give it a try again and send you a pull request or better description. I guess can do so tomorrow.

simi · 2013-11-01T13:57:36Z

@larskanis OK, after few troubles I have built binaries. Please can anyone test them?

I tried 1.8.7, 1.9.3 and 2.0.0 (32-bit) installed with RubyInstaller with https://github.com/ffi/ffi/wiki/Windows-Examples#send-a-keystroke and https://github.com/rpeev/ffi-wingui-core/blob/master/examples/Hello.rbw examples without problems.

Links:
http://gems.apicko.cz/ffi-1.9.3-x86-mingw32.gem
http://gems.apicko.cz/ffi-1.9.3-x64-mingw32.gem (untested)

djberg96 · 2013-11-01T17:32:50Z

@simi Your 32-bit version loaded ok, but I had trouble with the 64-bit version:

irb(main):001:0> require 'ffi'
LoadError: cannot load such file -- ffi_c
    from C:/Ruby200-x64/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:135:in `require'
    from C:/Ruby200-x64/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:135:in `rescue in require'
    from C:/Ruby200-x64/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:144:in `require'
    from C:/Ruby200-x64/lib/ruby/gems/2.0.0/gems/ffi-1.9.3-x64-mingw32/lib/ffi.rb:14:in `rescue in <top (required)>'
    from C:/Ruby200-x64/lib/ruby/gems/2.0.0/gems/ffi-1.9.3-x64-mingw32/lib/ffi.rb:3:in `<top (required)>'
    from C:/Ruby200-x64/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:135:in `require'
    from C:/Ruby200-x64/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:135:in `rescue in require'
    from C:/Ruby200-x64/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:144:in `require'
    from (irb):1
    from C:/Ruby200-x64/bin/irb:12:in `<main>'

larskanis · 2013-11-01T19:34:35Z

@simi: The content of the two gems you built is identical (the x86-mingw32 version), but was only packaged in two separate steps. This sounds like a rake-compiler version before 0.9.0. I've pushed #298, which ensures a proper rake-compiler version is running.

I can package the win32 gems in the rake-compiler-dev-box with a simple

$ cd /vagrant
$ ./package_win32_fat_binary.sh ffi

Which troubles did you run into and how did you solve it? I would like to ensure, that using the rake-compiler-dev-box is as easy as possible. And which commands did you use to build the mingw gems?

simi · 2013-11-02T11:18:36Z

@larskanis I had troubles with build in dev-box. I had to edit ~/.rake-compiler/config.yml manually to ensure that 32-bit version is build by x86 mingw compiler. By default I had weird gem build with 1.8 and 1.9 lib for 32-bit and 2.0 build with 64-bit. I'll try to get 64-bit windows and build proper version.

I used rake-compiler-dev-box with:

rake-compiler cross-ruby HOST=i386-mingw32
vim /home/vagrant/.rake-compiler/config.yml # I edited 2.0 toolchain path to use x86 instead of x64
rake gem:win32

x64 build is experimental and not tested as I mentioned. I think we can ship 32-bit build now if you're ok with it, since x64 build wasn't introduced yet.

Also @larskanis I had to monkey patch bundler (1.3.5 version has ruby 1.8 issues) and rake-compiler to avoid tjschuck/rake-compiler-dev-box#1. This patch (rake-compiler/rake-compiler#79) did not fix that error for me in rake-compiler-dev-box. I used similar monkey patch to cucumber-attic/bool#37 (comment).

I used 0.9.1 rake-compiler AFAIK.

Also I have bad feeling about gem size. Last win32 gem was

1.9.0 June 9, 2013 x86-mingw32 (7.13 MB)

My build is 7 times smaller. I'm not sure about reason of this.

larskanis · 2013-11-02T15:34:18Z

@simi: Thank you very much for the detailed description! Unfortunately it does not sound that good. Proper cross rubies are installed by this script, that is called by package_win32_fat_binary.sh. I did the setup of these cross rubies with several rake-compiler-dev-box-VMs in the meantime, so I wonder that it doesn't work. So again everything should work with a simple

cd /vagrant
./package_win32_fat_binary.sh ffi

This is essentially the same as calling rake gem:win32, but ensures proper cross ruby setup and proper rvm versions.

The issue with win32/resolve is best solved in rake-compiler, to my mind. Unfortunately @luislavena was too busy to verify this, right now.

Did you run into this issue ?

Changing rake-compiler/config.yml usually means that the given ruby version was build with a different compiler than the C-ext. This is not really a problem, since ffi is no C++ and C object files are mostly cross version compatible.

The gem size reduction is no problem. This is caused by the stripping here. Not sure why Wayne's are not stripped.

simi · 2013-11-02T15:46:49Z

@larskanis I'll try clean rake-compile-dev-box again.

As I mentioned, that patch you send to rake-compiler is not fixing that issue for me. I was still getting that error. I ensured that rake-compiler with that patch was used. After monkey-patching I mentioned it worked.

I wasn't facing this issue.

larskanis · 2013-11-02T19:45:33Z

Oops, I forgot to mention, that the rake-compiler patch will only take effect, if you drop the build/*-mingw32 directories, so that the fake.rb is newly installed.

larskanis · 2013-11-02T20:31:57Z

@simi I finally tested your gem for x86-mingw32 above and can acknowledge that everything works as expected. I used the libusb test suite and the ffi test suite on Ruby-1.9.3-p392 and Ruby-2.0.0-p0.

simi · 2013-11-03T14:55:35Z

@larskanis do we need only Ruby 2.0.0 x64 binary? I can't compile 1.8.7 and 1.9.3 with mingw32 for w64 platform.

simi · 2013-11-03T15:08:45Z

I rebuilt x64 for 2.0.

http://gems.apicko.cz/ffi-1.9.3-x64-mingw32.gem (older is renamed to http://gems.apicko.cz/ffi-1.9.3-x64-mingw32-yanked.gem).

/cc @djberg96 can you test please? I have not x64 windows for now.

luislavena · 2013-11-03T15:23:35Z

That is correct, there is no 1.9.3 or 1.8.7 in x64.

Sorry folks, been extremely busy with work.

Sorry for top posting. Sent from mobile.
On Nov 3, 2013 11:55 AM, "Josef Šimánek" notifications@github.com wrote:

@larskanis https://github.com/larskanis do we need only Ruby 2.0.0 x64
binary? I can't compile 1.8.7 and 1.9.3 with mingw32 for w64 platform.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/291#issuecomment-27646027
.

simi · 2013-11-03T15:24:46Z

@luislavena ahh, thanks for info.

larskanis · 2013-11-03T19:57:34Z

@simi the new gem is working fine 👍 Tested on RubyInstaller x64 v2.0.0-p0 with test suites of libusb and ffi running on Windows-7. Thank you very much for your effort!

simi · 2013-11-03T19:59:27Z

Thanks @larskanis for test.

@headius @enebo are you interested into releasing Win gems?

headius · 2013-11-04T15:45:11Z

If the new gems are good I don't see why we can't release them. Thanks!

luislavena · 2013-11-04T17:49:19Z

@larskanis @simi if you guys can contribute some instructions (as another document, no need to be README), then others will be able to build the gem without too much trouble.

That will be great and everybody will benefit when the windows master is not around.

@headius how you would like to proceed to release such gem, shall someone build it and make it available? or can be push rights be made to rubygems for that user account?

Thank you folks for dealing with this issue! ❤️ ❤️ ❤️

simi · 2013-11-04T17:50:47Z

@luislavena that's my plan. I made a lot of steps so I need to do it again only with needed steps and I'll add info to wiki or somewhere with all steps needed to release x32 and x64 binary gems.

headius · 2013-11-04T19:54:05Z

@luislavena @simi How about we just give @simi rights to push the gem and we'll sort out a formal process for future releases? Neither Tom nor I have ever built and released binary gems, so we're happy to add others who know how to do it and have made it work.

@simi What's your account email for rubygems.org?

simi · 2013-11-04T20:06:26Z

@headius retro@ballgag.cz

larskanis · 2013-11-05T12:08:26Z

@simi, @headius: The reason I opened this github issue was to provide my help for ffi on Windows. Even if Windows is not my primary platform, I care about it, did years of development with it and I always have all relevant and up to date RubyInstaller versions to test and debug code, together with an up to date cross compilation environment. That's why I do this job for several other gems since years (including ffi, when Wayne forgot to do so). So what I could offer here, is to build, test, debug and push the ffi-x86/x64-mingw-gems for future releases in a timely fashion.

@simi It's not against you, but it was my impression that building the mingw-gems burdens you with a task you're not so familiar with?

simi · 2013-11-05T12:11:15Z

I think I'm able to build and test binary gems. But I have not Win64 for testing now. Sure @larskanis, feel free to do that, since you're doing it for a while yet. I can help you with testing or building when you will not be around or when you'll be busy.

larskanis · 2013-11-05T13:40:57Z

@simi OK even better.
@headius So my account is lars@greiz-reinsdorf.de

simi · 2013-11-05T14:15:49Z

@larskanis Also I found out that my x64 gem is missing checksums.

larskanis · 2013-11-05T15:06:03Z

sha-1/sha512 checksums within the gem are new in rubygems-2.0.0. So the file is generated with a previous version. Since the tar format of the gem has it's own checksums, I think it's passable to have them not included.

A trusted gpg certificate would bring an additional security benefit, but most gems miss that and the infrastructure is not really there to do proper verification of it. (Correct me if I'm wrong)

simi · 2013-11-05T15:07:40Z

Yup, I went thru last part of http://guides.rubygems.org/publishing/#gem_security

there is no well-established chain of trust for gem signing keys

enebo · 2013-11-05T17:50:19Z

@simi you have been added to be able to push for ffi. Can you coordinate getting a proper 64 bit binary with @larskanis (although you guys are already coordinating so this sentence is a bit redundant :) ) that you can push once it is ready?

simi · 2013-11-05T17:54:55Z

Cool @enebo. I think my current 1.9.3 win gems are ready to push. Can you confirm @larskanis, @djberg96? I'll try few tests also before pushing.

@enebo can you also add or swap me with @larskanis to be able to push ffi gem?

enebo · 2013-11-05T18:15:00Z

I will swap you two then. @larskanis you're it for these fat gems :) Thanks guys for taking this on. Many windows users will no doubt be worshiping you guys from afar.

larskanis · 2013-11-05T18:26:03Z

No objection from my side. We should push @simi's gems for this release. They are quite well tested now.

simi · 2013-11-05T18:26:38Z

Go for it @larskanis!

larskanis · 2013-11-05T19:23:13Z

Done. Thanks all for building, testing and working together!

larskanis · 2013-11-05T19:45:18Z

@simi I did a complete fresh setup of a rake-compiler-dev-box, today. I'm not sure, if I got this errors in the past, but I got an build error while cross compiling ruby-1.9.3 because it used the native gcc instead of the cross compiler. This was fixed after deleting ~/.rake-compiler/builds/i586-mingw32msvc/ruby-1.9.3-p448 and rerun package_win32_fat_gems.sh. Then I got an error about the missing cross task. I did no further investigation, but at least can verify that you couldn't build the gems as easy as I hoped and described.

simi · 2013-11-05T19:47:46Z

@larskanis I'll write a short guide how I built it soon. I sent you email today, so we can continue there to stop disturbing another people in this issue.

larskanis closed this as completed Nov 5, 2013

No fat binary gems for windows #291

No fat binary gems for windows #291

Comments

larskanis commented Oct 29, 2013

simi commented Oct 30, 2013

larskanis commented Oct 30, 2013

simi commented Oct 30, 2013

larskanis commented Oct 30, 2013

simi commented Nov 1, 2013

djberg96 commented Nov 1, 2013

larskanis commented Nov 1, 2013

simi commented Nov 2, 2013

larskanis commented Nov 2, 2013

simi commented Nov 2, 2013

larskanis commented Nov 2, 2013

larskanis commented Nov 2, 2013

simi commented Nov 3, 2013

simi commented Nov 3, 2013

luislavena commented Nov 3, 2013

simi commented Nov 3, 2013

larskanis commented Nov 3, 2013

simi commented Nov 3, 2013

headius commented Nov 4, 2013

luislavena commented Nov 4, 2013

simi commented Nov 4, 2013

headius commented Nov 4, 2013

simi commented Nov 4, 2013

larskanis commented Nov 5, 2013

simi commented Nov 5, 2013

larskanis commented Nov 5, 2013

simi commented Nov 5, 2013

larskanis commented Nov 5, 2013

simi commented Nov 5, 2013

enebo commented Nov 5, 2013

simi commented Nov 5, 2013

enebo commented Nov 5, 2013

larskanis commented Nov 5, 2013

simi commented Nov 5, 2013

larskanis commented Nov 5, 2013

larskanis commented Nov 5, 2013

simi commented Nov 5, 2013