Support x64 builds #2

Merged
merged 9 commits into from Aug 23, 2013

Projects

None yet

4 participants

@luislavena
Contributor

The following changes enhance the build process of Ruby and cross-compilation, ensuring the cross-compiled gems will work with newer versions of RubyInstaller.

I'm opening this pull request to discuss the modifications. I'm not a bash guru and the script might be brittle, but seems to be getting the job done.

I've implemented a separate prepare_xrubies.sh script to compile all the versions as it depends on different compilers.

Right now, it compiles the following:

  • 1.8.7 with i586-mingw32msvc compiler (bundled with Ubuntu)
  • 1.9.3 with i686-w64-mingw32 compiler (GCC 4.7.2 from mingw-w64 project)
  • 2.0.0 with i686-w64-mingw32 compiler (same as above)
  • 2.0.0 with x86_64-w64-mingw32 compiler (64bits of the same compiler)

It still requires:

  • Add 2.0.0 to RUBY_CC_VERSION for the other scripts
  • Avoid mingw-w64 be downloaded multiple times
  • Avoid $PATH being altered every time
luislavena added some commits Jul 18, 2013
@luislavena luislavena Use 64bits VM
To be able to build 64bits binaries for Windows we need 64bits
compilers that are not provided by Ubuntu.

These compilers requires 64bits host OS, so we need to adjust the VM
for this.
52f23a1
@luislavena luislavena Install Ruby 2.0.0 in the VM
To target 64bits Windows, we need Ruby 2.0.0 as is the only version of
Ruby RubyInstaller provides binaries for.
b397291
@luislavena
Contributor

@tjschuck let me know what you think of these changes.

Last but not least, thank you so much for making this! ❤️ ❤️ ❤️

@luislavena
Contributor

@tjschuck I've updated the pull request and included a few changes.

This seems to work! (at least with the sample_gem).

Will test with sqlite3 next and let you know!

@luislavena luislavena referenced this pull request in eventmachine/eventmachine Jul 19, 2013
Merged

fix compilation for Ruby 2.0.0 in Windows x64 #411

@luislavena
Contributor

@tjschuck I've done some minor tweaks to ensure 1.9.3 compiled Rubies are as close as possible to Windows ones.

It remains to fix the provision details and remove from the other scripts the cross-ruby task.

I've used this with good success with some gems and right now tweaking sqlite3-ruby for a release.

Thank you so much for taking the time and work on this, I really appreciate what you did and looking forward your comments!

❤️ ❤️ ❤️

@tjschuck tjschuck was assigned Jul 23, 2013
@tjschuck tjschuck commented on the diff Jul 23, 2013
bootstrap.sh
@@ -7,6 +7,28 @@ as_vagrant='sudo -u vagrant -H bash -l -c'
apt-get -y update
apt-get install -y curl git-core mingw32 default-jdk
@tjschuck
tjschuck Jul 23, 2013 Owner

@luislavena The updates below mean we can remove the installation of mingw32 here, correct?

@luislavena
luislavena Jul 23, 2013 Contributor

Actually no, I found there is an issue compiling things for Ruby 1.9.3 with GCC 4.7.2 (headers and features) that will cause compiled extensions to fail loading on Windows 😢

That is why in prepare_xrubies.sh I build 1.9.3 with Ubuntu's mingw and then switch to mingw-w64 for Ruby 2.0.0.

@tjschuck
tjschuck Jul 23, 2013 Owner

Gotcha -- good to know.

@tjschuck tjschuck and 1 other commented on an outdated diff Jul 23, 2013
bootstrap.sh
+mingw64='x86_64-w64-mingw32-gcc-4.7.2-release-linux64_rubenvb.tar.xz'
+
+$as_vagrant 'mkdir -p ~/mingw'
+
+if [ ! -f $mingw32 ]; then
+ $as_vagrant "curl -L http://downloads.sourceforge.net/mingw-w64/$mingw32 -o ~/mingw/$mingw32"
+ $as_vagrant "tar -C ~/mingw -xf ~/mingw/$mingw32"
+fi
+
+if [ ! -f $mingw64 ]; then
+ $as_vagrant "curl -L http://downloads.sourceforge.net/mingw-w64/$mingw64 -o ~/mingw/$mingw64"
+ $as_vagrant "tar -C ~/mingw -xf ~/mingw/$mingw64"
+fi
+
+# add mingw-w64 to the PATH
+# TODO: do not add if already added
@tjschuck
tjschuck Jul 23, 2013 Owner

So I don't think this is a concern, or the file existence checks above, because this bootstrap script only runs as the provisioner, i.e. during vagrant up. Once you've created the box, this won't be run again when you restart it and/or SSH into it, only when you destroy-and-reprovision.

@luislavena
luislavena Jul 23, 2013 Contributor

Ah, thing is that I kept using halt and when doing up again it re-provision it. Then I learn you could use suspend instead 😉

I'll remove the check then.

@tjschuck
tjschuck Jul 23, 2013 Owner

Ah, yeah, I never use halt -- I usually just destroy, but otherwise, I'll do suspend and resume.

But since I even recommend halt + up in the docs, maybe you're right. Should probably make the bootstrapper reasonably idempotent.

@tjschuck
Owner

Your "Avoid mingw-w64 be downloaded multiple times" todo is already handled by bootstrap.sh#L16-L24, right? If so, check it off!

@luislavena
Contributor

@tjschuck actually doesn't work. My bash-fu is pretty limited so by checking if a file exist when is set to a local variable seems do not work and it always attempt to download it and extract it 😢

The same happens with the last item in the list, the PATH gets added every time, so I think a check using grep might be necessary.

Thank you for taking the time to review this!

@tjschuck
Owner

@luislavena Working on both right now! More soon.

@tjschuck
Owner

@luislavena For both of those todos, give this a shot: https://gist.github.com/tjschuck/0e49f912f7dd197fe950

@luislavena
Contributor

@tjschuck it worked!

I just committed your gist to the branch (and gave credits, is your patch) 😉

I think this looks good for merging!

I'm working now into fixing both sqlite3 and mysql gems to use this Vagrant box!

Again, dunno how I can thank your effort building this, as I was pulling my hair out last time attempting to get this setup properly.

Thank you! ❤️ ❤️ ❤️

@tjschuck tjschuck commented on an outdated diff Jul 29, 2013
package_all.sh
@@ -3,6 +3,7 @@
set -e
source "$HOME/.rvm/scripts/rvm"
+./prepare_xrubies.sh
./package_native.sh $1
./package_java.sh $1
./package_win32_1.9.3_binary.sh $1
@tjschuck
tjschuck Jul 29, 2013 Owner

@luislavena Can you update this to package_win32_fat_binary.sh? The only reason I had the 1.9.3 version here is because the fat binary one didn't work!

@tjschuck
tjschuck Jul 29, 2013 Owner

Once this is merged, I'll probably remove the 1.9.3 win packager altogether -- FAT BINS 4EVA.

@tjschuck
Owner

@luislavena One last update: https://gist.github.com/tjschuck/0f356bceb6aa3816dae3

rake-compiler --pre requires RubyGems version >= 1.8.25, and the jruby version that gets installed has an out-of-date one. Updating rubygems fixes the issue.

Once that's done, this looks good to go!

@luislavena
Contributor

requires RubyGems version >= 1.8.25, and the jruby version that gets installed has an out-of-date one. Updating rubygems fixes the issue.

Can't we lock at RubyGems 1.8.25? I'm not very confortable with updating it to latest RubyGems 2.0.x.

Will do the updates soon and get back to you!

@tjschuck
Owner

@luislavena Yep -- bump to whatever version you'd like. It was only jruby that wasn't up-to-date enough.

@luislavena
Contributor

Hello @tjschuck, I've made the changes that you commented and tested this against several projects with success.

However, noticed that the extension generated for Ruby 2.0 contains debug symbols.

This is something I've already reported to Ruby-Core and didn't get fixed yet 😢

I'm looking into which flags I need to set for Ruby 2.0 compilation and will push my last commit for that.

Will keep you posted.

Cheers.

@luislavena
Contributor

Hello @tjschuck, I found a solution!

I've tested against bcrypt-ruby (after doing bundle update rake-compiler due the old version) and all the following gems were generated:

bcrypt-ruby/pkg/bcrypt-ruby-3.1.1-x64-mingw32.gem
bcrypt-ruby/pkg/bcrypt-ruby-3.1.1-x86-mingw32.gem
bcrypt-ruby/pkg/bcrypt-ruby-3.1.1-x86-mswin32-60.gem

All of them include the extensions:

$ gem spec bcrypt-ruby/pkg/bcrypt-ruby-3.1.1-x64-mingw32.gem files
...
- spec/spec_helper.rb
- lib/2.0/bcrypt_ext.so

$ gem spec bcrypt-ruby/pkg/bcrypt-ruby-3.1.1-x86-mingw32.gem files
...
- spec/spec_helper.rb
- lib/1.8/bcrypt_ext.so
- lib/1.9/bcrypt_ext.so
- lib/2.0/bcrypt_ext.so

Cheers!

@luislavena luislavena referenced this pull request in libgit2/rugged Aug 4, 2013
Merged

Add Rugged cross compilation support #241

2 of 2 tasks complete
@johnnyshields

@tjschuck @luislavena Just wanted to express my appreciation for the great work you both are doing!

@DannyOcean

Agree with @johnnyshields, your posts helped me a lot. Thanks!

@luislavena luislavena referenced this pull request in sparklemotion/sqlite3-ruby Aug 5, 2013
Closed

Integration of sqlite knapsack package into gem #101

@tjschuck
Owner
tjschuck commented Aug 5, 2013

@luislavena This is awesome. Thanks! Can you squash this down to a handful of logical commits? Then it'll be good to merge!

@luislavena
Contributor

@tjschuck will rebase interactive tonight and force-push.

Thank you! ❤️ ❤️ ❤️

@tjschuck
Owner
tjschuck commented Aug 5, 2013

Thank YOU!

luislavena added some commits Jul 19, 2013
@luislavena luislavena Download and setup mingw-w64 compilers
These compilers will open the room for cross-compilation of newer
versions of Ruby, like 2.0.0

- Download the latest stable/release binaries (4.7.2) used by
RubyInstaller too and setup inside ~/mingw

- Append the `bin` directories of mingw-w64 to the PATH.
13045c7
@luislavena luislavena Task to cross-compile all Rubies
Instead of delaying the compilation of the cross-version of Ruby, we
trigger all the compilation at once.

- Use Ubuntu's GCC to compile 1.9.3 to match the GCC version used in
Windows (DevKit)

We do this to avoid cross-compilation of some gems detect
characteristics of mingw-w64 headers that will not be present on
Windows version of 1.9.3

This happen when using mingw-w64 to cross-compile projects like
EventMachine that uses some settings of mingw-w64 headers not present
in Windows counterpart for Ruby 1.9.3

Doing this we reduce the chances of messing things up when
cross-compiling gems.

- Turn debugging off for Ruby 2.0

For some reason, Ruby 2.0 enables it's debugging flags when
cross-compiled, resulting in massive binaries for all extensions
compiled with it.

While this has been reported to Ruby-Core developers, this still hasn't
been fixed yet.

Instead, reduce the debugging symbols to the minimum to allow gdb
remain usable.
07c4b05
@luislavena luislavena Remove cross-compilation from specific tasks
Prepare cross-compiled Rubies in one place and use them.
059b4e7
@luislavena luislavena Use rake-compiler 0.9.1 1354f2a
@luislavena luislavena Adjust platforms used by sample gem
Support:

- x86-mingw32 (i386-mingw32)
- x86-mswin32-60 (i386-mswin32 - backward compatibility)
- x64-mingw32 (x64-mingw32 - newer 64bits)
d0ae996
@luislavena luislavena Include Ruby 2.0.0 on fat-binaries too
Now that we have 1.8.7, 1.9.3 and 2.0.0, we can safely build
fat-binaries.
d364b21
@luislavena luislavena JRuby: update and lock version of RubyGems
JRuby 1.7.4 do not have RubyGems 1.8.25 with it, so an update is
required.
a04c13a
@luislavena
Contributor

@tjschuck I've consolidated all the commits and is ready for final review (and hopefully merge)

Cheers!

@luislavena
Contributor

@tjschuck I've tested this with good results for sqlite3 gem, shall I remove the package 1.9.3 script?

Thank you.

@fbehrens fbehrens referenced this pull request in rails-sqlserver/tiny_tds Aug 19, 2013
Closed

Problem installing tiny_tds with x64-mingw32 and ruby 2.0.0p0 #110

@tjschuck
Owner

@luislavena is my hero.

I owe you a beer next time you're in NYC 🍻

@tjschuck tjschuck merged commit 643fe76 into tjschuck:master Aug 23, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment