Make use of rake-compiler-dock for building windows binary gems

[larskanis]

rake-compiler-dock is the successor of the rake-compiler-dev-box - at least in my little universe 😄
No really - it builds gems faster, with little to no setup and is easy to integrate. A simple 'rake gem:windows' usually does everything.

The ruby versions are the same as in this rake-compiler-dev-box pull request , so both environments could be used.

I tested 'rake gem:windows' on Linux and Windows successfully. I also tested the resulting gems on ruby-2.1.4-x64, ruby-2.2.1-x86 and ruby-2.2.1-x64 on Windows-7 with the test suite and got no failures. However OS-X is currently completely untested, for a lack of such a box.

[paulgrant999]

deleted some comments - as they are specific to a virtualbox bug.

http://www.nokogiri.org/tutorials/installing_nokogiri.html

+1 to skipping all the virtualbox stuff and just setting up a devkit compile.

[larskanis]

@paulgrant999 Thank you for trying out this pull request. I'll fix rake-compiler-dock to accept spaces in the username.

To the native build: I fixed this already in #989 and release 1.6.2 was the only one that built successfully on RubyInstaller Devkit. Then it got broken short time later. However it is probably not too much work to fix it again (here and in mini_portile) and add a appveyor build run, so that it doesn't get broken accidentally.

But, which specific Virtualbox bug did you encounter? I already implemented several workarounds for Virtualbox bugs. Maybe this one can also be addressed.

[paulgrant999]

@paulgrant999 Thank you for trying out this pull request. I'll fix rake-compiler-dock to accept spaces in the username.

the command executed on the dev-vm looked proper (the space was escaped) but for some reason the username caused an issue on actual execution. Not a big deal, the fix is easy - change the username (which is good practice if you are planning on doing anything cross-linux/windows) i.e. I do this already with anything I plan on compiling in path, just to avoid the headaches. I was surprised to find it using my username in the first place but I figure you are using the VM config stored by boot2docker/virtualbox somehow and need to access c:\Users-username-.whatever.

To the native build: I fixed this already in #989 and release 1.6.2 was the only one that built successfully on RubyInstaller Devkit. Then it got broken short time later. However it is probably not too much work to fix it again (here and in mini_portile) and add a appveyor build run, so that it doesn't get broken accidentally.

LOL. That is what I was trying to avoid; launching a VM just to cross-compile it. I thought the whole point of installing mingw was the ability to have a standard compilation method on windows (so you didnt have to screw with MS stuff regarding configs etc). So my question was much simpler:
if I have mingw installed, ruby 2.2.2 installed, devkit (from rubyinstaller), and I can supply both the headers/lib and/or precompiled so/dlls.... why wouldn't I just run a native compile, rather than fire up a vm/linux to cross-compile?

But, which specific Virtualbox bug did you encounter? I already implemented several workarounds for Virtualbox bugs. Maybe this one can also be addressed.

I got lucky; it turns out its not a virtualbox error. Mingw under devkit has ln installed; the ln isn't creating softlinks properly on the NTFS filesystem. Thats why I had such a huge problem following your instructions - the automount read/write in virtualbox uses ln -s to create softlinks; so the links are created but don't actually update (when the underlying disk nodes are changed) - its acting more like a clone. I come from a unix background, so I'm used to typing ln to make links, not the microsoft equivalent, mklink. As it happens, I'm storing some key information in a bat file outside my projects (to make sure it doesn't get put into git, as git default is not to follow softlinks) and when I created the link, out of habit I typed ln -s instead of mklink....with the result that the underlying link created by mingw created the soft-link BUT IT DOESN'T UPDATE or show up as a softlink on DIR. Compound this with the nature of the rake-compiler-dock commands being proxied through another VM to get to dev VM, where the underlying filesystem was a dup of the initial filesystem that didn't change (and had some permission issues) and this caused one hell of a screwup.

I'm new to the whole cross-compile/vm thing, so I'm not sure exactly which program is causing the problem (i.e. calling ln -s or its equivalent on WIN filesystem) but I do know that its a major problem when your soft-links act as a clone rather than a softlink. If you're prog is using ln -s to make softlinks on the WIN32 side, you might add in an assert that kills it if the ln -s doesn't work as expected i.e. check it. Its stupid but it would have significantly cut down on my problems if the program had just crapped out when the ln started acting wonky.

The second thing that caused me problems was rake-compiler-dock not acting like boot2docker; the command to drop into ::shell vm:: is boot2docker ssh; however when I tried that with rake-compiler-dock, ssh takes parameters (presumably related to to its function). rake-compiler-dock by itself just entered a wait state (without showing a command prompt) so I figured this was taking input on STDIN (for scripting purposes), which left me with no real way to get into the ::dev-vm:: without a lot of hunting (for keys); and since the dev-vm wasn't compiled by me, I had no way to get access to it - only way in was to the minimalist linux vm hosting the vm image for ::dev-vm:: which wouldn't let me just root around to see what the problem was on the ::dev-vm::.

Later on I caught the sentence in one of the docs for rake-compiler-dock (no args) not taking input on STDIN, but acting as a shell. The point is, the behavior was counter-intuitive from a user perspective (not from a programming perspective, in retrospect). if rake-compiler-dock is acting as a shell proxy to the ::dev-vm::, it should display a prompt (at least at the very first call) indicating it is acting-in-proxy for the dev-vm, rather than NO PROMPT, which would lead one to believe that its simply acting as a STDIN for local shell, for local stuff. You follow?

On the plus side I got the gem compiled and installed ;) Which is a positive. Seems like its working, but I don't know how to test the gem (run the tests). had to run an FTP server on my host just to get the stuff off, which was mildly irritating. But it is what it is ;)

I had serious hopes that the VM stuff had matured to the point where it was bullet-proof. Now I see, not so much. A non-syncing, soft-linked read-write host filesystem should definitely have raised some sort of error. I understand that virtualboxes read/write shares suffer from problems related to delays in updates.

[paulgrant999]

also for some reason, despite a day or two of googling:
this link:
http://www.nokogiri.org/tutorials/installing_nokogiri.html
didn't show up.

Obviously the first thing I looked for was an option to install nokogiri so that it would dynamically link to the precompiled binaries I downloaded from libxml/libxslt/libiconv/zlib in order to get around the fact that 2.2.2 binaries (dll vs so) weren't supplied with the gem. But as I wasn't even sure this was an option, I was left guessing at flags based on random posts from other people who had similar problems over the years.

So my first question, is this an option? A dll is just an SO meant to be linked against dynamically at run-time, vs at compile time, yes? Wouldn't supplying the dll's on the path, with an option for a smaller nokogiri.so work the same, provided you used the same binaries? Also for my own edification, can you link at compile-time to a DLL (under linux) using gcc/g++ (i.e. can you use a dll as an so?)? Also for my own edication, how portable is the SO - does it have its own symbol table and internal binary packing method; or is it more like a architecture-specific stack?

[larskanis]

@paulgrant999 Sorry, I understand only half of your comments. I've tried to make things more clear in the README of rake-compiler-dock .

The username-with-space issue is fixed now and we avoid spaces in the path as well . Several people stumbled across starting boot2docker correctly, therefore this is now integrated, too . This does not help against IP address issues or other (boot2)docker issues, but should help to make rake-compiler-dock more convenient on Windows and OSX. I'll release 0.4.0 on Monday, when I've got a chance to test this on Windows.

[paulgrant999]

#1 - thank you for the name fix ;)
#2 - Readme says:

rake-compiler-dock without arguments starts an interactive shell session. This is best suited to try out and debug a build.

rake-compiler-dock can also take the build command(s) from STDIN or as command arguments.

this is the behavior I am referring to:

when you start a program that is intended to read from STDIN, there is no prompt e.g.
--- example --
echo "some STDIN commands " | c:\rake-compiler-dock

^z
--- /example --

when you start a program that is intended to be an interactive shell (to a unix box), there is usually a prompt, or connect statement e.g.
-- example
c:\rake-compiler-dock
Connecting: Linux Ubuntu 12.04
~/
quit
-- /example --

A) the behavior of an "interactive shell" with no prompt, and no connect statement is the behavior I would expect from a program taking input from STDIN, not an interactive shell.

B) the most natural way to run an interactive shell is to pass it an argument; for boot2docker, this argument is: c:\boot2docker ssh...

but when you run rake-compiler-dock (with no args or STDIN redirect), the only behavior you get is no prompt (indicating this is a program meant to run on the local machine, NOT a proxy to the dev-vm).

if you run rake-compiler-dock ssh, instead of taking you into an interactive shell (with prompt), it is probably trying to use ssh on the dev-vm.

This is counterintuitive and unexpected.

Suggestion(s):
A) Put a prompt in or at least a machine connect statement, when rake-compiler-dock is interactive mode (no STDIN, or ARGS)

B) Provide (and document) a command, preferably 'ssh' (no arguments) to drop into the vm image. If you need to ssh from the dev-vm, you will need to supply args i.e.
c:\rake-compiler-dock ssh
Welcome to Ubuntu 12.04
~/ >

c:\rake-compiler-dock ssh some-other-machine.com
Welcome to Ubuntu 12.04
SSH some-other-machine.com
SSH ~/ >

(start interactive ssh shell on dev-vm)

SSH > open

optionally: if you wish to support interactive ssh from dev-vm to something else, you may consider supplying a second arg like:

c:\rake-compiler-dock ssh -from-vm
Welcome to Ubuntu 12.04

(start interactive ssh shell on dev-vm)

SSH>
SSH > open some-other-machine

Thats just a suggestion. Fit the syntax to what makes sense from a unix perspective ;)
Keep in mind from the user perspective, rake-compiler-dock is a gem on the WINDOWS machine.

it would be quite normal to get the connect args, and open an ssh direct to the underlying vm ;) (i.e. run ssh.exe or putty (commandline)

No doubt I could just:
rake-compiler-dock sh
and that might bring up an interactive shell, right?

so when I run
rake-compiler-dock ssh,
I don't mean to run ssh on the dev-vm,
I mean to run ssh on the host machine, with args/keys to be supplied by rake-compiler-dock.

Last thing, you might supply the direct connect args to the underlying dev-vm..
Reason is, cmd.exe is a crappy crappy version of a shell. This way I can just open up a proper terminal program, and connect directly to the underlying dev-vm (with full support). Example of problems using the straight STDiN technique versus launching an interactive shell, is CMD is horrible at handling ansi sequences for redraw scrolling - which makes seeing the commands being run very difficult, as it generally just straight scrolls; putty would render it properly.

Its a user-interface thing.

as to the other questions, no worries - it was for my own learning. I can look them up ;)

[paulgrant999]

I'm not sure how you are implementing the pass through, but I think you are parsing input on STDIN and passing it to dev-vm. If you were for example, to open up sh on dev-vm (rake-compiler-dock sh) then the prompt would be supplied by the shell running on dev-vm....

[larskanis]

but when you run rake-compiler-dock (with no args or STDIN redirect), the only behavior you get is no prompt (indicating this is a program meant to run on the local machine, NOT a proxy to the dev-vm).

This is not intended. There must be something wrong. It should show a prompt like this on Linux:

lars@netzbuch:~$ rake-compiler-dock
lars@61e81dc4c740:/home/lars$ exit
lars@netzbuch:~$ 
lars@netzbuch:~$ rake-compiler-dock bash
lars@8df3ab8c418b:/home/lars$ exit
lars@netzbuch:~$ 

... and it usually looks similar on Windows. It makes no difference if I add bash as a parameter or not. bash is just the default command - see here .

I'm not sure how you are implementing the pass through, but I think you are parsing input on STDIN and passing it to dev-vm.

Nothing fancy, it's more simply. All arguments to rake-compiler-dock are passed to the container as command with arguments. STDIN is directly connected to the docker run command - it is not parsed in any way. And STDOUT/STDERR is directly connected as well. But note that the use of STDIN to rake-compiler-dock is broken in 0.3.x, because it allocates a pseudo-TTY, even if STDIN is no TTY. This is fixed here and will be part of 0.4.0. So for now only interactive session and command line works.

It could be, that boot2docker makes use of ssh internally, to connect to the VM. But this is an implementation detail of boot2docker only. rake-compiler-dock ssh should simply start the ssh command within the container, print it's help page and exits the container with statuscode 255. There is no further interpretation of the arguments:

lars@netzbuch:~$ rake-compiler-dock ssh
usage: ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]
           [-D [bind_address:]port] [-E log_file] [-e escape_char]
           [-F configfile] [-I pkcs11] [-i identity_file]
[...]
lars@netzbuch:~$ 

[knu]

Looks fascinating! I'll try this with boot2docker-xhyve on OS X to see if it works for me.

[knu]

Hmm, I get Gem::RemoteFetcher::FetchError: Errno::ETIMEDOUT: Connection timed out - connect(2) for "rubygems.global.ssl.fastly.net" port 443 (https://rubygems.org/gems/***.gem) too often and the build does not seem to proceed. I'll have to see how to fix the network problem.

[larskanis]

@knu boot2docker installed per homebrew or the official installer makes use of VirtualBox and automatically sets a host-only network up for the docker communication and NAT for the internet connection. I guess something similar is required for xhyve.

[knu]

It's not that the network isn't set up, but what's happening is a gem once succeeded fails in another run. The network stack might not be stable enough yet on xhyve. I've almost succeeded in eliminating the use of VirtualBox when I use VMware Fusion and xhyve, but I'll give it a try again.

[larskanis]

docker-machine will eventually be the successor of the boot2docker-installer. It already supports VMware-Fusion. However rake-compiler-dock has no support for setting up a VM per docker-machine, so far. But if you get docker running with whatever environment, then rake-compiler-dock should be able to use it.

[knu]

@larskanis I reinstalled boot2docker 1.7.1 with VirtualBox and tried rake gem:windows on OS X 10.10.4, but it failed as follows:

% rake gem:windows
WARNING: cross compilation not available: No MinGW tools or unknown setup platform?
rake-compiler-dock bash -c "bundle && rake cross native gem MAKE='nice make -j`nproc`' RUBY_CC_VERSION=1.9.3:2.0.0:2.1.6:2.2.2"

boot2docker is available, but not ready to use. Trying to start.
Virtual machine boot2docker-vm already exists
.oWriting /Users/knu/.boot2docker/certs/boot2docker-vm/ca.pem
Writing /Users/knu/.boot2docker/certs/boot2docker-vm/cert.pem
Writing /Users/knu/.boot2docker/certs/boot2docker-vm/key.pem
Waiting for VM and Docker daemon to start...

Started.

To connect the Docker client to the Docker daemon, please set:
    export DOCKER_HOST=tcp://192.168.59.103:2376
    export DOCKER_CERT_PATH=/Users/knu/.boot2docker/certs/boot2docker-vm
    export DOCKER_TLS_VERIFY=1

Or run: `eval "$(boot2docker shellinit)"`

Using above environment variables for starting rake-compiler-dock.
Fetching gem metadata from https://rubygems.org/........
Fetching version metadata from https://rubygems.org/..
Installing rake 10.3.2
Installing hoe 3.13.0
Installing hoe-bundler 1.2.0
Installing hoe-debugging 1.2.0
Installing hoe-gemspec 1.0.0
Installing hoe-git 1.6.0
Using json 1.8.1
Installing mini_portile 0.6.1
Installing minitest 2.2.2
Installing racc 1.4.12 with native extensions
Installing rake-compiler 0.9.3
Installing rake-compiler-dock 0.4.3
Installing rdoc 4.1.2
Installing rexical 1.0.5
Using bundler 1.10.5
Bundle complete! 13 Gemfile dependencies, 15 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
Post-install message from rdoc:
Depending on your version of ruby, you may need to install ruby rdoc/ri data:

<= 1.8.6 : unsupported
 = 1.8.7 : gem install rdoc-data; rdoc-data --install
 = 1.9.1 : gem install rdoc-data; rdoc-data --install
>= 1.9.2 : nothing to do! Yay!
no configuration section for specified version of Ruby (rbconfig-x64-mingw32-1.9.3)
cd tmp/x86-mingw32/nokogiri/1.9.3
/usr/local/rvm/rubies/ruby-2.2.2/bin/ruby -I. ../../../../ext/nokogiri/extconf.rb  --enable-cross-build
Could not find json-1.8.1 in any of the sources
Run `bundle install` to install missing gems.
rake aborted!
Command failed with status (7): [/usr/local/rvm/rubies/ruby-2.2.2/bin/ruby ...]
/usr/local/rvm/gems/ruby-2.2.2/gems/rake-compiler-0.9.3/lib/rake/extensiontask.rb:188:in `block (2 levels) in define_compile_tasks'
/usr/local/rvm/gems/ruby-2.2.2/gems/rake-compiler-0.9.3/lib/rake/extensiontask.rb:185:in `block in define_compile_tasks'
/usr/local/rvm/gems/ruby-2.2.2/bin/ruby_executable_hooks:15:in `eval'
/usr/local/rvm/gems/ruby-2.2.2/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => native => native:x86-mingw32 => native:nokogiri:x86-mingw32 => tmp/x86-mingw32/stage/lib/nokogiri/1.9/nokogiri.so => copy:nokogiri:x86-mingw32:1.9.3 => tmp/x86-mingw32/nokogiri/1.9.3/nokogiri.so => tmp/x86-mingw32/nokogiri/1.9.3/Makefile
(See full trace by running task with --trace)
rake aborted!
Command failed with status (1): [docker run -v /Users/knu/src/github/nokogiri:/Users/knu/src/github/nokogiri -e UID\=1000 -e GID\=1000 -e USER\=knu -e GROUP\=_staff -e ftp_proxy\= -e http_proxy\= -e https_proxy\= -e RCD_HOST_RUBY_PLATFORM\=x86_64-darwin14 -e RCD_HOST_RUBY_VERSION\=2.2.3 -e RCD_IMAGE\=larskanis/rake-compiler-dock:0.4.2 -w /Users/knu/src/github/nokogiri --rm -i -t larskanis/rake-compiler-dock:0.4.2 runas sigfw bash -c bundle\ \&\&\ rake\ cross\ native\ gem\ MAKE\=\'nice\ make\ -j\`nproc\`\'\ RUBY_CC_VERSION\=1.9.3:2.0.0:2.1.6:2.2.2]
/Users/knu/src/github/nokogiri/Rakefile:358:in `block in <top (required)>'
Tasks: TOP => gem:windows
(See full trace by running task with --trace)

Is this OS X specific?

[larskanis]

@knu Thanks for trying out! Your Gemfile.lock is seems to be rather old. You could update it, or add RUBYOPT=--disable-rubygems to the rake call above.

[larskanis]

@knu Did you have success with the comment above ?

[knu]

@larskanis Yes, thanks for your help. It took awhile so I went to bed, woke up in the morning and just saw it had finished in a couple hours.

[flavorjones]

@larskanis Great stuff here. Merging into master momentarily.