Error while running on Windows with jRuby

[shtirlic]

<script>:1: Cannot run program "uname" (in directory "D:\ruby-exp\sunatra"): CreateProcess error=2, The system cannot find the file specified (IOError) from C:/jruby/lib/ruby/gems/1.8/gems/shotgun-0.6/bin/shotgun:135:in `each' from C:/jruby/lib/ruby/gems/1.8/gems/shotgun-0.6/bin/shotgun:135:in`find' from C:/jruby/lib/ruby/gems/1.8/gems/shotgun-0.6/bin/shotgun:135 from C:/jruby/lib/ruby/gems/1.8/gems/shotgun-0.6/bin/shotgun:19:in `load' from C:\jruby\bin\shotgun:19

[rkh]

Shotgun forks the ruby runtime. Forking is not supported, neither on windows, nor by jruby. You could try sinatra-reloader.

[shtirlic]

Thx for workaround, maybe FFI can help? Is it possible to make CreateProcess and Wait wrappers?

[rkh]

Should be possible. I know you can do a fork via FFI on JRuby. Still, forking the JVM in general really does not work well (or at least so I'm told). I think it should be able to start processes and access the handlers through FFI on MRI, however, you'll probably have to write the code yourself, sine there is a constant lack of windows developers in the ruby community.

[shtirlic]

Found that on http://rubyforge.org/projects/win32utils/ some work done already, also for win32-process http://rubyforge.org/docman/view.php/85/707/README.html, but now for me it's too complicated.

[mscottford]

I have actually spent a lot of time on this problem, and I gave up after a lot of frustration. I thought I would share what I have learned in case anyone else wants to pick up where I left off. Be forewarned, the description is very long, but very detailed.

I don't think win32-process's implementation will not work. It's implementation does not result in semantics that are the same as unix fork. It does not create a copy of the current process, like unix for does. Instead in creates a separate ruby instance that executes FILE, but starts at the top of the script. Stubborn as I am, I tried anyway.

I discovered that win32-process's fork, tacks on a command line parameter that it uses to keep track of whether or not the process is the parent or the child. (The child is called with the parameter, but the parent was not.) Having this parameter on the command line throws off shotgun's command-line processing.

I hacked shotgun to detect if the extra parameter existed, removed it before the command parsing took place, and then added it back, so that the win32-process code could detect it.

With that solved, I then realized that I somehow needed to create a copy of the Rack listener instance from the original process. I think that doing so will require some hacking around in Rack, because it does not like to start up listening to the same port that another process is listening to.

I gave up on that implementation, and started doing more research into implementations of fork on Windows. My research led me to two possible candidates for a solution, either Cygwin's implementation or AT&T's uwin implementation. Since I was using MRI ruby, the trouble I ran into is that I was unable to build a version of either one that linked to the same version of the C runtime as MRI. If they don't agree, then runtime errors are thrown that the C runtime was loaded incorrectly. At least I think that is why it was crashing. I really have no idea.

Even if I did get it working, the process by which it works sounds pretty slow. To understand why we need to explore the differences in the implementations that are imposed by the differences between the underlying operating systems.

Unix fork is really fast because it leverages "copy on write" semantics. This enables the creation of a child process with a pointer the same memory space as the original. If the copy needs to make any changes, like changing the a variable stored in the heap, then only the changed memory is copied before the write is performed.

Because Windows does not have any built in way to implement copy on write semantics for child processes, my research has led me to believe that the cygwin and uwin implementations instead make a full copy of the executing process, thus causing a delay before the child process executes. The amount of delay would be relative to the size of the parent processes working set.

Here is a description of the implementations in more detail. (1) They first create a child process in a non-running state, (2) overwrite the child processes address space (heap and code) with the contents of the parent process, (3) insert code at the top of the child processes main function that causes it to instantly jump to the location of the call to fork and modify the child process's call to fork so that it calls a separate implementation that prevents an infinite loop and also returns the current value, (4) then they place the child process in a running state, (5) the process jumps to the line that calls fork, (6) the child process calls the alternative fork implementation. After going through all of that, the child process will behave with the same semantics of the original fork implementation as described in the fork man-page.

I think that the best course of action is to start with either the cygwin or uwin fork implementation. I would pull all of the fork logic into a separate dll, or convert the implementation into calls to the underlying Win32 functions via FFI. The later would make the implementation more portable across ruby runtimes and would avoid any inconsistencies between different versions of the C runtime. The only snag I see, is potential open source license conflicts. I'll leave figuring all of that out as an exercise for the reader.

Wow. That was a long braindump. I hope someone does something good with it, because I am unable to spend any more mental energy on this issue. Best of luck!

Updated to fix links.

[rtomayko]

Wow. Thanks for the detailed write up mscottford. Windows lack of POSIX fork(2) makes it really hard to take the shotgun approach to reloading. My feeling is that some other non-fork based solution needs to be figured out for Windows, and I think that should probably be a different tool.

[Wardrop]

I'm a bit of a "noob" when it comes to Ruby (only started learning it a few weeks ago), but how does the Rails server implementation work when it comes to reloading the application? It seems to be as a robust as shotgun in the sense that no matter what file is changed, that change is reflected on browser refresh, as opposed to sinatra/reloader which fails to update things such as filters. Doesn't Rails use Rack? If so, I'd be looking at what they've done to implement such similar functionality as Shotgun without the use of forks.

Sinatra/reloader is a pain as I often find myself having to quit and re-launch the app anyway in order for changes to be reflected. Rails doesn't seem to have this problem, hence my curiosity.

[rkh]

Wardrop: http://rkh.im/2010/08/code-reloading

Also, could you provide an example where you have to restart manually with sinatra-reloader? Maybe I can work on that.

[Wardrop]

Run this...

http://pastie.org/1226732

Then remove the "before" filter, and reload the page. You'll see that the content type being sent is still "application/octet-stream". Reloading in another browser produces the same result, hence I can only conclude that this is a problem or limitation of reloader. This does seem to work the other way around. Adding the "before" filter and then reloading the page makes the "before" filter take effect; it's the removing of the filter that seems to be the problem. It's just little things like this which may me like confidence in reloader, and hence I often end up relaunching the app so I can be sure what I'm seeing isn't just a problem with reloader.

I like shotgun because you know it's reloading the entire app every time, and that no matter what you change, it'll be reflected on reload. I can use shotgun on my mac, but on can't obviously on either my home work and windows machines.

[rkh]

Oh, yeah, the filter thing is an issue, but I'm planning to fix that. Another option would also be using Unicorn: http://namelessjon.posterous.com/magical-reloading-sparkles.