Broken 1.9-mode deploys due to load path modifications #89
Comments
ghost
assigned 
|
I tried backing off jruby-rack versions in small steps as far back as 1.0.10 and still no sign of to_yaml. Removing 1.9 syntax from my code and deploying without the 1.9 flags has resolved my immediate deploy issues. So deploy works fine in ruby 1.8 mode and doesn't work at all in ruby 1.9 mode. |
|
this conversation from #jruby today may also be helpful: 18:17:18 dobbs: so perhaps this never worked with 1.9 mode |
|
I explicitly put "require 'yaml.rb'" in my boot.rb .. It seemed to work on my 3.2 app in 1.9 mode. |
|
nice one ! @headius I'm not sure I understand why's there : LoadService: trying resourceFromJarURLWithLoadPath: /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/jruby-jars-1.6.5.1/lib/jruby-stdlib-1.6.5.1.jar!/META-INF/jruby.home/lib/ruby/site_ruby/1.9/enumerator.rb but then for psych.rb there's "only" : LoadService: trying resourceFromLoadPath: '/Users/headius/projects/jruby/debug/war2/WEB-INF/tmp/classpath:META-INF/jruby.home/lib/ruby/site_ruby/1.9/psych.rb' isn't the load service deterministically trying the same path for each .rb ? @trajar I'm guessing you've put that in rails/boot.rb also are you deploying as a .war ? I'm sorry I only had a quick look and try, unfortunately just about when this got reported I got sick :( |
|
Yeah I'm deploying inside a war (actually technically custom war-based OSGi container). I had to do the same thing for 'digest.rb' in order to get all the MD5 goodness. I spend an afternoon trying to figure it out, and it seems that if you are loading something that might be custom jar or jruby library, that gets preference, and the .rb gets skipped. |
|
confirming that putting a |
|
after more digging in - I debugged the load service while deploying a .war in 1.9 mode onto JBoss - I am starting to think the most elegant solution, that would avoid similar "surprises" for the the future as well, would be to adjust the load path on JRuby's side (when dealing with classpath:/ entries). first of all let's clarify the title: there's no $LOAD_PATH-modifications happening on our side, in this case ! we would modify the $LOAD_PATH if jruby.home == /tmp since than it would most likely contain non-sense such as : but this is not the problem here ... I'm certainly no expert on what edge cases the Recall the or even better the if the resourceFromLoadPath was smart enough to detect a path entry starting with classpath:/ instead of expanding it from CWD it would certainly be for good and solve this issue, I even think it would do some minor performance boost for cases like these. But most importantly if say I append Solving this on our side seems to be only possible with a $LOAD_PATH modification hack : But since this seems to be only necessary for a few jar conflicting cases so far (yaml and digest) I would rather do a workaround of preloading these libraries with Any suggestions are welcome ... |
|
I think you may be right about modifying JRuby to recognize load path entries starting with classpath:/ and resolve them as normal classloader resources. I'll look into that today. |
|
Here's a first stab at a patch: https://gist.github.com/1760124 I've confirmed that this resolves issues at the command line when the load path contains only classpath:/ entries: before: after: And a similar result for the psych libraries. I'm not sure where the classpath:/ entries in load path are coming from, however. They appear to be built up like normal, with site_lib and 1.9 entries, which makes me think somewhere jruby.home is getting set up as a classpath:/ path, like classpath:/META-INF/jruby.home. I'd like to know exactly how that's happening, but this patch appears to make JRuby honor classpath:/ load path entries correctly. |
|
@headius looks great but since I commented I was thinking - what if the path does not have the "classpath:" prefix |
|
Hmm yes, that could be. I thought it should be looking for classloader resources before builtins, but perhaps I'm mistaken. |
|
we might need to do something like this https://gist.github.com/1760422 (based on jruby-1_6_5) |
|
Ah yes. So that code is part of findLibraryWithoutCWD, called from NormalSearcher which is the first real search done. And I remember why we did it this way: because that's the order MRI seemed to find extensions at some point in the past. However, I think MRI's ordering can be explained by looking at its load path: Each normal lib entry (1.9.1 dirs) is followed by a platform-specific dir. When loading psych, the .bundle (OS X) comes from one of these: So our searching for builtins first does not quite match MRI. We should search builtins as though they're a load path entry after each source-related entry. For 1.6.7 we want a minimally-invasive patch, though. I wonder if my patch above is good enough for 1.6.7? |
|
Ok, so here's an updated summary. MRI sets up its load path so that its own source and ext dirs are at the front of the load path. This allows it to ensure the "built in" libraries get a chance to load before user-added load path entries. Our case is set up a bit differently. Since there's no "load path" entry for built in extensions (they're just special names we know are part of JRuby), we check for built-in extensions before searching load path. Now the obvious idea would be to move built-in ext searching after load path searching, but this is not right for a couple reasons:
The reason it doesn't solve this bug is because our classloader searching always comes after normal filesystem-based load path searching. As a result, the examples in this bug (psych and digest) end up finding the ".jar" built-in ext before ever trying to search classloader. This is further aggravated because under some deployments, the load path is getting filled with classpath:/ entries instead of filesystem entries, since there's no searchable filesystem paths. In that case, built-in exts end up getting found before any .rb files from load path, which ultimately causes psych.jar and digest.jar exts to load instead of psych.rb and digest.rb. We might be able to fix this by moving our built-in ext search after classloader searching, but that has even more chance of loading a user-provided ".jar" instead of our built-in ext ".jar". I'm going to ponder this a bit more and try a few changes. |
|
FWIW, I think my fix to make classpath:/ load path entries work properly does represent a good fix, because:
@kares: I think your point about bare classloader resource loads is valid, but perhaps the real solution is for us to push for classloader resources to either be on load path (with classpath:/ entries) or to have a full classpath:/ URL for there resource itself. That would allow us to always recognize that it's supposed to come from classloader, rather than hoping the classloader phase will get lucky and pick it up. |
|
Ok, I think I know where the classpath: load path entries come from now too. If no jruby.home is set, the following code is eventually called in JRuby to determine it (called from RubyInstanceConfig.calculateJRubyHome): The logic here basically looks for META-INF/jruby.home as a classloader resource. If it finds it, it checks the format of the URL. If the URL does not start with jar:file:, it ends up using classpath:/META-INF/jruby.home as jruby.home, and then builds up load paths from there. So there's a couple possibilities here:
Looking at jruby-rack, I can only see a few places where it attempts to set up jruby.home:
My current theory is that jruby.home is not being set, and for whatever reason, the URL provided for the "META-INF/jruby.home" resource resolves to a URL that does not start with jar:file:. If it resolved to a jar:file: URL, then we'd add jar:file: URLs to load path which are currently handled right. In this case, however, we add classpath: entries, which were not handled properly. This again leads me to conclude that my fix is probably good enough for JRuby 1.6.7, since either jar:file URLs or classpath: URLs in load path will resolve properly. There is still one mystery for me: why this only fails to work in 1.9 mode. |
|
my attempt was a quick shoot - I sure was aware that I probably broke something by changing the loading order. your fix seems perfectly valid and would solve the |
|
Ok, I will mark this closed. I have pushed the classpath: fix to JRuby master and 1.6 branches, and kicked off a dist build of 1.6. It should be possible to use that to test jruby-rack for deployment. |
|
@headius sorry for the delay, I've verified this to be working with jruby-1.6.7.dev gem on JBoss 7.1, Tomcat 6.0 as well as WebLogic 10.3.5 all of which seems to be using classpath: entries by default. unfortunately it's still the same failure on WebSphere but that seems to be caused by an incorrect jruby.home detection and I'll try to modify those $LOAD_PATH entries to be cp prefixed as well ... thanks for your help ! |
…ers where jruby.home is detected as /tmp - verified that this works in 1.9 mode on WebSphere 8 with the JRuby 1.6.7 LoadService fix (related to jruby#89)
|
Thanks so much for looking into this, especially since it wasn't ultimately a jruby-rack bug! If there's anything else you think we need to do in JRuby for e.g. WebSphere, let me know ASAP. |
|
NP, well there's the "obvious" better jruby.home detection - I've just tried shooting some alternative |
There appears to be a problem with recent jruby-rack that's causing a complete failure to deploy 1.9-mode applications. The full output is here: https://gist.github.com/1710589
The problem happens during boot, when the 'yaml' library is loaded. It attempts to require 'psych', which should load lib/ruby/1.9/psych.rb and related files. However, because of an apparent load path misconfiguration, it only finds the internal 'psych.jar' extension (which doesn't exist as a jar...we just use that name...it's PsychLibrary in JRuby).
I added some writes to a file to show loaded features and load path at the point of failure (inside yaml.rb). At that point, loaded features ($") contains psych.jar but none of the psych .rb files, which define the "to_yaml" method in question. The load path contains the following four stdlib-related entries:
I believe this is the source of the problem. These paths do not appear to be handled as expected in JRuby. If I make a similar modification at the command line, I get a similar failure:
Notice that the classpath: paths are getting prefixed with my current dir, which makes them fail to resolve any of the jarred-up files.
Reproducing this issue should be easy. I tested on Glassfish, and a user that initiated the JRuby bug (http://jira.codehaus.org/browse/JRUBY-6397) is using JBoss.
The app will fail to boot with the error reported here.
This is a pretty critical issue, since it means nobody can deploy WARed up applications in 1.9 mode right now. I'm assigning @kares to look at it since he's had most of the recent commits, some of them load path-related.
Help me Karol Bucek...you're my only hope!
The text was updated successfully, but these errors were encountered: