Problems with paths containing space characters #1248

Closed
marktriggs opened this Issue Nov 21, 2013 · 15 comments

Projects

None yet

7 participants

@marktriggs
Contributor

Hi there,

This is a bit of an essay, sorry. Writing this up in detail to get it straight in my head as much as anything :)

I've recently been having some problems running a JRuby webapp (ArchivesSpace) from within a directory whose path contains space characters. The sorts of errors I'm seeing are:

 # require "openssl" on a Linux machine
 LoadError: load error: openssl -- java.lang.NoClassDefFoundError: org/bouncycastle/asn1/DERBoolean
   require at org/jruby/RubyKernel.java:1082
   require at file:/mnt/ssd/archivesspace/dist with spaces/archivesspace/gems/gems/jruby-jars-1.7.8/lib/new/jruby-complete-1.7.6.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
    (root) at -e:1

 # on a Windows machine, during Rack app startup
 org.jruby.rack.RackInitializationException: unable to resolve type 'size_t'
         from jar:file:/C:/Users/westbrook/Desktop/archivesspace-20131031-2140%20(2)/archivesspace/gems/gems/jruby-jars-1.7.6/lib/jruby-stdlib-complete-1.7.6.jar!/META-INF/jruby.home/lib/ruby/shared/ffi/types.rb:55:in `find_type'
         from jar:file:/C:/Users/westbrook/Desktop/archivesspace-20131031-2140%20(2)/archivesspace/gems/gems/jruby-jars-1.7.6/lib/jruby-stdlib-complete-1.7.6.jar!/META-INF/jruby.home/lib/ruby/shared/ffi/library.rb:345:in `find_type'
         from jar:file:/C:/Users/westbrook/Desktop/archivesspace-20131031-2140%20(2)/archivesspace/gems/gems/jruby-jars-1.7.6/lib/jruby-stdlib-complete-1.7.6.jar!/META-INF/jruby.home/lib/ruby/shared/ffi/library.rb:154:in `attach_function'
         from org/jruby/RubyArray.java:2413:in `map'

After a bit of a deep dive into the JRuby internals, I think I've tracked down the cause of the issue and a potential fix.

The crux of the issue seems to be the way spaces in the $LOAD_PATH are handled by the LoadService class. If I start JRuby and require openssl, with the default $LOAD_PATH, there's no problem:

 $ java -jar jruby-complete-1.7.8.jar -e 'puts $LOAD_PATH.inspect; require "openssl"'
 ["file:/home/mst/tmp/path%20with%20spaces/jruby-complete-1.7.8.jar!/META-INF/jruby.home/lib/ruby/1.9/site_ruby",
  "file:/home/mst/tmp/path%20with%20spaces/jruby-complete-1.7.8.jar!/META-INF/jruby.home/lib/ruby/shared",
  "file:/home/mst/tmp/path%20with%20spaces/jruby-complete-1.7.8.jar!/META-INF/jruby.home/lib/ruby/1.9"]                                                       

(note that the default $LOAD_PATH has the spaces encoded)

But when I look at $LOAD_PATH in my webapp (running a warbler war file with jruby-rack), the spaces aren't URL encoded. If I simulate that in my test I see the same sorts of errors I've been seeing from ArchivesSpace:

 $ java -jar jruby-complete-1.7.8.jar -e 'orig = $LOAD_PATH.clone; $LOAD_PATH.clear; orig.each {|s| $LOAD_PATH << s.gsub("%20", " ")}; require "openssl"; puts "hooray"'
 LoadError: load error: jopenssl/load -- java.lang.NoClassDefFoundError: org/bouncycastle/asn1/DERBoolean
   require at org/jruby/RubyKernel.java:1084
   require at file:/home/mst/tmp/path with spaces/jruby-complete-1.7.8.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55                                                                                                    

So there seems to be a difference in the way the standard JRuby launcher sets the $LOAD_PATH and the way jruby-rack does it (the former encodes spaces, the latter doesn't).

That NoClassDefFoundError is thrown because bouncy-castle-java.rb hasn't been able to load the bouncy castle .jar files. It loads them like this:

 Dir[File.expand_path('bc*.jar', File.dirname(__FILE__))].each do |file|
   require File.basename(file)
 end

If __FILE__ contains %20 escapes, File.expand_path yields no matches and nothing gets loaded (and we see the NoClassDefFoundError). If it contains the literal space characters, all is well. But this is a bit confusing, because it's turned around:

  • If $LOAD_PATH contains encoded spaces, __FILE__ does not and everything works. That's the standard JRuby launcher case.
  • If $LOAD_PATH contains literal spaces, __FILE__ contains encoded spaces. That's what I see with jruby-rack running my webapp.

So if $LOAD_PATH is escaped, __FILE__ will be unescaped, and vice versa.

All of this finally brings me to LoadService.java, which I think is where the problem starts. The findFileForLoad method tries a bunch of different strategies for finding a given file, using several LoadSearcher instances to search the filesystem, classpath, etc.. When a file is found successfully, the LoadSearcher instance loads the resource and assigns its filename (which __FILE__ is derived from).

What I'm seeing is different LoadSearcher classes being hit depending on whether or not the $LOAD_PATH contains encoded spaces.

In this example:

 $ java -jar jruby-complete-1.7.8.jar -e 'puts $LOAD_PATH.inspect; require "openssl"'

the LoadService class tries to locate openssl in:

 file:/home/mst/tmp/path%20with%20spaces/jruby-complete-1.7.8.jar`.

First the NormalSearcher class tries to find the file on the filesystem by plucking out the path, but it gets a FileNotFoundException because it doesn't decode the '%20' characters before checking, and they're not present on the filesystem. That causes NormalSearcher to miss this resource, and the ClassLoaderSearcher (next in the list of search strategies) is what finds it. ClassLoaderSearcher returns the decoded version of the path, so __FILE__ winds up with literal space characters.

But in this example (analogous to what jruby-rack is doing):

 $ java -jar jruby-complete-1.7.8.jar -e 'orig = $LOAD_PATH.clone; $LOAD_PATH.clear; orig.each {|s| $LOAD_PATH << s.gsub("%20", " ")}; require "openssl"; puts "hooray"'

the NormalSearcher class doesn't miss, since it's looking in:

 file:/home/mst/tmp/path with spaces/jruby-complete-1.7.8.jar

and that file exists on the filesystem. Unlike ClassLoaderSearcher, this returns the encoded version of the path, so __FILE__ winds up with encoded spaces.

So that's where the difference lies: NormalSearcher returns encoded paths, while ClassLoaderSearcher returns unencoded paths. And which one you hit depends on whether your original $LOAD_PATH was encoded or not.

So after all of that, all of my problems went away when I modified NormalSearcher to decode its paths before returning them. That still leaves the slightly weird behaviour of paths with spaces and paths without being handled by different LoadSearcher instances, but I thought it best to make as small a chance as possible.

Here's the patch for my fix: https://gist.github.com/marktriggs/7575352, and please let me know if I can do any further testing.

Thanks,

Mark

@marktriggs
Contributor

Oops, forgot to mention, this may relate to http://jira.codehaus.org/browse/JRUBY-7159

@timhayman

I've encountered this problem with war files. In attempting to isolate the problem, I have a one line command that fails, but unlike JRUBY-7159, this one liner still fails with jruby 1.7.8:

java -Xmx512m -cp "d:/jruby 178/jruby-core-complete-1.7.8.jar;d:/jruby 178/jruby-stdlib-complete-1.7.8.jar" org.jruby.Main -e "require 'yaml'"

NameError: method 'to_yaml' not defined in Object
remove_method at org/jruby/RubyModule.java:2340
(eval) at file:/D:/jruby 178/jruby-stdlib-complete-1.7.8.jar!/META-INF/jruby.home/lib/ruby/1.9/yaml.rb:47
module_eval at org/jruby/RubyModule.java:2312
yamler= at file:/D:/jruby 178/jruby-stdlib-complete-1.7.8.jar!/META-INF/jruby.home/lib/ruby/1.9/yaml.rb:43
(root) at file:/D:/jruby 178/jruby-stdlib-complete-1.7.8.jar!/META-INF/jruby.home/lib/ruby/1.9/yaml.rb:82
require at org/jruby/RubyKernel.java:1084
(root) at file:/D:/jruby 178/jruby-stdlib-complete-1.7.8.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1
(root) at -e:1

Note the spaces in the directory name. Without the spaces, this command works fine.

@marktriggs
Contributor

Hi Tim,

That's an interesting one. That seems to relate more to the inconsistency I mentioned in my original message--that a .jar file with spaces in its path is loaded by ClassLoaderSearcher, while others are loaded by NormalSearcher.

If I patch LoadService$NormalSearcher to cope with encoded characters, your problem goes away too:

https://gist.github.com/marktriggs/7608007

So maybe both patches are needed after all: the first to stop NormalSearcher producing filenames with encoded characters in them, and the second to allow it to handle paths with encoded URLs in them.

I've run the 'mri19' test suite against both patches and everything passes, so hopefully I'm not too far off the mark :)

@mjansing

Got the same Problem. I would appreciate a solution.

@scverano

Both are using jRuby version 1.7.3 and jruby-openssl version 0.9.4

WORKING VERSION PATH WITH SPACE

I tried to downgrade from jRuby 1.7.10 to 1.7.3 and the path with space is working fine. I think there is an issue with the jruby-jars 1.7.4 to the current one (1.7.10). But in jRuby version 1.7.4 - 1.7.10 it is not working if the path contains space.

SUCCESS

NOT WORKING VERSION PATH WITH SPACE

When the path contains space the application generates this errors. If using the jruby-openssl version 0.9.4 but if the version is "~> 0.8.2" it is still working even-though the path contains space.

FAILED

I'am hoping this will be fixed soon.

@klinden
klinden commented Feb 3, 2014

@marktriggs have you been able to integrate your patches? Would love to see this get in the next release. I've tried building your patches, but have been getting issues with Maven.

@marktriggs
Contributor

@klinden I've just tried building my patches against the 1.7 branch and they're still applying and building cleanly. Here are the commands I used:

 # Using mvn 3.0.4, for what that's worth
 git checkout -t origin/jruby-1_7
 curl -s https://gist.github.com/marktriggs/7575352/raw | patch -p1
 curl -s https://gist.github.com/marktriggs/7608007/raw | patch -p1
 mvn -Pjruby-jars clean
 mvn -Pjruby-jars

Maven had a bit of a "download the Internet" moment, and after a while produced the files:

 maven/jruby-core-complete/target/jruby-core-complete-1.7.11-SNAPSHOT.jar
 maven/jruby-stdlib-complete/target/jruby-stdlib-complete-1.7.11-SNAPSHOT.jar
@klinden
klinden commented Feb 3, 2014

@marktriggs I have Maven 3.1.1 and the JRuby Integration Tests step is failing "Failed to execute goal de.saumya:mojo:gem-maven-plugin:1.0.0-rc2-initialize (default) on project jruby-tests..."

@mkristian
Member

@klinden are you running it on Windows ? there is more to the error you get from maven (I hope so) - could make a gist out of it ?

@marktriggs there is an IT for jruby-complete.jar which produces following:

$ java -jar jruby-complete-1.7.11-SNAPSHOT.jar -e 'puts $LOAD_PATH.inspect; require "openssl"'
["file:/home/christian/projects/active/maven/jruby/maven/jruby-complete/target/it/hello%20world/jruby-complete-1.7.11-SNAPSHOT.jar!/META-INF/jruby.home/lib/ruby/2.1/site_ruby", "file:/home/christian/projects/active/maven/jruby/maven/jruby-complete/target/it/hello%20world/jruby-complete-1.7.11-SNAPSHOT.jar!/META-INF/jruby.home/lib/ruby/shared", "file:/home/christian/projects/active/maven/jruby/maven/jruby-complete/target/it/hello%20world/jruby-complete-1.7.11-SNAPSHOT.jar!/META-INF/jruby.home/lib/ruby/2.1"]

even the jruby-complete-1.7.8,jar gives the same result - so I am not sure what is different with you and me

@mkristian
Member

@marktriggs it could be a Windows thing ? could you convert your patches to PR ?

even #1393 is a related to Windows

@marktriggs
Contributor

@mkristian I don't think it's Windows specific, but for my project (ArchivesSpace) we've seen Windows users hit this issue the most often, because spaces in pathnames are more common.

The example you showed works for me too. I think the key thing is that spaces in your $LOAD_PATH have been encoded as '%20', so everything works. But when running a warbler-built .war file under jruby-rack, the $LOAD_PATH ends up with paths containing spaces, and the class loader searcher can't cope with that. This line reproduces what I see when that happens:

 $ mkdir "path with spaces"
 $ cd "path with spaces"
 $ wget http://jruby.org.s3.amazonaws.com/downloads/1.7.10/jruby-complete-1.7.10.jar
 $ java -jar jruby-complete-1.7.10.jar -e 'orig = $LOAD_PATH.clone; $LOAD_PATH.clear; orig.each {|s| $LOAD_PATH << s.gsub("%20", " ")}; require "openssl"; puts "hooray"'

I'll convert to a PR and send that along. Is a PR against the 1_7 branch OK? Or should I rework to apply against master?

Thanks!
Mark

@mkristian
Member

@marktriggs ok _ I got it. will use you LOAD_PATH example for a IT on jruby-complete then. PR against 1_7 is OK. I will take care of master the IT. thanx

@klinden
klinden commented Feb 3, 2014

@mkristian Here's the gist for the rest of the Maven error: https://gist.github.com/klinden/8792910

@mkristian
Member

@klinden hmm - not much. anyway I still assume you have windows. so please remove the launcher gem dependency from test/pom.xml or install a C compiler which can compile the launcher. but I am just guessing right now.

@marktriggs marktriggs added a commit to marktriggs/jruby that referenced this issue Feb 4, 2014
@marktriggs marktriggs Bugfix for issue #1248
Change LoadService to cope with classpath entries containing spaces.  This
doesn't normally happen when JRuby is invoked from the command-line, but seems
to happen when running via jruby-rack.
ebd2d8f
@mkristian
Member

thanx

@mkristian mkristian closed this Feb 4, 2014
@mkristian mkristian added a commit that referenced this issue Feb 4, 2014
@marktriggs @mkristian marktriggs + mkristian Bugfix for issue #1248
Change LoadService to cope with classpath entries containing spaces.  This
doesn't normally happen when JRuby is invoked from the command-line, but seems
to happen when running via jruby-rack.
26f4885
@enebo enebo added this to the JRuby 1.7.11 milestone Feb 21, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment