Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to java_import a class on jruby-9.0.0.0.rc1 #3062

Closed
bigsur0 opened this issue Jun 17, 2015 · 15 comments
Closed

Unable to java_import a class on jruby-9.0.0.0.rc1 #3062

bigsur0 opened this issue Jun 17, 2015 · 15 comments
Assignees
Labels
Milestone

Comments

@bigsur0
Copy link
Contributor

@bigsur0 bigsur0 commented Jun 17, 2015

java_import 'org.apache.hadoop.hbase.protobuf.ProtobufUtil' works fine on jruby-1.7.19 but not on jruby-9.0.0.0.rc1, could this be due to new class loading behavior?

NameError: cannot initialize Java class org.apache.hadoop.hbase.protobuf.ProtobufUtil
              for_name at org/jruby/javasupport/JavaClass.java:204
       get_proxy_class at org/jruby/javasupport/JavaUtilities.java:34
  block in java_import at uri:classloader:/jruby/java/core_ext/object.rb:47
                   map at org/jruby/RubyArray.java:2309
           java_import at uri:classloader:/jruby/java/core_ext/object.rb:34
                 <top> at ~/Developer/work/bar/my_proj/lib/my_proj/foo/seqfile_utils.rb:14
               require at org/jruby/RubyKernel.java:940
                 <top> at ~/Developer/work/bar/my_proj/foo/src/spec/ruby/integration/my_proj/foo/my_spec.rb:1
                  load at org/jruby/RubyKernel.java:958
       block in (root) at ~/Developer/work/bar/my_proj/foo/src/spec/ruby/integration/my_proj/foo/my_spec.rb:7
                  each at org/jruby/RubyArray.java:1571
                (root) at ~/.rvm/gems/jruby-9.0.0.0.rc1@my_proj/gems/rspec-core-2.14.5/lib/rspec/core/configuration.rb:1
       load_spec_files at ~/.rvm/gems/jruby-9.0.0.0.rc1@my_proj/gems/rspec-core-2.14.5/lib/rspec/core/configuration.rb:896
       load_spec_files at ~/.rvm/gems/jruby-9.0.0.0.rc1@my_proj/gems/rspec-core-2.14.5/lib/rspec/core/configuration.rb:896
          block in run at ~/.rvm/gems/jruby-9.0.0.0.rc1@my_proj/gems/rspec-core-2.14.5/lib/rspec/core/command_line.rb:22
@kares
Copy link
Member

@kares kares commented Jun 24, 2015

@r6p could you check it's not ~ the same usage pattern as with #3036 (you're manually loading the .jar)
tried to explain some of whats going on #3036 (comment) (you should be able to confirm it's the same issue the same way - setting the context class-loader)

@bigsur0
Copy link
Contributor Author

@bigsur0 bigsur0 commented Jul 5, 2015

I get a slightly different message when seeing the context class-loader, but it still fails.

>> java.lang.Thread.currentThread.setContextClassLoader JRuby.runtime.jruby_class_loader
=> nil
>> java_import 'org.apache.hadoop.hbase.protobuf.ProtobufUtil'
NameError: cannot link Java class org.apache.hadoop.hbase.protobuf.ProtobufUtil, probable missing dependency: Could not initialize class org.apache.hadoop.hbase.protobuf.ProtobufUtil
    from org/jruby/javasupport/JavaClass.java:204:in `for_name'
    from org/jruby/javasupport/JavaUtilities.java:34:in `get_proxy_class'
    from uri:classloader:/jruby/java/core_ext/object.rb:47:in `block in java_import'
    from org/jruby/RubyArray.java:2309:in `map'
    from uri:classloader:/jruby/java/core_ext/object.rb:34:in `java_import'
    from (irb):3:in `<eval>'
    from org/jruby/RubyKernel.java:979:in `eval'
    from org/jruby/RubyKernel.java:1292:in `loop'
    from org/jruby/RubyKernel.java:1099:in `catch'
    from org/jruby/RubyKernel.java:1099:in `catch'
@kares
Copy link
Member

@kares kares commented Jul 6, 2015

in that case we would need a (simple) script to reproduce, including how you're loading the .jar files

@bigsur0
Copy link
Contributor Author

@bigsur0 bigsur0 commented Jul 6, 2015

cmd: cat Jarfile
repository :first, 'https://repository.cloudera.com/artifactory/cloudera-repos/'
jar 'org.apache.hbase:hbase-client', '0.98.1-cdh5.1.0'
cmd: cat bootstrap.sh
chmod +x *.sh
gem install jbundler
cmd: cat java_import.rb
require 'jbundler' # JBundler establishes the classpath
java_import 'org.apache.hadoop.hbase.protobuf.ProtobufUtil'
cmd: cat using-jruby-1.7.19.sh
#!/bin/bash --login
rvm use jruby-1.7.19
jbundle install
chmod +x java_import.rb
ruby java_import.rb
cmd: cat reproduce-jruby-9.0.0.0.rc1.sh
#!/bin/bash --login
rvm use jruby-9.0.0.0.rc1
jbundle install
chmod +x java_import.rb
ruby java_import.rb
cmd: ./reproduce-jruby-9.0.0.0.rc1.sh
NameError: cannot initialize Java class org.apache.hadoop.hbase.protobuf.ProtobufUtil
              for_name at org/jruby/javasupport/JavaClass.java:204
       get_proxy_class at org/jruby/javasupport/JavaUtilities.java:34
  block in java_import at uri:classloader:/jruby/java/core_ext/object.rb:47
                   map at org/jruby/RubyArray.java:2309
           java_import at uri:classloader:/jruby/java/core_ext/object.rb:34
                 <top> at java_import.rb:2
@mkristian mkristian added the JRuby 9000 label Jul 6, 2015
@enebo enebo added this to the JRuby 9.0.0.0.rc2 milestone Jul 7, 2015
@headius
Copy link
Member

@headius headius commented Jul 7, 2015

@r6p Thank you! I have reproduced locally.

@mkristian mkristian self-assigned this Jul 7, 2015
@headius
Copy link
Member

@headius headius commented Jul 7, 2015

This may still be a JRuby bug, but it's starting to look like a bug in JBundler (probably some buggy interaction with JRuby 9k).

From what I can tell, jbundler simply isn't requiring all the jars in. I added logging to its main bit of logic, in JBundler.require_jars, and all the preconditions it checks fail. As a result, it quietly skips requiring the jars.

I think jbundler should probably hard error if it can't bootstrap the jars, or else give some indication that it didn't bootstrap right. I'm not sure why it's behaving this way on 9k but working on 1.7.

Reassigning to @mkristian for now since jbundler is his baby.

@mkristian
Copy link
Member

@mkristian mkristian commented Jul 8, 2015

my fix for this:

--- java_import_orig.rb 2015-07-08 08:43:26.674348905 +0200
+++ java_import.rb  2015-07-08 08:41:52.258718796 +0200
@@ -1,2 +1,3 @@
+java.lang.Thread.current_thread.set_context_class_loader(JRuby.runtime.jruby_class_loader)
 require 'jbundler' # JBundler establishes the classpath
 java_import 'org.apache.hadoop.hbase.protobuf.ProtobufUtil'

which leads to

$ ruby java_import.rb 
15/07/08 08:45:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

@headius will be later on irc to dicuss this context-classloader problem

@mkristian
Copy link
Member

@mkristian mkristian commented Jul 8, 2015

@r6p could you verify my fix on your side ? since you did try something similar before

@mkristian
Copy link
Member

@mkristian mkristian commented Jul 8, 2015

still investigating - since if I execute the code manually it will not raise the error but the moment the cinit of org.apache.hadoop.hbase.protobuf.ProtobufUtil runs the same code blows up

@mkristian
Copy link
Member

@mkristian mkristian commented Jul 8, 2015

my mystery resolved: the code I used was from newer version of the libraries. jar 'org.apache.hbase:hbase-client', '1.0.0-cdh5.4.4' solves this problem but has other issues popping up.

the current JRuby situation is: whatever runs inside an OSGi container works with JRuby. if you need to set the context classloader in the OSGi case you also need to do this on JRuby

see also last code snippet here (karaf is meta osgi container or so) https://dzone.com/articles/apache-karaf-meets-apache

@bigsur0
Copy link
Contributor Author

@bigsur0 bigsur0 commented Jul 8, 2015

I reproduced the problem using JBundler but it also occurs with LockJar. But using your code above, i.e java.lang.Thread.current_thread.set_context_class_loader(JRuby.runtime.jruby_class_loader), I can no longer reproduce.

I am interested to know how the class-loading issues you describe affect Jruby code that is running in a Java context, i.e. compiled with jrubyc.

@mkristian
Copy link
Member

@mkristian mkristian commented Jul 8, 2015

so apparently when org.apache.hadoop.hbase.protobuf.ProtobufUtil gets loaded it loads a file 'hbase-default.xml' via the Thread.currentThread.contextClassLoader. this always depends on the java context and whether your application runs just from ClassLoader or from a ContextClassLoader.

for me the savest code would be:

ocl = java.lang.Thread.current_thread.context_class_loader
begin do
    java.lang.Thread.current_thread.set_context_class_loader(JRuby.runtime.jruby_class_loader)
    java_import 'org.apache.hadoop.hbase.protobuf.ProtobufUtil'
ensure
     java.lang.Thread.current_thread.context_class_loader = ocl
end

which should work for any java context you code runs in and independent to your jruby version - 1.7.x or 9k

@bigsur0
Copy link
Contributor Author

@bigsur0 bigsur0 commented Jul 8, 2015

I would hate to have to pepper my code with this type of fallback class loader logic. I guess I'll have to try both scenarios to ensure when, java.lang.Thread.current_thread.set_context_class_loader(JRuby.runtime.jruby_class_loader), is truly needed. As it may only be needed when running from a context that is principally Jruby app based and not Java app based.

@enebo enebo modified the milestones: JRuby 9.0.0.0.rc2, JRuby 9.0.0.0 Jul 9, 2015
@mkristian
Copy link
Member

@mkristian mkristian commented Jul 16, 2015

fixed via 7d3bc47

@mkristian mkristian closed this Jul 16, 2015
@bigsur0
Copy link
Contributor Author

@bigsur0 bigsur0 commented Jul 21, 2015

Verified that with jruby-head from the evening of 7/17, that this is now fixed. Thanks @mkristian!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.