New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to java_import a class on jruby-9.0.0.0.rc1 #3062

Closed
bigsur0 opened this Issue Jun 17, 2015 · 15 comments

Comments

Projects
None yet
5 participants
@bigsur0
Contributor

bigsur0 commented Jun 17, 2015

java_import 'org.apache.hadoop.hbase.protobuf.ProtobufUtil' works fine on jruby-1.7.19 but not on jruby-9.0.0.0.rc1, could this be due to new class loading behavior?

NameError: cannot initialize Java class org.apache.hadoop.hbase.protobuf.ProtobufUtil
              for_name at org/jruby/javasupport/JavaClass.java:204
       get_proxy_class at org/jruby/javasupport/JavaUtilities.java:34
  block in java_import at uri:classloader:/jruby/java/core_ext/object.rb:47
                   map at org/jruby/RubyArray.java:2309
           java_import at uri:classloader:/jruby/java/core_ext/object.rb:34
                 <top> at ~/Developer/work/bar/my_proj/lib/my_proj/foo/seqfile_utils.rb:14
               require at org/jruby/RubyKernel.java:940
                 <top> at ~/Developer/work/bar/my_proj/foo/src/spec/ruby/integration/my_proj/foo/my_spec.rb:1
                  load at org/jruby/RubyKernel.java:958
       block in (root) at ~/Developer/work/bar/my_proj/foo/src/spec/ruby/integration/my_proj/foo/my_spec.rb:7
                  each at org/jruby/RubyArray.java:1571
                (root) at ~/.rvm/gems/jruby-9.0.0.0.rc1@my_proj/gems/rspec-core-2.14.5/lib/rspec/core/configuration.rb:1
       load_spec_files at ~/.rvm/gems/jruby-9.0.0.0.rc1@my_proj/gems/rspec-core-2.14.5/lib/rspec/core/configuration.rb:896
       load_spec_files at ~/.rvm/gems/jruby-9.0.0.0.rc1@my_proj/gems/rspec-core-2.14.5/lib/rspec/core/configuration.rb:896
          block in run at ~/.rvm/gems/jruby-9.0.0.0.rc1@my_proj/gems/rspec-core-2.14.5/lib/rspec/core/command_line.rb:22
@kares

This comment has been minimized.

Show comment
Hide comment
@kares

kares Jun 24, 2015

Member

@r6p could you check it's not ~ the same usage pattern as with #3036 (you're manually loading the .jar)
tried to explain some of whats going on #3036 (comment) (you should be able to confirm it's the same issue the same way - setting the context class-loader)

Member

kares commented Jun 24, 2015

@r6p could you check it's not ~ the same usage pattern as with #3036 (you're manually loading the .jar)
tried to explain some of whats going on #3036 (comment) (you should be able to confirm it's the same issue the same way - setting the context class-loader)

@bigsur0

This comment has been minimized.

Show comment
Hide comment
@bigsur0

bigsur0 Jul 5, 2015

Contributor

I get a slightly different message when seeing the context class-loader, but it still fails.

>> java.lang.Thread.currentThread.setContextClassLoader JRuby.runtime.jruby_class_loader
=> nil
>> java_import 'org.apache.hadoop.hbase.protobuf.ProtobufUtil'
NameError: cannot link Java class org.apache.hadoop.hbase.protobuf.ProtobufUtil, probable missing dependency: Could not initialize class org.apache.hadoop.hbase.protobuf.ProtobufUtil
    from org/jruby/javasupport/JavaClass.java:204:in `for_name'
    from org/jruby/javasupport/JavaUtilities.java:34:in `get_proxy_class'
    from uri:classloader:/jruby/java/core_ext/object.rb:47:in `block in java_import'
    from org/jruby/RubyArray.java:2309:in `map'
    from uri:classloader:/jruby/java/core_ext/object.rb:34:in `java_import'
    from (irb):3:in `<eval>'
    from org/jruby/RubyKernel.java:979:in `eval'
    from org/jruby/RubyKernel.java:1292:in `loop'
    from org/jruby/RubyKernel.java:1099:in `catch'
    from org/jruby/RubyKernel.java:1099:in `catch'
Contributor

bigsur0 commented Jul 5, 2015

I get a slightly different message when seeing the context class-loader, but it still fails.

>> java.lang.Thread.currentThread.setContextClassLoader JRuby.runtime.jruby_class_loader
=> nil
>> java_import 'org.apache.hadoop.hbase.protobuf.ProtobufUtil'
NameError: cannot link Java class org.apache.hadoop.hbase.protobuf.ProtobufUtil, probable missing dependency: Could not initialize class org.apache.hadoop.hbase.protobuf.ProtobufUtil
    from org/jruby/javasupport/JavaClass.java:204:in `for_name'
    from org/jruby/javasupport/JavaUtilities.java:34:in `get_proxy_class'
    from uri:classloader:/jruby/java/core_ext/object.rb:47:in `block in java_import'
    from org/jruby/RubyArray.java:2309:in `map'
    from uri:classloader:/jruby/java/core_ext/object.rb:34:in `java_import'
    from (irb):3:in `<eval>'
    from org/jruby/RubyKernel.java:979:in `eval'
    from org/jruby/RubyKernel.java:1292:in `loop'
    from org/jruby/RubyKernel.java:1099:in `catch'
    from org/jruby/RubyKernel.java:1099:in `catch'
@kares

This comment has been minimized.

Show comment
Hide comment
@kares

kares Jul 6, 2015

Member

in that case we would need a (simple) script to reproduce, including how you're loading the .jar files

Member

kares commented Jul 6, 2015

in that case we would need a (simple) script to reproduce, including how you're loading the .jar files

@bigsur0

This comment has been minimized.

Show comment
Hide comment
@bigsur0

bigsur0 Jul 6, 2015

Contributor
cmd: cat Jarfile
repository :first, 'https://repository.cloudera.com/artifactory/cloudera-repos/'
jar 'org.apache.hbase:hbase-client', '0.98.1-cdh5.1.0'
cmd: cat bootstrap.sh
chmod +x *.sh
gem install jbundler
cmd: cat java_import.rb
require 'jbundler' # JBundler establishes the classpath
java_import 'org.apache.hadoop.hbase.protobuf.ProtobufUtil'
cmd: cat using-jruby-1.7.19.sh
#!/bin/bash --login
rvm use jruby-1.7.19
jbundle install
chmod +x java_import.rb
ruby java_import.rb
cmd: cat reproduce-jruby-9.0.0.0.rc1.sh
#!/bin/bash --login
rvm use jruby-9.0.0.0.rc1
jbundle install
chmod +x java_import.rb
ruby java_import.rb
cmd: ./reproduce-jruby-9.0.0.0.rc1.sh
NameError: cannot initialize Java class org.apache.hadoop.hbase.protobuf.ProtobufUtil
              for_name at org/jruby/javasupport/JavaClass.java:204
       get_proxy_class at org/jruby/javasupport/JavaUtilities.java:34
  block in java_import at uri:classloader:/jruby/java/core_ext/object.rb:47
                   map at org/jruby/RubyArray.java:2309
           java_import at uri:classloader:/jruby/java/core_ext/object.rb:34
                 <top> at java_import.rb:2
Contributor

bigsur0 commented Jul 6, 2015

cmd: cat Jarfile
repository :first, 'https://repository.cloudera.com/artifactory/cloudera-repos/'
jar 'org.apache.hbase:hbase-client', '0.98.1-cdh5.1.0'
cmd: cat bootstrap.sh
chmod +x *.sh
gem install jbundler
cmd: cat java_import.rb
require 'jbundler' # JBundler establishes the classpath
java_import 'org.apache.hadoop.hbase.protobuf.ProtobufUtil'
cmd: cat using-jruby-1.7.19.sh
#!/bin/bash --login
rvm use jruby-1.7.19
jbundle install
chmod +x java_import.rb
ruby java_import.rb
cmd: cat reproduce-jruby-9.0.0.0.rc1.sh
#!/bin/bash --login
rvm use jruby-9.0.0.0.rc1
jbundle install
chmod +x java_import.rb
ruby java_import.rb
cmd: ./reproduce-jruby-9.0.0.0.rc1.sh
NameError: cannot initialize Java class org.apache.hadoop.hbase.protobuf.ProtobufUtil
              for_name at org/jruby/javasupport/JavaClass.java:204
       get_proxy_class at org/jruby/javasupport/JavaUtilities.java:34
  block in java_import at uri:classloader:/jruby/java/core_ext/object.rb:47
                   map at org/jruby/RubyArray.java:2309
           java_import at uri:classloader:/jruby/java/core_ext/object.rb:34
                 <top> at java_import.rb:2

@mkristian mkristian added the JRuby 9000 label Jul 6, 2015

@enebo enebo added this to the JRuby 9.0.0.0.rc2 milestone Jul 7, 2015

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Jul 7, 2015

Member

@r6p Thank you! I have reproduced locally.

Member

headius commented Jul 7, 2015

@r6p Thank you! I have reproduced locally.

@mkristian mkristian self-assigned this Jul 7, 2015

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Jul 7, 2015

Member

This may still be a JRuby bug, but it's starting to look like a bug in JBundler (probably some buggy interaction with JRuby 9k).

From what I can tell, jbundler simply isn't requiring all the jars in. I added logging to its main bit of logic, in JBundler.require_jars, and all the preconditions it checks fail. As a result, it quietly skips requiring the jars.

I think jbundler should probably hard error if it can't bootstrap the jars, or else give some indication that it didn't bootstrap right. I'm not sure why it's behaving this way on 9k but working on 1.7.

Reassigning to @mkristian for now since jbundler is his baby.

Member

headius commented Jul 7, 2015

This may still be a JRuby bug, but it's starting to look like a bug in JBundler (probably some buggy interaction with JRuby 9k).

From what I can tell, jbundler simply isn't requiring all the jars in. I added logging to its main bit of logic, in JBundler.require_jars, and all the preconditions it checks fail. As a result, it quietly skips requiring the jars.

I think jbundler should probably hard error if it can't bootstrap the jars, or else give some indication that it didn't bootstrap right. I'm not sure why it's behaving this way on 9k but working on 1.7.

Reassigning to @mkristian for now since jbundler is his baby.

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jul 8, 2015

Member

my fix for this:

--- java_import_orig.rb 2015-07-08 08:43:26.674348905 +0200
+++ java_import.rb  2015-07-08 08:41:52.258718796 +0200
@@ -1,2 +1,3 @@
+java.lang.Thread.current_thread.set_context_class_loader(JRuby.runtime.jruby_class_loader)
 require 'jbundler' # JBundler establishes the classpath
 java_import 'org.apache.hadoop.hbase.protobuf.ProtobufUtil'

which leads to

$ ruby java_import.rb 
15/07/08 08:45:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

@headius will be later on irc to dicuss this context-classloader problem

Member

mkristian commented Jul 8, 2015

my fix for this:

--- java_import_orig.rb 2015-07-08 08:43:26.674348905 +0200
+++ java_import.rb  2015-07-08 08:41:52.258718796 +0200
@@ -1,2 +1,3 @@
+java.lang.Thread.current_thread.set_context_class_loader(JRuby.runtime.jruby_class_loader)
 require 'jbundler' # JBundler establishes the classpath
 java_import 'org.apache.hadoop.hbase.protobuf.ProtobufUtil'

which leads to

$ ruby java_import.rb 
15/07/08 08:45:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

@headius will be later on irc to dicuss this context-classloader problem

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jul 8, 2015

Member

@r6p could you verify my fix on your side ? since you did try something similar before

Member

mkristian commented Jul 8, 2015

@r6p could you verify my fix on your side ? since you did try something similar before

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jul 8, 2015

Member

still investigating - since if I execute the code manually it will not raise the error but the moment the cinit of org.apache.hadoop.hbase.protobuf.ProtobufUtil runs the same code blows up

Member

mkristian commented Jul 8, 2015

still investigating - since if I execute the code manually it will not raise the error but the moment the cinit of org.apache.hadoop.hbase.protobuf.ProtobufUtil runs the same code blows up

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jul 8, 2015

Member

my mystery resolved: the code I used was from newer version of the libraries. jar 'org.apache.hbase:hbase-client', '1.0.0-cdh5.4.4' solves this problem but has other issues popping up.

the current JRuby situation is: whatever runs inside an OSGi container works with JRuby. if you need to set the context classloader in the OSGi case you also need to do this on JRuby

see also last code snippet here (karaf is meta osgi container or so) https://dzone.com/articles/apache-karaf-meets-apache

Member

mkristian commented Jul 8, 2015

my mystery resolved: the code I used was from newer version of the libraries. jar 'org.apache.hbase:hbase-client', '1.0.0-cdh5.4.4' solves this problem but has other issues popping up.

the current JRuby situation is: whatever runs inside an OSGi container works with JRuby. if you need to set the context classloader in the OSGi case you also need to do this on JRuby

see also last code snippet here (karaf is meta osgi container or so) https://dzone.com/articles/apache-karaf-meets-apache

@bigsur0

This comment has been minimized.

Show comment
Hide comment
@bigsur0

bigsur0 Jul 8, 2015

Contributor

I reproduced the problem using JBundler but it also occurs with LockJar. But using your code above, i.e java.lang.Thread.current_thread.set_context_class_loader(JRuby.runtime.jruby_class_loader), I can no longer reproduce.

I am interested to know how the class-loading issues you describe affect Jruby code that is running in a Java context, i.e. compiled with jrubyc.

Contributor

bigsur0 commented Jul 8, 2015

I reproduced the problem using JBundler but it also occurs with LockJar. But using your code above, i.e java.lang.Thread.current_thread.set_context_class_loader(JRuby.runtime.jruby_class_loader), I can no longer reproduce.

I am interested to know how the class-loading issues you describe affect Jruby code that is running in a Java context, i.e. compiled with jrubyc.

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jul 8, 2015

Member

so apparently when org.apache.hadoop.hbase.protobuf.ProtobufUtil gets loaded it loads a file 'hbase-default.xml' via the Thread.currentThread.contextClassLoader. this always depends on the java context and whether your application runs just from ClassLoader or from a ContextClassLoader.

for me the savest code would be:

ocl = java.lang.Thread.current_thread.context_class_loader
begin do
    java.lang.Thread.current_thread.set_context_class_loader(JRuby.runtime.jruby_class_loader)
    java_import 'org.apache.hadoop.hbase.protobuf.ProtobufUtil'
ensure
     java.lang.Thread.current_thread.context_class_loader = ocl
end

which should work for any java context you code runs in and independent to your jruby version - 1.7.x or 9k

Member

mkristian commented Jul 8, 2015

so apparently when org.apache.hadoop.hbase.protobuf.ProtobufUtil gets loaded it loads a file 'hbase-default.xml' via the Thread.currentThread.contextClassLoader. this always depends on the java context and whether your application runs just from ClassLoader or from a ContextClassLoader.

for me the savest code would be:

ocl = java.lang.Thread.current_thread.context_class_loader
begin do
    java.lang.Thread.current_thread.set_context_class_loader(JRuby.runtime.jruby_class_loader)
    java_import 'org.apache.hadoop.hbase.protobuf.ProtobufUtil'
ensure
     java.lang.Thread.current_thread.context_class_loader = ocl
end

which should work for any java context you code runs in and independent to your jruby version - 1.7.x or 9k

@bigsur0

This comment has been minimized.

Show comment
Hide comment
@bigsur0

bigsur0 Jul 8, 2015

Contributor

I would hate to have to pepper my code with this type of fallback class loader logic. I guess I'll have to try both scenarios to ensure when, java.lang.Thread.current_thread.set_context_class_loader(JRuby.runtime.jruby_class_loader), is truly needed. As it may only be needed when running from a context that is principally Jruby app based and not Java app based.

Contributor

bigsur0 commented Jul 8, 2015

I would hate to have to pepper my code with this type of fallback class loader logic. I guess I'll have to try both scenarios to ensure when, java.lang.Thread.current_thread.set_context_class_loader(JRuby.runtime.jruby_class_loader), is truly needed. As it may only be needed when running from a context that is principally Jruby app based and not Java app based.

@enebo enebo modified the milestones: JRuby 9.0.0.0.rc2, JRuby 9.0.0.0 Jul 9, 2015

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jul 16, 2015

Member

fixed via 7d3bc47

Member

mkristian commented Jul 16, 2015

fixed via 7d3bc47

@mkristian mkristian closed this Jul 16, 2015

@bigsur0

This comment has been minimized.

Show comment
Hide comment
@bigsur0

bigsur0 Jul 21, 2015

Contributor

Verified that with jruby-head from the evening of 7/17, that this is now fixed. Thanks @mkristian!

Contributor

bigsur0 commented Jul 21, 2015

Verified that with jruby-head from the evening of 7/17, that this is now fixed. Thanks @mkristian!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment