New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Since JRuby-9.0.0.0 changes to classloaders are failing ruby-processing #3152

Closed
monkstone opened this Issue Jul 19, 2015 · 14 comments

Comments

Projects
None yet
3 participants
@monkstone
Contributor

monkstone commented Jul 19, 2015

JRuby is losing the magic that let ruby-processing just work:-
To make ruby-processing work it is important that ruby and java classes can be added somewhat dynamically (including those with native binaries), the recent changes to the class loaders in JRuby-9.0.0.0 are making it all but impossible to combine say video libraries (incorporating gstreamer and the like) and graphics (OPENGL). In some ways this is a continuation the issue that I reported earlier #3036 which can be fixed for simple cases by loading the jars onto the java classpath. Now if there were a new way of doing this, but incompatible with the old way I would be very interested as I am trying to create a new version of ruby-processing JRubyArt. Here is what I currently have to do add video library to JRubyArt (this is hard coded in the app and OK for core libraries.

    def video
      Dir["#{Processing::RP_CONFIG['sketchbook_path']}/libraries/video/library/\*.jar"]
    end

previously with ruby-processing we were able to add the jars from ruby and in our library _loader used the following hack to load the native binaries (even when not in a jar)

    # HACK: For pure java libraries, such as the ones that are available
    # on this page: http://processing.org/reference/libraries/index.html
    # that include native code, we mess with the 'Java ClassLoader', so that
    # you don't have to futz with your PATH. But it's probably bad juju.
    def load_java_library(library_name)
      library_name = library_name.to_sym
      return true if @loaded_libraries.include?(library_name)
      jpath = get_library_directory_path(library_name, 'jar')
      jars = get_library_paths(library_name, 'jar')
      return false if jars.empty?
      jars.each { |jar| require jar }
      platform_specific_library_paths = get_platform_specific_library_paths(jpath)
      platform_specific_library_paths = platform_specific_library_paths.select do |ppath|
        FileTest.directory?(ppath) && !Dir.glob(File.join(ppath, '*.{so,dll,jnilib}')).empty?
      end
      unless platform_specific_library_paths.empty?
        platform_specific_library_paths << java.lang.System.getProperty('java.library.path')
        new_library_path = platform_specific_library_paths.join(java.io.File.pathSeparator)
        java.lang.System.setProperty('java.library.path', new_library_path)
        field = java.lang.Class.for_name('java.lang.ClassLoader').get_declared_field('sys_paths')
        if field
          field.accessible = true
          field.set(java.lang.Class.for_name('java.lang.System').get_class_loader, nil)
        end
      end
      @loaded_libraries[library_name] = true
    end
@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jul 19, 2015

Member

could you be more specific, please.

version 9.0.0.0.rc2 did revert the changes regarding the CLI interface of jruby putting its dynamic classloader into Thread.currentThread.contextClassLoader.

BTW jruby-9k is not 2.2 compliant

Member

mkristian commented Jul 19, 2015

could you be more specific, please.

version 9.0.0.0.rc2 did revert the changes regarding the CLI interface of jruby putting its dynamic classloader into Thread.currentThread.contextClassLoader.

BTW jruby-9k is not 2.2 compliant

@monkstone

This comment has been minimized.

Show comment
Hide comment
@monkstone

monkstone Jul 19, 2015

Contributor

I think my updating overlapped your response. With ruby-processing I still need load to natives to classpath (with latest SNAPSHOT), which I found conflicts with previous loading (jruby-1.7.**). So I did a hack to only load binaries that way for jruby-9.0.0.0, but this leaves me with incompatability. I'm not worried about 2.2 compliance nice to use some 2.1 stuff though.

Contributor

monkstone commented Jul 19, 2015

I think my updating overlapped your response. With ruby-processing I still need load to natives to classpath (with latest SNAPSHOT), which I found conflicts with previous loading (jruby-1.7.**). So I did a hack to only load binaries that way for jruby-9.0.0.0, but this leaves me with incompatability. I'm not worried about 2.2 compliance nice to use some 2.1 stuff though.

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jul 19, 2015

Member

could you point me to the hack you are using for loading binaries with jruby-9k ?

Member

mkristian commented Jul 19, 2015

could you point me to the hack you are using for loading binaries with jruby-9k ?

@monkstone

This comment has been minimized.

Show comment
Hide comment
@monkstone

monkstone Jul 19, 2015

Contributor

Here is is the hack see line 197 for my crude detection of jruby version (note we could arrive here from mri ruby and mostly do).

Contributor

monkstone commented Jul 19, 2015

Here is is the hack see line 197 for my crude detection of jruby version (note we could arrive here from mri ruby and mostly do).

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jul 19, 2015

Member

thanx. is this the piece of code which works with jruby-1.7.* but not with jruby-9k https://github.com/jashkenas/ruby-processing/blob/master/lib/ruby-processing/library_loader.rb#L68
?

or is it https://github.com/jashkenas/ruby-processing/blob/master/lib/ruby-processing/library_loader.rb#L62 ?

any sample jar or so file which I can use for an isolated test ?

Member

mkristian commented Jul 19, 2015

thanx. is this the piece of code which works with jruby-1.7.* but not with jruby-9k https://github.com/jashkenas/ruby-processing/blob/master/lib/ruby-processing/library_loader.rb#L68
?

or is it https://github.com/jashkenas/ruby-processing/blob/master/lib/ruby-processing/library_loader.rb#L62 ?

any sample jar or so file which I can use for an isolated test ?

@monkstone

This comment has been minimized.

Show comment
Hide comment
@monkstone

monkstone Jul 19, 2015

Contributor

Oh my god I thought I was going mad for bit, lots of things working that previously had not, including video capture, so thumbs up for reversal of classloader. In summary the major problem now appears to be the need to load the native OPENGL binaries into the java classpath for jruby-9.0.0.0 (this is needed for any sketch using processing P2D or P3D modes but not the default java_mode) so any test will need to emulate a P2D or P3D sketch. Simple P2D and P3D sketches also run OK. However when I try and include my arcball library for example (a jruby extension), then we get this failure:-

Java::JavaLang::RuntimeException
org.jruby.exceptions.RaiseException: (NameError) cannot load Java class processing.arcball.Arcball
    com.jogamp.common.util.awt.AWTEDTExecutor.invoke(AWTEDTExecutor.java:58)
    jogamp.opengl.awt.AWTThreadingPlugin.invokeOnOpenGLThread(AWTThreadingPlugin.java:103)
    jogamp.opengl.ThreadingImpl.invokeOnOpenGLThread(ThreadingImpl.java:206)
    javax.media.opengl.Threading.invokeOnOpenGLThread(Threading.java:172)
    javax.media.opengl.Threading.invoke(Threading.java:191)
    javax.media.opengl.awt.GLCanvas.display(GLCanvas.java:541)
    processing.opengl.PJOGL.requestDraw(PJOGL.java:688)
    processing.opengl.PGraphicsOpenGL.requestDraw(PGraphicsOpenGL.java:1651)
    processing.core.PApplet.run(PApplet.java:2256)
    java.lang.Thread.run(Thread.java:745)

Or say for example we try and run a video capture sketch with P2D (OPENGL rendering)
and we get this:-

Java::JavaLang::RuntimeException
org.jruby.exceptions.RaiseException: (NameError) Capture not found in packages processing.video; last error: cannot link Java class processing.video.Capture, probable missing dependency: org/gstreamer/elements/BufferDataAppSink$Listener
    com.jogamp.common.util.awt.AWTEDTExecutor.invoke(AWTEDTExecutor.java:58)
    jogamp.opengl.awt.AWTThreadingPlugin.invokeOnOpenGLThread(AWTThreadingPlugin.java:103)
    jogamp.opengl.ThreadingImpl.invokeOnOpenGLThread(ThreadingImpl.java:206)
    javax.media.opengl.Threading.invokeOnOpenGLThread(Threading.java:172)
    javax.media.opengl.Threading.invoke(Threading.java:191)
    javax.media.opengl.awt.GLCanvas.display(GLCanvas.java:541)
    processing.opengl.PJOGL.requestDraw(PJOGL.java:688)
    processing.opengl.PGraphicsOpenGL.requestDraw(PGraphicsOpenGL.java:1651)
    processing.core.PApplet.run(PApplet.java:2256)
    java.lang.Thread.run(Thread.java:745)

the same sketch works fine and dandy if we use Java2D rendering, that is OK for this sketch but won't allow us to do GLSL filtering for example. Problem is creating a suitable test, although I think possibly related errors have been reported for Swing apps?

Contributor

monkstone commented Jul 19, 2015

Oh my god I thought I was going mad for bit, lots of things working that previously had not, including video capture, so thumbs up for reversal of classloader. In summary the major problem now appears to be the need to load the native OPENGL binaries into the java classpath for jruby-9.0.0.0 (this is needed for any sketch using processing P2D or P3D modes but not the default java_mode) so any test will need to emulate a P2D or P3D sketch. Simple P2D and P3D sketches also run OK. However when I try and include my arcball library for example (a jruby extension), then we get this failure:-

Java::JavaLang::RuntimeException
org.jruby.exceptions.RaiseException: (NameError) cannot load Java class processing.arcball.Arcball
    com.jogamp.common.util.awt.AWTEDTExecutor.invoke(AWTEDTExecutor.java:58)
    jogamp.opengl.awt.AWTThreadingPlugin.invokeOnOpenGLThread(AWTThreadingPlugin.java:103)
    jogamp.opengl.ThreadingImpl.invokeOnOpenGLThread(ThreadingImpl.java:206)
    javax.media.opengl.Threading.invokeOnOpenGLThread(Threading.java:172)
    javax.media.opengl.Threading.invoke(Threading.java:191)
    javax.media.opengl.awt.GLCanvas.display(GLCanvas.java:541)
    processing.opengl.PJOGL.requestDraw(PJOGL.java:688)
    processing.opengl.PGraphicsOpenGL.requestDraw(PGraphicsOpenGL.java:1651)
    processing.core.PApplet.run(PApplet.java:2256)
    java.lang.Thread.run(Thread.java:745)

Or say for example we try and run a video capture sketch with P2D (OPENGL rendering)
and we get this:-

Java::JavaLang::RuntimeException
org.jruby.exceptions.RaiseException: (NameError) Capture not found in packages processing.video; last error: cannot link Java class processing.video.Capture, probable missing dependency: org/gstreamer/elements/BufferDataAppSink$Listener
    com.jogamp.common.util.awt.AWTEDTExecutor.invoke(AWTEDTExecutor.java:58)
    jogamp.opengl.awt.AWTThreadingPlugin.invokeOnOpenGLThread(AWTThreadingPlugin.java:103)
    jogamp.opengl.ThreadingImpl.invokeOnOpenGLThread(ThreadingImpl.java:206)
    javax.media.opengl.Threading.invokeOnOpenGLThread(Threading.java:172)
    javax.media.opengl.Threading.invoke(Threading.java:191)
    javax.media.opengl.awt.GLCanvas.display(GLCanvas.java:541)
    processing.opengl.PJOGL.requestDraw(PJOGL.java:688)
    processing.opengl.PGraphicsOpenGL.requestDraw(PGraphicsOpenGL.java:1651)
    processing.core.PApplet.run(PApplet.java:2256)
    java.lang.Thread.run(Thread.java:745)

the same sketch works fine and dandy if we use Java2D rendering, that is OK for this sketch but won't allow us to do GLSL filtering for example. Problem is creating a suitable test, although I think possibly related errors have been reported for Swing apps?

@monkstone

This comment has been minimized.

Show comment
Hide comment
@monkstone

monkstone Jul 20, 2015

Contributor

@mkristian OK here is the test https://github.com/monkstone/animationloop which illustrates the problem of loading my ArcBall extension. I have not done any fancy automation I'm afraid you just need to edit runner.rb to test the permutations (runner is required for adding jars to classpath). Also I realise now that my HACK added all the processing core jars to the classpath (my filter wasn't working because I was using tap) not just the binaries (but this was probably required anyway).

Contributor

monkstone commented Jul 20, 2015

@mkristian OK here is the test https://github.com/monkstone/animationloop which illustrates the problem of loading my ArcBall extension. I have not done any fancy automation I'm afraid you just need to edit runner.rb to test the permutations (runner is required for adding jars to classpath). Also I realise now that my HACK added all the processing core jars to the classpath (my filter wasn't working because I was using tap) not just the binaries (but this was probably required anyway).

@enebo enebo added this to the JRuby 9.0.0.0 milestone Jul 20, 2015

@enebo enebo added the JRuby 9000 label Jul 20, 2015

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jul 20, 2015

Member

@monkstone thanx for the repo - I am looking what I can find now

Member

mkristian commented Jul 20, 2015

@monkstone thanx for the repo - I am looking what I can find now

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jul 20, 2015

Member

@monkstone this is what I found so far to get your sketch.rb working: https://gist.github.com/mkristian/9ce9eece566c1ac03301

there is still some strange things going on and I think overall the code should work with jruby-1.7 as well jruby-9k as is.

please note the public init method on Rarcball which I needed to add to avoid such errors https://gist.github.com/mkristian/9ce9eece566c1ac03301#file-error - @enebo any idea why this inner class can not access this package private static init method ?

Member

mkristian commented Jul 20, 2015

@monkstone this is what I found so far to get your sketch.rb working: https://gist.github.com/mkristian/9ce9eece566c1ac03301

there is still some strange things going on and I think overall the code should work with jruby-1.7 as well jruby-9k as is.

please note the public init method on Rarcball which I needed to add to avoid such errors https://gist.github.com/mkristian/9ce9eece566c1ac03301#file-error - @enebo any idea why this inner class can not access this package private static init method ?

@enebo

This comment has been minimized.

Show comment
Hide comment
@enebo

enebo Jul 20, 2015

Member

As an added note to @mkristian question. We realized this behavior for @JRubyMethod has always existed and we are only seeing it due to the other issues associated with loading. I do not think it is a bad idea for us to recommend public but we should probably fix this in both 1.7.x and 9k.x

Member

enebo commented Jul 20, 2015

As an added note to @mkristian question. We realized this behavior for @JRubyMethod has always existed and we are only seeing it due to the other issues associated with loading. I do not think it is a bad idea for us to recommend public but we should probably fix this in both 1.7.x and 9k.x

@monkstone

This comment has been minimized.

Show comment
Hide comment
@monkstone

monkstone Jul 20, 2015

Contributor

@mkristian that seems to solve that one now, it looks as though I will need to look at the other loading error, which might not be so simple to model (and fix). That is the loading of the video library jars that work fine when sketch is of the Java2D type but fails when the sketch is of OPENGL type.

Contributor

monkstone commented Jul 20, 2015

@mkristian that seems to solve that one now, it looks as though I will need to look at the other loading error, which might not be so simple to model (and fix). That is the loading of the video library jars that work fine when sketch is of the Java2D type but fails when the sketch is of OPENGL type.

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jul 20, 2015

Member

@monkstone the fact that you can just run 'sketch.rb' without the runner with jruby-1.7.x is still enough "example" to see what is going wrong with "requiring" jars in 9k. once this is fixed (hopefully soon) then we can look into the further problems of yours.

Member

mkristian commented Jul 20, 2015

@monkstone the fact that you can just run 'sketch.rb' without the runner with jruby-1.7.x is still enough "example" to see what is going wrong with "requiring" jars in 9k. once this is fixed (hopefully soon) then we can look into the further problems of yours.

mkristian added a commit that referenced this issue Jul 20, 2015

do close jrubyclassloader on ScriptingContainer#terminate only
closing it on the teardown on the runtime did cause unexpected behaviour
executing jruby via CLI

fixes #3149 and #3152
@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jul 20, 2015

Member

@monkstone the current master can run sketch.rb as is and I think this is a good starting point to look into your other sketches. thanx again for repo - it was very helpful

Member

mkristian commented Jul 20, 2015

@monkstone the current master can run sketch.rb as is and I think this is a good starting point to look into your other sketches. thanx again for repo - it was very helpful

@mkristian mkristian closed this Jul 20, 2015

@monkstone

This comment has been minimized.

Show comment
Hide comment
@monkstone

monkstone Jul 21, 2015

Contributor

Excellent @mkristian that seems to be problem solved as far as I can tell, the classpath issue had put me off creating a version of JRubyArt where sketches run without a runner (for jruby) but that is a possibility again, and in the future ruby-processing also could be updated to use jruby-9.0.0.0.

Contributor

monkstone commented Jul 21, 2015

Excellent @mkristian that seems to be problem solved as far as I can tell, the classpath issue had put me off creating a version of JRubyArt where sketches run without a runner (for jruby) but that is a possibility again, and in the future ruby-processing also could be updated to use jruby-9.0.0.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment