New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to use protected java method in jruby with jdk9 #4851

Closed
monkstone opened this Issue Nov 10, 2017 · 17 comments

Comments

Projects
None yet
6 participants
@monkstone
Contributor

monkstone commented Nov 10, 2017

Unable to use inherited, protected method with jdk9.

Environment

Failing Setup:

  • jruby 9.1.14.0 (2.3.3) 2017-11-08 2176f24 Java HotSpot(TM) 64-Bit Server VM 9.0.1+11 on 9.0.1+11 +jit [linux-x86_64]

  • Linux tux-PC-2163 4.10.0-38-generic #42~16.04.1-Ubuntu SMP Tue Oct 10 16:32:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Passing Setup

  • jruby 9.1.14.0 (2.3.3) 2017-11-08 2176f24 OpenJDK 64-Bit Server VM 25.151-b12 on 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12 +jit [linux-x86_64]

  • Linux tux-PC-2163 4.10.0-38-generic #42~16.04.1-Ubuntu SMP Tue Oct 10 16:32:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Expected Behavior

Actual Behavior

  • Fails with TypeError message
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by jnr.posix.JavaLibCHelper$ReflectiveAccess to method sun.nio.ch.SelChImpl.getFD()
WARNING: Please consider reporting this to the maintainers of jnr.posix.JavaLibCHelper$ReflectiveAccess
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
TypeError: illegal access on 'runSketch': class org.jruby.javasupport.JavaMethod cannot access a member of class processing.core.PApplet with modifiers "protected"
  initialize at /home/tux/rbprocessing/lib/rbprocessing/app.rb:10
      <main> at test.rb:18

The runSketch method

The runSketch method is provided by processing.org, primarily for the benefit of processing.py, that uses jython to create a python wrapper around processing. In JRubyArt and propane I have created a ruby wrapper around processing using jruby. Is there a way I can still access this method with jdk9?

Behaviour since confirmed with simpler java test case, that is also included in repo.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Nov 10, 2017

Member

Get a full trace for me with -Xbacktrace.style=full passed to JRuby or otherwise jruby.backtrace.style property set to "full" in JVM.

In order to reduce warnings on Java 9, we now check if a module is open before we try to setAccessible the otherwise-inaccessible methods. That may mean we're binding methods that we can't set accessible, and then they're getting called. That's obviously not ideal, but we can't avoid binding it altogether, especially for these inheritance cases that should have access to the parent class.

Long term, we are likely going to need to do more with invokedynamic to properly get access to call those protected methods in parent classes, rather than setting them accessible. But we need a decent workaround now.

The full trace will help me sort out where this is happening. We need to start running our Java integration tests on JRuby to see how much breaks with modules.

Member

headius commented Nov 10, 2017

Get a full trace for me with -Xbacktrace.style=full passed to JRuby or otherwise jruby.backtrace.style property set to "full" in JVM.

In order to reduce warnings on Java 9, we now check if a module is open before we try to setAccessible the otherwise-inaccessible methods. That may mean we're binding methods that we can't set accessible, and then they're getting called. That's obviously not ideal, but we can't avoid binding it altogether, especially for these inheritance cases that should have access to the parent class.

Long term, we are likely going to need to do more with invokedynamic to properly get access to call those protected methods in parent classes, rather than setting them accessible. But we need a decent workaround now.

The full trace will help me sort out where this is happening. We need to start running our Java integration tests on JRuby to see how much breaks with modules.

@headius headius added this to the JRuby 9.2.0.0 milestone Nov 10, 2017

@monkstone

This comment has been minimized.

Show comment
Hide comment
@monkstone

monkstone Nov 11, 2017

Contributor

To simplify testing I have created a new repo https://github.com/monkstone/protected_method
Here is the fullbacktrace:-

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by jnr.posix.JavaLibCHelper$ReflectiveAccess to method sun.nio.ch.SelChImpl.getFD()
WARNING: Please consider reporting this to the maintainers of jnr.posix.JavaLibCHelper$ReflectiveAccess
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
TypeError: illegal access on 'test_method': class org.jruby.javasupport.JavaMethod cannot access a member of class monkstone.ProtectedSuper with modifiers "protected"
getStackTrace at java/lang/Thread.java:1654
getBacktraceData at org/jruby/runtime/backtrace/TraceType.java:246
getBacktrace at org/jruby/runtime/backtrace/TraceType.java:47
prepareBacktrace at org/jruby/RubyException.java:235
preRaise at org/jruby/exceptions/RaiseException.java:216
preRaise at org/jruby/exceptions/RaiseException.java:183
at org/jruby/exceptions/RaiseException.java:111
newRaiseException at org/jruby/Ruby.java:4138
newTypeError at org/jruby/Ruby.java:3809
handleIllegalAccessEx at org/jruby/javasupport/JavaCallable.java:169
invokeDirectWithExceptionHandling at org/jruby/javasupport/JavaMethod.java:443
tryProxyInvocation at org/jruby/javasupport/JavaMethod.java:623
invokeDirect at org/jruby/javasupport/JavaMethod.java:299
call at org/jruby/java/invokers/InstanceMethodInvoker.java:36
cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:298
call at org/jruby/runtime/callsite/CachingCallSite.java:127
invokeOther7:test_method at ruby_protected.rb:12
initialize at ruby_protected.rb:12
call at org/jruby/internal/runtime/methods/CompiledIRMethod.java:90
cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:308
call at org/jruby/runtime/callsite/CachingCallSite.java:137
newInstance at org/jruby/RubyClass.java:994
call at org/jruby/RubyClass$INVOKER$i$newInstance.gen:-1
call at org/jruby/internal/runtime/methods/DynamicMethod.java:192
call at org/jruby/java/proxies/ConcreteJavaProxy.java:151
cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:298
call at org/jruby/runtime/callsite/CachingCallSite.java:127
invokeOther10:new at ruby_protected.rb:17

at ruby_protected.rb:17
invokeWithArguments at java/lang/invoke/MethodHandle.java:638
load at org/jruby/ir/Compiler.java:95
runScript at org/jruby/Ruby.java:828
runNormally at org/jruby/Ruby.java:747
runNormally at org/jruby/Ruby.java:765
runFromMain at org/jruby/Ruby.java:578
doRunFromMain at org/jruby/Main.java:417
internalRun at org/jruby/Main.java:305
run at org/jruby/Main.java:232
main at org/jruby/Main.java:204

Contributor

monkstone commented Nov 11, 2017

To simplify testing I have created a new repo https://github.com/monkstone/protected_method
Here is the fullbacktrace:-

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by jnr.posix.JavaLibCHelper$ReflectiveAccess to method sun.nio.ch.SelChImpl.getFD()
WARNING: Please consider reporting this to the maintainers of jnr.posix.JavaLibCHelper$ReflectiveAccess
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
TypeError: illegal access on 'test_method': class org.jruby.javasupport.JavaMethod cannot access a member of class monkstone.ProtectedSuper with modifiers "protected"
getStackTrace at java/lang/Thread.java:1654
getBacktraceData at org/jruby/runtime/backtrace/TraceType.java:246
getBacktrace at org/jruby/runtime/backtrace/TraceType.java:47
prepareBacktrace at org/jruby/RubyException.java:235
preRaise at org/jruby/exceptions/RaiseException.java:216
preRaise at org/jruby/exceptions/RaiseException.java:183
at org/jruby/exceptions/RaiseException.java:111
newRaiseException at org/jruby/Ruby.java:4138
newTypeError at org/jruby/Ruby.java:3809
handleIllegalAccessEx at org/jruby/javasupport/JavaCallable.java:169
invokeDirectWithExceptionHandling at org/jruby/javasupport/JavaMethod.java:443
tryProxyInvocation at org/jruby/javasupport/JavaMethod.java:623
invokeDirect at org/jruby/javasupport/JavaMethod.java:299
call at org/jruby/java/invokers/InstanceMethodInvoker.java:36
cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:298
call at org/jruby/runtime/callsite/CachingCallSite.java:127
invokeOther7:test_method at ruby_protected.rb:12
initialize at ruby_protected.rb:12
call at org/jruby/internal/runtime/methods/CompiledIRMethod.java:90
cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:308
call at org/jruby/runtime/callsite/CachingCallSite.java:137
newInstance at org/jruby/RubyClass.java:994
call at org/jruby/RubyClass$INVOKER$i$newInstance.gen:-1
call at org/jruby/internal/runtime/methods/DynamicMethod.java:192
call at org/jruby/java/proxies/ConcreteJavaProxy.java:151
cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:298
call at org/jruby/runtime/callsite/CachingCallSite.java:127
invokeOther10:new at ruby_protected.rb:17

at ruby_protected.rb:17
invokeWithArguments at java/lang/invoke/MethodHandle.java:638
load at org/jruby/ir/Compiler.java:95
runScript at org/jruby/Ruby.java:828
runNormally at org/jruby/Ruby.java:747
runNormally at org/jruby/Ruby.java:765
runFromMain at org/jruby/Ruby.java:578
doRunFromMain at org/jruby/Main.java:417
internalRun at org/jruby/Main.java:305
run at org/jruby/Main.java:232
main at org/jruby/Main.java:204

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Nov 12, 2017

Member

Notes on getting indy to do a better job here:

  • In order for indy to be able to dispatch to protected methods, it would need to have a lookup object from the child class.
  • This would currently only fit into the dispatch side; we'd check if the caller is a proxied or extended Java object and then use a lookup from that wrapper.
  • Unfortunately this would only work for indy-based dispatch. If a protected Java call is dispatched through reflection, there's no way to make it work right.

I have not looked at your repo yet (or processing.org + jruby in a long time). Can you clarify: this is a call you're attempting to make to a protected method in a parent class from a Ruby subclass of that class, right?

Member

headius commented Nov 12, 2017

Notes on getting indy to do a better job here:

  • In order for indy to be able to dispatch to protected methods, it would need to have a lookup object from the child class.
  • This would currently only fit into the dispatch side; we'd check if the caller is a proxied or extended Java object and then use a lookup from that wrapper.
  • Unfortunately this would only work for indy-based dispatch. If a protected Java call is dispatched through reflection, there's no way to make it work right.

I have not looked at your repo yet (or processing.org + jruby in a long time). Can you clarify: this is a call you're attempting to make to a protected method in a parent class from a Ruby subclass of that class, right?

@monkstone

This comment has been minimized.

Show comment
Hide comment
@monkstone

monkstone Nov 12, 2017

Contributor

Thats correct, although in a test I found issue exists with a direct call to a protected java method of java class from jruby.

Contributor

monkstone commented Nov 12, 2017

Thats correct, although in a test I found issue exists with a direct call to a protected java method of java class from jruby.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Nov 14, 2017

Member

Yes, those are two different cases though. Your case is more important, since when extending a class you should be able to call its protected methods. The other case is not as important to support, since Java code could not legally make such a call either.

Member

headius commented Nov 14, 2017

Yes, those are two different cases though. Your case is more important, since when extending a class you should be able to call its protected methods. The other case is not as important to support, since Java code could not legally make such a call either.

@faisalrabbani

This comment has been minimized.

Show comment
Hide comment
@faisalrabbani

faisalrabbani Nov 18, 2017

I am having same issue with activerecord-jdbc-adapter.
error states:
TypeError (illegal access on 'clone': class org.jruby.javasupport.JavaMethod cannot access a member of class java.lang.Object (in module java.base) with modifiers "protected native")
at following line:
gems/gems/activerecord-jdbc-adapter-50.0/lib/arjdbc/jdbc/driver.rb:36

properties = self.properties.clone

Is there any solution ?

faisalrabbani commented Nov 18, 2017

I am having same issue with activerecord-jdbc-adapter.
error states:
TypeError (illegal access on 'clone': class org.jruby.javasupport.JavaMethod cannot access a member of class java.lang.Object (in module java.base) with modifiers "protected native")
at following line:
gems/gems/activerecord-jdbc-adapter-50.0/lib/arjdbc/jdbc/driver.rb:36

properties = self.properties.clone

Is there any solution ?

@kares

This comment has been minimized.

Show comment
Hide comment
@kares

kares Nov 18, 2017

Member

@faisalrabbani there isn't except for downgrading to Java 8 ATM, your case might also be considered slightly more specific. we can fix that at AR-JDBC's end -> that does not fix the issue reported thought

Member

kares commented Nov 18, 2017

@faisalrabbani there isn't except for downgrading to Java 8 ATM, your case might also be considered slightly more specific. we can fix that at AR-JDBC's end -> that does not fix the issue reported thought

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Nov 19, 2017

Member

@faisalrabbani Please report your issue at jruby/activerecord-jdbc-adapter please. This one should be fixable. The issue here is more related to the cases where we can't fix or workaround the protected access.

Member

headius commented Nov 19, 2017

@faisalrabbani Please report your issue at jruby/activerecord-jdbc-adapter please. This one should be fixable. The issue here is more related to the cases where we can't fix or workaround the protected access.

@monkstone

This comment has been minimized.

Show comment
Hide comment
@monkstone

monkstone Mar 25, 2018

Contributor

For my propane project I have created a custom PApplet class, where I create public method to replace the protected method, and this creates a partial fix.

protected void runSketch() {
        runSketch(new String[0]);
    }

fix

public void runPropane() {
        runSketch(new String[0]);
    }

The runSketch with String[] args

    protected void runSketch(final String[] args) {
        final String[] argsWithSketchName = new String[args.length + 1];
        System.arraycopy(args, 0, argsWithSketchName, 0, args.length);
        final String className = this.getClass().getSimpleName();
        final String cleanedClass
                = className.replaceAll("__[^_]+__\\$", "").replaceAll("\\$\\d+", "");
        argsWithSketchName[args.length] = cleanedClass;
        runSketch(argsWithSketchName, this);
    }

PS: the argsWithSketchName resolves to [PApplet$Proxy1]

However this is not possible with JRubyArt which relies on a user installed version of processing

Further selected sketches in propane now failing with other inherited protected method eg when we subclass javax.swing.JSlider we are unable to call fireStateChanged to notify listeners.

Contributor

monkstone commented Mar 25, 2018

For my propane project I have created a custom PApplet class, where I create public method to replace the protected method, and this creates a partial fix.

protected void runSketch() {
        runSketch(new String[0]);
    }

fix

public void runPropane() {
        runSketch(new String[0]);
    }

The runSketch with String[] args

    protected void runSketch(final String[] args) {
        final String[] argsWithSketchName = new String[args.length + 1];
        System.arraycopy(args, 0, argsWithSketchName, 0, args.length);
        final String className = this.getClass().getSimpleName();
        final String cleanedClass
                = className.replaceAll("__[^_]+__\\$", "").replaceAll("\\$\\d+", "");
        argsWithSketchName[args.length] = cleanedClass;
        runSketch(argsWithSketchName, this);
    }

PS: the argsWithSketchName resolves to [PApplet$Proxy1]

However this is not possible with JRubyArt which relies on a user installed version of processing

Further selected sketches in propane now failing with other inherited protected method eg when we subclass javax.swing.JSlider we are unable to call fireStateChanged to notify listeners.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Mar 26, 2018

Member

Is there some way we we could get the Processing folks to expose this logic? I get that it's not a great solution to make these methods public, but protected is a difficult visibility for us to call through.

The general problem of not being able to call protected methods will require Java 9 flags (expose/open reflection up for specific classes/modules, or disable checking completely) or a fully invokedynamic-based solution that can properly handle protected calls from an interface implementation in Ruby.

Member

headius commented Mar 26, 2018

Is there some way we we could get the Processing folks to expose this logic? I get that it's not a great solution to make these methods public, but protected is a difficult visibility for us to call through.

The general problem of not being able to call protected methods will require Java 9 flags (expose/open reflection up for specific classes/modules, or disable checking completely) or a fully invokedynamic-based solution that can properly handle protected calls from an interface implementation in Ruby.

@monkstone

This comment has been minimized.

Show comment
Hide comment
@monkstone

monkstone Mar 27, 2018

Contributor

I can understand your reluctance to create a fix however there is an expectation in java that inherited classes can use protected method, so when a ruby class inherits from a java class there is a similar expectation. So in addition to the one critical case that I can fix with propane there is a problem with our control_panel library see ruby-processing/propane#21. I somewhat doubt that processing guys will agree to fix, they only use runSketch() method for processing.py and jython development seems to be going nowhere. I'm inclined to suggest the use of propane over JRubyArt. Further for the future I'm worried by ruby-processing/propane#22 and lack of recent support from JOGL group also there are issues with FX2D sketches ruby-processing/propane#20. This after all is only a hobby for me.

Contributor

monkstone commented Mar 27, 2018

I can understand your reluctance to create a fix however there is an expectation in java that inherited classes can use protected method, so when a ruby class inherits from a java class there is a similar expectation. So in addition to the one critical case that I can fix with propane there is a problem with our control_panel library see ruby-processing/propane#21. I somewhat doubt that processing guys will agree to fix, they only use runSketch() method for processing.py and jython development seems to be going nowhere. I'm inclined to suggest the use of propane over JRubyArt. Further for the future I'm worried by ruby-processing/propane#22 and lack of recent support from JOGL group also there are issues with FX2D sketches ruby-processing/propane#20. This after all is only a hobby for me.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Apr 10, 2018

Member

Right, I understand that. It's not impossible to support, but with the many levels of method dispatch plumbing it will take some creative use of invokedynamic to do so. I have not had a chance to look at module system changes in Java 10...perhaps they have made it easier to soften these checks?

Member

headius commented Apr 10, 2018

Right, I understand that. It's not impossible to support, but with the many levels of method dispatch plumbing it will take some creative use of invokedynamic to do so. I have not had a chance to look at module system changes in Java 10...perhaps they have made it easier to soften these checks?

@monkstone

This comment has been minimized.

Show comment
Hide comment
@monkstone

monkstone Jul 5, 2018

Contributor

@headius The problem seems to go away with jdk10
environment:-
jruby 9.2.0.0 (2.5.0) 2018-05-24 81156a8 OpenJDK 64-Bit Server VM 10.0.1+10-Ubuntu-3ubuntu1 on 10.0.1+10-Ubuntu-3ubuntu1 +jit [linux-x86_64]

Contributor

monkstone commented Jul 5, 2018

@headius The problem seems to go away with jdk10
environment:-
jruby 9.2.0.0 (2.5.0) 2018-05-24 81156a8 OpenJDK 64-Bit Server VM 10.0.1+10-Ubuntu-3ubuntu1 on 10.0.1+10-Ubuntu-3ubuntu1 +jit [linux-x86_64]

@monkstone monkstone changed the title from Unable to use protected java method in jruby since jdk9 to Unable to use protected java method in jruby with jdk9 Jul 18, 2018

@monkstone

This comment has been minimized.

Show comment
Hide comment
@monkstone

monkstone Jul 18, 2018

Contributor

As this seems to be jdk9 specific (and therefore transitory) I am closing issue as fixed with jdk10

Contributor

monkstone commented Jul 18, 2018

As this seems to be jdk9 specific (and therefore transitory) I am closing issue as fixed with jdk10

@prashantvithani

This comment has been minimized.

Show comment
Hide comment
@prashantvithani

prashantvithani Sep 11, 2018

Contributor

Looks like it still persists with jdk10
JRuby version:

jruby 9.2.0.0 (2.5.0) 2018-05-24 81156a8 Java HotSpot(TM) 64-Bit Server VM 10.0.2+13 on 10.0.2+13 [darwin-x86_64]

Error:

[1] pry(main)> java_import 'java.io.FileOutputStream;
[2] pry(main)> java_import 'java.util.logging.StreamHandler;
[3] pry(main)> Java::JavaUtilLogging::StreamHandler.new.setOutputStream(FileOutputStream.new('/tmp/file.log'))
NoMethodError: undefined method `setOutputStream' for #<Java::JavaUtilLogging::StreamHandler:0x1b8117f4>
from (pry):3:in `<eval>'

@headius

Contributor

prashantvithani commented Sep 11, 2018

Looks like it still persists with jdk10
JRuby version:

jruby 9.2.0.0 (2.5.0) 2018-05-24 81156a8 Java HotSpot(TM) 64-Bit Server VM 10.0.2+13 on 10.0.2+13 [darwin-x86_64]

Error:

[1] pry(main)> java_import 'java.io.FileOutputStream;
[2] pry(main)> java_import 'java.util.logging.StreamHandler;
[3] pry(main)> Java::JavaUtilLogging::StreamHandler.new.setOutputStream(FileOutputStream.new('/tmp/file.log'))
NoMethodError: undefined method `setOutputStream' for #<Java::JavaUtilLogging::StreamHandler:0x1b8117f4>
from (pry):3:in `<eval>'

@headius

@monkstone

This comment has been minimized.

Show comment
Hide comment
@monkstone

monkstone Sep 11, 2018

Contributor

@prashantvithani Are you sure this is same issue, I don't recognize your error message as being the same I had.

TypeError: illegal access on 'test_method': class org.jruby.javasupport.JavaMethod cannot access a member of class monkstone.ProtectedSuper with modifiers "protected"

Further I can confirm my original issue (call to runSketch) and my protected test both work as expected with java 10.0.1 2018-04-17.

Contributor

monkstone commented Sep 11, 2018

@prashantvithani Are you sure this is same issue, I don't recognize your error message as being the same I had.

TypeError: illegal access on 'test_method': class org.jruby.javasupport.JavaMethod cannot access a member of class monkstone.ProtectedSuper with modifiers "protected"

Further I can confirm my original issue (call to runSketch) and my protected test both work as expected with java 10.0.1 2018-04-17.

@kares

This comment has been minimized.

Show comment
Hide comment
@kares

kares Sep 12, 2018

Member

as @monkstone noted its not entirely the same, and I would argue that it was uncharted territory before:
would not expect Java::JavaUtilLogging::StreamHandler.new.setOutputStream(...) to work since its a protected instance method. the fact that it did on Java 8 would be an implementation detail. it probably still should work with send which it seems it does not. let's open a separate issue for that.

Member

kares commented Sep 12, 2018

as @monkstone noted its not entirely the same, and I would argue that it was uncharted territory before:
would not expect Java::JavaUtilLogging::StreamHandler.new.setOutputStream(...) to work since its a protected instance method. the fact that it did on Java 8 would be an implementation detail. it probably still should work with send which it seems it does not. let's open a separate issue for that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment