Fix InaccessibleObjectException on JDK16+ #73

reta · 2021-01-16T20:42:42Z

Since JEP-396 [1] integration into JDK16, the JDK internals become much less accessible. Johnzon uses setAccessible in several places to relax the visibility rules but it now throws InaccessibleObjectException on JDK16 and above (for a number of JDK classes). For example, we run into following issues with Apache CXF while running JSON-B tests on JDK16/JDK17:

java.lang.reflect.InaccessibleObjectException: Unable to make field private int java.util.ArrayList.size accessible: module java.base does not "opens java.util" to unnamed module @567d299b
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:357)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:177)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:171)
	at org.apache.johnzon.mapper.access.FieldAccessMode$FieldDecoratedType.<init>(FieldAccessMode.java:105)
	at org.apache.johnzon.mapper.access.FieldAccessMode$FieldReader.<init>(FieldAccessMode.java:169)
	at org.apache.johnzon.mapper.access.FieldAccessMode.doFindReaders(FieldAccessMode.java:49)
	at org.apache.johnzon.mapper.access.BaseAccessMode.findReaders(BaseAccessMode.java:82)
	at org.apache.johnzon.mapper.access.FieldAndMethodAccessMode.doFindReaders(FieldAndMethodAccessMode.java:68)
	at org.apache.johnzon.mapper.access.BaseAccessMode.findReaders(BaseAccessMode.java:82)
	at org.apache.johnzon.jsonb.JsonbAccessMode.findReaders(JsonbAccessMode.java:468)
	at org.apache.johnzon.mapper.Mappings.createClassMapping(Mappings.java:475)
	at org.apache.johnzon.mapper.Mappings.doFindOrCreateClassMapping(Mappings.java:437)
	at org.apache.johnzon.mapper.Mappings.findOrCreateClassMapping(Mappings.java:411)
	at org.apache.johnzon.mapper.Mapper.isDeduplicateObjects(Mapper.java:215)
	at org.apache.johnzon.mapper.Mapper.writeObjectWithGenerator(Mapper.java:209)
	at org.apache.johnzon.mapper.Mapper.writeObject(Mapper.java:203)
	at org.apache.johnzon.mapper.Mapper.writeObject(Mapper.java:230)

The suggested fix in this PR is to ignore the properties which are not accessible anymore (basically when setAccessible ends up with InaccessibleObjectException exception). Alternatively, it is still possible to use --add-opens java.base/java.util=ALL-UNNAMED but arguably this not considered acceptable.

[1] https://openjdk.java.net/jeps/396

rmannibucau · 2021-01-16T20:46:45Z

Hi @reta

Can you keep the public constructors please? It is used by extensions/custom codes.
Also wonder if:

Optional usage can be dropped
Accessibility can be tested
Accessibility can still be enforced (worse case adding a build time generation) - can help on that in ~7dayd

reta · 2021-01-16T21:08:52Z

Hey @rmannibucau , thanks for taking a look:

Optional usage can be dropped

Could drop it but replace with null checks?

Accessibility can be tested

Build with JDK16 or run test with JDK16, I could help with that on Jenkins fe.

Accessibility can still be enforced (worse case adding a build time generation) - can help on that in ~7dayd

This part is unclear to me in a sense, those are internal JDK classes, how would build time generation help?

rmannibucau · 2021-01-16T21:18:21Z

Overall null check is not needed, entry is only added if accessible.
Generator using TheClass$Accessor naming can enable to make public private fields without reflection. Can be a workaround but we need to define when accessibility is lost and needed. Regressions there are not that acceptable for the spec and future :s.

reta · 2021-01-16T22:22:47Z

@rmannibucau comments addressed, the only regression I have seen so far across the test suite is this guy:

    @Test
    public void dontStackOverFlowUsingFieldAccess() {
        final Throwable oopsImVicous = new Exception("circular");
        oopsImVicous.getStackTrace(); // fill it
        oopsImVicous.initCause(new IllegalArgumentException(oopsImVicous));
        final String serialized = new MapperBuilder().setAccessModeName("field").build().writeObjectAsString(oopsImVicous);
        assertTrue(serialized.contains("\"detailMessage\":\"circular\""));
        assertTrue(serialized.contains("\"stackTrace\":[{"));
    }

Essentially, with field access mode, none of the Exception fields are accessible and serialized is effectively empty. Personally, I don't think this is a concern since with method access level things work fine.

reta · 2021-01-17T03:25:53Z

The Travis build failed but issue seems to be unrelated:

[ERROR] Failed to execute goal on project apache-johnzon: Could not resolve dependencies for project org.apache.johnzon:apache-johnzon:jar:1.2.11-SNAPSHOT: The following artifacts could not be resolved: org.apache.johnzon:johnzon-core:jar:javadoc:1.2.11-SNAPSHOT, org.apache.johnzon:johnzon-mapper:jar:javadoc:1.2.11-SNAPSHOT, org.apache.johnzon:johnzon-jaxrs:jar:javadoc:1.2.11-SNAPSHOT, org.apache.johnzon:johnzon-websocket:jar:javadoc:1.2.11-SNAPSHOT, org.apache.johnzon:johnzon-maven-plugin:jar:javadoc:1.2.11-SNAPSHOT, org.apache.johnzon:johnzon-jsonb:jar:javadoc:1.2.11-SNAPSHOT: Could not find artifact org.apache.johnzon:johnzon-core:jar:javadoc:1.2.11-SNAPSHOT in sonatype-snapshots (https://oss.sonatype.org/content/repositories/snapshots/) -> [Help 1]

[ERROR] 

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR] 

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

[ERROR] 

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <args> -rf :apache-johnzon

rmannibucau · 2021-01-17T08:09:08Z

I care more of a regression out of the test suite, guess running tck is an option. Test suite will get some priviledged access. If no regression, not sure the change is useful too so we should refine it i think.

rmannibucau · 2021-01-17T09:14:51Z

Hi @reta, had a quick look. Seems it is only an issue if in a not yet opened module so we should be able yo skip fields in this case no - and not have to catch it as a fallback. Wdyt? Can help on code side in 7days.

reta · 2021-01-17T13:45:23Z

Hi @reta, had a quick look. Seems it is only an issue if in a not yet opened module so we should be able yo skip fields in this case no - and not have to catch it as a fallback. Wdyt? Can help on code side in 7days.

It seems like a viable option as well, probably would need some reflection in order to use the APIs available in JDK9+, I could try it for sure.

reta · 2021-01-17T13:50:24Z

I care more of a regression out of the test suite, guess running tck is an option. Test suite will get some priviledged access. If no regression, not sure the change is useful too so we should refine it i think.

Running TCK would be good, the regressions should be unlikely (🤞 ), from the access checks I have run into, Johnzon just introspects more than it needs (probably) in fact.

rmannibucau · 2021-01-17T14:56:23Z

@reta tested with master (plain master, not patched) and it works well on java 16 so guess you went into a particular case where we try to do reflection on jdk classes (java.util.ArrayList.size it seems) which is something we can avoid with a "if known() then do something clever" earlier in introspection process so I think we don't need to do anything about setAccessible but rather we should fix the cause of this arraylist introspection (maybe hashmap too). Any way to write a reproducer test?

edit: can be interesting to have the frame before .Mapper.writeObject since for a list it shouldnt enter in this method but writeCollection or friends.

reta · 2021-01-17T21:35:46Z

@rmannibucau Yes, I do have a test but I am very surprised with your outcomes, I saw a number (but not many) of test failures. Just to double check, you used latest JDK-16 EA build, right? (the JEP in question was integrated ~ month ago). Could you also try JDK17-ea? Thanks.

Edit: Just a number of different failures in ReadPrimitiveTest & CircularExceptionTest fe:

java.lang.reflect.InaccessibleObjectException: Unable to make field private final char java.lang.Character.value accessible: module java.base does not "opens java.lang" to unnamed module @33833882
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:357)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:177)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:171)
	at org.apache.johnzon.mapper.access.FieldAccessMode$FieldDecoratedType.<init>(FieldAccessMode.java:105)
	at org.apache.johnzon.mapper.access.FieldAccessMode$FieldReader.<init>(FieldAccessMode.java:169)
	at org.apache.johnzon.mapper.access.FieldAccessMode.doFindReaders(FieldAccessMode.java:49)
	at org.apache.johnzon.mapper.access.BaseAccessMode.findReaders(BaseAccessMode.java:82)
	at org.apache.johnzon.mapper.access.FieldAndMethodAccessMode.doFindReaders(FieldAndMethodAccessMode.java:68)
	at org.apache.johnzon.mapper.access.BaseAccessMode.findReaders(BaseAccessMode.java:82)
	at org.apache.johnzon.mapper.Mappings.createClassMapping(Mappings.java:475)
        ...
	at org.apache.johnzon.mapper.Mapper.readObject(Mapper.java:295)
	at org.apache.johnzon.mapper.ReadPrimitiveTest.testCharacter(ReadPrimitiveTest.java:48)

java.lang.reflect.InaccessibleObjectException: Unable to make field private java.lang.Throwable java.lang.Throwable.cause accessible: module java.base does not "opens java.lang" to unnamed module @33833882
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:357)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:177)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:171)
	at org.apache.johnzon.mapper.access.FieldAccessMode$FieldDecoratedType.<init>(FieldAccessMode.java:105)
	at org.apache.johnzon.mapper.access.FieldAccessMode$FieldReader.<init>(FieldAccessMode.java:169)
	at org.apache.johnzon.mapper.access.FieldAccessMode.doFindReaders(FieldAccessMode.java:49)
	at org.apache.johnzon.mapper.access.BaseAccessMode.findReaders(BaseAccessMode.java:82)
	at org.apache.johnzon.mapper.Mappings.createClassMapping(Mappings.java:475)
	at org.apache.johnzon.mapper.Mappings.doFindOrCreateClassMapping(Mappings.java:437)
        ...
	at org.apache.johnzon.mapper.Mapper.writeObjectAsString(Mapper.java:252)
	at org.apache.johnzon.mapper.CircularExceptionTest.dontStackOverFlow(CircularExceptionTest.java:31)

I will shortly extract the test for ArrayList as well

reta · 2021-01-18T01:36:37Z

johnzon-jsonb/src/test/java/org/apache/johnzon/jsonb/JsonbWriteTest.java

@@ -92,6 +94,14 @@ public void list() {
        assertEquals("[\"a\",\"b\"]", JsonbProvider.provider().create().build().toJson(map));
    }

+    @Test
+    public void listOfSimple() {


@rmannibucau Here is the test, consistently fails (for me) on the master (unpatched) on JDK16-ea+28 and above:

[INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running org.apache.johnzon.jsonb.JsonbWriteTest [ERROR] Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.27 s <<< FAILURE! - in org.apache.johnzon.jsonb.JsonbWriteTest [ERROR] listOfSimple(org.apache.johnzon.jsonb.JsonbWriteTest) Time elapsed: 0.016 s <<< ERROR! java.lang.reflect.InaccessibleObjectException: Unable to make field private int java.util.ArrayList.size accessible: module java.base does not "opens java.util" to unnamed module @6b71769e at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:357) at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297) at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:177) at java.base/java.lang.reflect.Field.setAccessible(Field.java:171) at org.apache.johnzon.mapper.access.FieldAccessMode$FieldDecoratedType.<init>(FieldAccessMode.java:105) at org.apache.johnzon.mapper.access.FieldAccessMode$FieldReader.<init>(FieldAccessMode.java:169) at org.apache.johnzon.mapper.access.FieldAccessMode.doFindReaders(FieldAccessMode.java:49) at org.apache.johnzon.mapper.access.BaseAccessMode.findReaders(BaseAccessMode.java:82) at org.apache.johnzon.mapper.access.FieldAndMethodAccessMode.doFindReaders(FieldAndMethodAccessMode.java:68) at org.apache.johnzon.mapper.access.BaseAccessMode.findReaders(BaseAccessMode.java:82) at org.apache.johnzon.jsonb.JsonbAccessMode.findReaders(JsonbAccessMode.java:468) at org.apache.johnzon.mapper.Mappings.createClassMapping(Mappings.java:475) at org.apache.johnzon.mapper.Mappings.doFindOrCreateClassMapping(Mappings.java:437) at org.apache.johnzon.mapper.Mappings.findOrCreateClassMapping(Mappings.java:411) at org.apache.johnzon.mapper.Mapper.isDeduplicateObjects(Mapper.java:215) at org.apache.johnzon.mapper.Mapper.writeObjectWithGenerator(Mapper.java:209) at org.apache.johnzon.mapper.Mapper.writeObject(Mapper.java:203) at org.apache.johnzon.mapper.Mapper.writeObjectAsString(Mapper.java:252) at org.apache.johnzon.jsonb.JohnzonJsonb.toJson(JohnzonJsonb.java:342) at org.apache.johnzon.jsonb.JsonbWriteTest.listOfSimple(JsonbWriteTest.java:102)

Interesting, just read on the phone but our list detection is something along "is parameterized and raw type is a collection or is an array" so passing List.class was moving it to a fallback handling and lead to the unexpected stack you get.
Guess it is what we must fix and probably for maps too.

Hope it makes sense

Yeah, in this particular case for ArrayList it could be certainly done, I run into it by accindent ;-). I am wondering if you were able to see the other failures, ReadPrimitiveTest & CircularExceptionTest?

Didnt run it but looks like it is the same issue of "too late type handling" so guess fix is to move it earlier trying to limit runtime impact.
If not urgent can work on it next week.

@rmannibucau no, it is not urgent, please let me know if I could help in scope of this PR (or I can just close it).

Let's keep it open until we review the final solution and agree on it. Please ping me if you dont get news in 10 days ;)

Hey @rmannibucau, I gave it a second thought, particularly when you mentioned the presence of the extensions and completely refactored the approach to eliminate flows driven by exceptions. I think it is more clean but let us discuss it when you have time, thanks.

Hi @reta,

I like this version more but it still fixes the consequence instead of the source of the issue which is an unexpected branch in the mapper (class for collection/primitive/map/throwable/jdk-class) so even if this flavor can be kept I still think we should fix first the cause then maybe review this flavor.

Wdyt?

@rmannibucau This very correct, I will continue digging in so we could be proactive in introspection (we are indeed reactive now), will be back on that :) Thanks for the feedback.

…ssible and added accessibility property to readers and writers

rmannibucau · 2021-01-23T17:04:06Z

@reta can you review https://issues.apache.org/jira/browse/JOHNZON-331 and the related commit (15a4cc6). Does it solve your issue or do we need "more" - note that it can need some more love from cxf side is type is unexpectedly erased like using getType instead of getGenericType somewhere maybe?

reta · 2021-01-23T17:35:23Z

@rmannibucau for my issue with ArrayList - definitely enough, do you want to address InaccessibleObjectException in general in scope of this PR? I have collected enough details regarding the issue but have not updated the change yet.

rmannibucau · 2021-01-23T18:43:30Z

@reta question is: can it still happen in a valid setup (mapping a not opened/imported model is assumed wrong i guess). I didnt have more issues on java 16 so for me this exception is gone but if you see other cases we must fix them. Maybe exception case? Didnt check this last one but fix is likely to map it directly and not by reflection. Wdyt?

reta · 2021-01-23T19:17:19Z

@rmannibucau Mapping a not opened/imported model is certainly not advisable, probably the question would be: should Johnzon do best effort by not trying to access what it cannot, or just fail fast. Sadly I do not have the only right answer, but I think certain measures from the library/framework could help. Knowing that the semantic of the setAccessible has changed, try to access and skip if it is not possible (log warning may be?). As for the user of the library, it is logical and reasonable expectation, but this is just my opinion.

rmannibucau · 2021-01-23T19:23:31Z

Well my reasoning is:

Jpms is not adopted and will not probably so maybe not a big deal
If not opened it is likely intended in jpms land so probably better to respect it in the spec (was not discussed)

So overall your impl could be a toggle but my feeling is that it shouldnt be the default if we want to support jpms correctly - but once again jpms can be dead today so maybe not a big deal.

Side note: no logging is likely desired so failling or swallowing probably?

Anyone else has an opinion on it?

reta · 2021-01-23T20:00:17Z

@rmannibucau thanks, let me just keep it open for a few days if someone has opinion on the matter, otherwise let us just close it as not needed, wdyt?

rmannibucau · 2021-01-25T10:55:47Z

@reta added some basic exception support: 4f53103 .

reta · 2021-01-25T12:40:52Z

Closing as not needed

Fix InaccessibleObjectException on JDK16+

27a75cc

Addressing code review comments

26b029c

reta force-pushed the fix.set.accessible branch from 771498b to 26b029c Compare January 17, 2021 02:18

Added more test cases

ea41196

reta commented Jan 18, 2021

View reviewed changes

reta force-pushed the fix.set.accessible branch from 4c82dd8 to fbff720 Compare January 19, 2021 01:53

Completely refacored the approach to use AccessibleObject::trySetAcce…

155558f

…ssible and added accessibility property to readers and writers

reta force-pushed the fix.set.accessible branch from fbff720 to 155558f Compare January 19, 2021 02:16

reta closed this Jan 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix InaccessibleObjectException on JDK16+ #73

Fix InaccessibleObjectException on JDK16+ #73

reta commented Jan 16, 2021

rmannibucau commented Jan 16, 2021

reta commented Jan 16, 2021 •

edited

Loading

rmannibucau commented Jan 16, 2021

reta commented Jan 16, 2021 •

edited

Loading

reta commented Jan 17, 2021

rmannibucau commented Jan 17, 2021

rmannibucau commented Jan 17, 2021

reta commented Jan 17, 2021

reta commented Jan 17, 2021

rmannibucau commented Jan 17, 2021 •

edited

Loading

reta commented Jan 17, 2021 •

edited

Loading

reta Jan 18, 2021

rmannibucau Jan 18, 2021

reta Jan 18, 2021

rmannibucau Jan 18, 2021

reta Jan 18, 2021

rmannibucau Jan 18, 2021

reta Jan 19, 2021

rmannibucau Jan 19, 2021

reta Jan 19, 2021

rmannibucau commented Jan 23, 2021 •

edited

Loading

reta commented Jan 23, 2021

rmannibucau commented Jan 23, 2021

reta commented Jan 23, 2021

rmannibucau commented Jan 23, 2021

reta commented Jan 23, 2021

rmannibucau commented Jan 25, 2021

reta commented Jan 25, 2021

Fix InaccessibleObjectException on JDK16+ #73

Fix InaccessibleObjectException on JDK16+ #73

Conversation

reta commented Jan 16, 2021

rmannibucau commented Jan 16, 2021

reta commented Jan 16, 2021 • edited Loading

rmannibucau commented Jan 16, 2021

reta commented Jan 16, 2021 • edited Loading

reta commented Jan 17, 2021

rmannibucau commented Jan 17, 2021

rmannibucau commented Jan 17, 2021

reta commented Jan 17, 2021

reta commented Jan 17, 2021

rmannibucau commented Jan 17, 2021 • edited Loading

reta commented Jan 17, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmannibucau commented Jan 23, 2021 • edited Loading

reta commented Jan 23, 2021

rmannibucau commented Jan 23, 2021

reta commented Jan 23, 2021

rmannibucau commented Jan 23, 2021

reta commented Jan 23, 2021

rmannibucau commented Jan 25, 2021

reta commented Jan 25, 2021

reta commented Jan 16, 2021 •

edited

Loading

reta commented Jan 16, 2021 •

edited

Loading

rmannibucau commented Jan 17, 2021 •

edited

Loading

reta commented Jan 17, 2021 •

edited

Loading

rmannibucau commented Jan 23, 2021 •

edited

Loading