Easymock doesn't work in Java 11 with --illegal-access=deny

[yrodiere]

When running tests in a Java 11 JVM with --illegal-access=deny, Easymock will fail with a ExceptionInInitializerError upon creating a mock.

It's mainly a problem for library maintainers using EasyMock for their tests: --illegal-access=deny allows us to check that our library does not require any illegal access to other modules, and is thus a good way to check that the library stays usable in a modular environment.

Stack trace

java.lang.ExceptionInInitializerError
	at org.easymock.internal.ClassProxyFactory.createEnhancer(ClassProxyFactory.java:233)
	at org.easymock.internal.ClassProxyFactory.createProxy(ClassProxyFactory.java:165)
	at org.easymock.internal.MocksControl.createMock(MocksControl.java:98)
	at org.easymock.internal.MocksControl.createMock(MocksControl.java:78)
	at org.easymock.IMocksControl.mock(IMocksControl.java:67)
	at org.easymock.EasyMock.mock(EasyMock.java:96)
	at org.easymock.EasyMock.createMock(EasyMock.java:374)
	at org.hibernate.search.elasticsearch.processor.impl.DefaultContextualErrorHandlerTest.luceneWork(DefaultContextualErrorHandlerTest.java:232)
	at org.hibernate.search.elasticsearch.processor.impl.DefaultContextualErrorHandlerTest.initMocks(DefaultContextualErrorHandlerTest.java:53)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: org.easymock.cglib.core.CodeGenerationException: java.lang.reflect.InaccessibleObjectException-->Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to unnamed module @a9cd3b1
	at org.easymock.cglib.core.ReflectUtils.defineClass(ReflectUtils.java:464)
	at org.easymock.cglib.core.AbstractClassGenerator.generate(AbstractClassGenerator.java:336)
	at org.easymock.cglib.core.AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:93)
	at org.easymock.cglib.core.AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:91)
	at org.easymock.cglib.core.internal.LoadingCache$2.call(LoadingCache.java:54)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.easymock.cglib.core.internal.LoadingCache.createEntry(LoadingCache.java:61)
	at org.easymock.cglib.core.internal.LoadingCache.get(LoadingCache.java:34)
	at org.easymock.cglib.core.AbstractClassGenerator$ClassLoaderData.get(AbstractClassGenerator.java:116)
	at org.easymock.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:291)
	at org.easymock.cglib.core.KeyFactory$Generator.create(KeyFactory.java:221)
	at org.easymock.cglib.core.KeyFactory.create(KeyFactory.java:174)
	at org.easymock.cglib.core.KeyFactory.create(KeyFactory.java:153)
	at org.easymock.cglib.proxy.Enhancer.<clinit>(Enhancer.java:73)
	... 31 more
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to unnamed module @a9cd3b1
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:340)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:280)
	at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:198)
	at java.base/java.lang.reflect.Method.setAccessible(Method.java:192)
	at org.easymock.cglib.core.ReflectUtils$1.run(ReflectUtils.java:61)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.easymock.cglib.core.ReflectUtils.<clinit>(ReflectUtils.java:52)
	at org.easymock.cglib.core.KeyFactory$Generator.generateClass(KeyFactory.java:243)
	at org.easymock.cglib.core.DefaultGeneratorStrategy.generate(DefaultGeneratorStrategy.java:25)
	at org.easymock.cglib.core.AbstractClassGenerator.generate(AbstractClassGenerator.java:329)
	... 43 more

Details of the problem

The source of the problem seems to lie in cglib, but since the cglib project no longer aims at supporting new JVMs, I guess it becomes Easymock's problem.

In short, the error happens because the Enhancer class has static initalizers relying on KeyFactory.create, which indirectly relies on ReflectUtils, which in turn has a static initializer that tries to find a way to define classes (ClassLoader#defineClass or Unsafe#defineClass), but in this case cannot find any. Interestingly enough, defining a class does not seem absolutely necessary, as the method being used in ReflectUtils does not use it. It's just that an unrelated static initializer of ReflectUtils fails...

More precisely, the problem lies in this code:

    static {
        ProtectionDomain protectionDomain;
        Method defineClass, defineClassUnsafe;
        Object unsafe;
        Throwable throwable = null;
        try {
            protectionDomain = getProtectionDomain(ReflectUtils.class);
            try {
                defineClass = (Method) AccessController.doPrivileged(new PrivilegedExceptionAction() {
                    public Object run() throws Exception {
                            Class loader = Class.forName("java.lang.ClassLoader"); // JVM crash w/o this
                            Method defineClass = loader.getDeclaredMethod("defineClass",
                                                                    new Class[]{ String.class,
                                                                                 byte[].class,
                                                                                 Integer.TYPE,
                                                                                 Integer.TYPE,
                                                                                 ProtectionDomain.class });
                            defineClass.setAccessible(true);
                            return defineClass;
                    }
                });
                defineClassUnsafe = null;
                unsafe = null;
            } catch (Throwable t) {
                // Fallback on Jigsaw where this method is not available.
                throwable = t;
                defineClass = null;
                unsafe = AccessController.doPrivileged(new PrivilegedExceptionAction() {
                    public Object run() throws Exception {
                        Class u = Class.forName("sun.misc.Unsafe");
                        Field theUnsafe = u.getDeclaredField("theUnsafe");
                        theUnsafe.setAccessible(true);
                        return theUnsafe.get(null);
                    }
                });
                Class u = Class.forName("sun.misc.Unsafe");
                defineClassUnsafe = u.getMethod("defineClass",
                                        new Class[]{ String.class,
                                                     byte[].class,
                                                     Integer.TYPE,
                                                     Integer.TYPE,
                                                     ClassLoader.class,
                                                     ProtectionDomain.class });
            }
            AccessController.doPrivileged(new PrivilegedExceptionAction() {
                public Object run() throws Exception {
                    Method[] methods = Object.class.getDeclaredMethods();
                    for (Method method : methods) {
                        if ("finalize".equals(method.getName())
                                || (method.getModifiers() & (Modifier.FINAL | Modifier.STATIC)) > 0) {
                            continue;
                        }
                        OBJECT_METHODS.add(method);
                    }
                    return null;
                }
            });
        } catch (Throwable t) {
            if (throwable == null) {
                throwable = t;
            }
            protectionDomain = null;
            defineClass = null;
            defineClassUnsafe = null;
            unsafe = null;
        }
        PROTECTION_DOMAIN = protectionDomain;
        DEFINE_CLASS = defineClass;
        DEFINE_CLASS_UNSAFE = defineClassUnsafe;
        UNSAFE = unsafe;
        THROWABLE = throwable;
    }

As you can see, cglib first tries to make ClassLoader#defineClass, and when that fails (because of the --illegal-access=deny parameter), it falls back to using sun.misc.Unsafe#defineClass. Problem is, sun.misc.Unsafe#defineClass does not exist anymore in Java 11, so this code ends up throwing the original exception ("InaccessibleObjectException-->Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to unnamed module").

Workarounds

Obviously one can remove the --illegal-access=deny option or set it to warn, but that would go against the original intent, which is to be able to detect illegal access while testing a library.

It's possible to check for illegal accesses almost everywhere by keeping the option, but also adding --add-opens java.base/java.lang=ALL-UNNAMED. Then illegal accesses to java.lang will go undetected, but others will be correctly detected and trigger errors.

Solutions

One solution would be to fix cglib, but it seems the project is not going to support new JVMs, so we would need an external contributor to provide the fix. And I'm not sure if it's even possible.

Another solution would be to make EasyMock work as a Java Module, declaring its dependencies; then all users would have to do is to add the --add-opens java.base/java.lang=org.easymock argument when launching the test JVM. All accesses would be checked, except those from Easymock.

I think moving from cglib to bytebuddy would work too, and would be a more long-term solution, but that would obviously require much more work.

How to reproduce

Within your Easymock workspace:

my_java_home=<put the path to your Java 11 home directory here>
export JAVA_HOME=$my_java_home
export PATH=$my_java_home/bin:$PATH
echo "Java Home: $JAVA_HOME"
echo "Path: $PATH"
java -version
mvn clean install -DargLine='--illegal-access=deny'

[henri-tremblay]

Thanks for the detailed explanation. Indeed, migration to ByteBuddy will be the way to go in the long run. If anyone is interested in doing it, I'm all in.

Cglib is doing a ClassLoader.defineClass which it needs to. It should probably do a MethodHandles.defineClass when on Java 11. And Unsafe.defineClass before that...

@raphw What are your thoughts on that?

[raphw]

With cglib, you can never overcome a general design flaw with regards to the module system what is cglib writing itself into target classes. This does no longer work in modularized code if the code does not declare a dependency onto cglib: http://mydailyjava.blogspot.com/2018/04/jdk-11-and-proxies-in-world-past.html

[henri-tremblay]

@raphw Nice blog. However, I don't get one thing. ByteBuddy will create a proxy that is where? In the caller module? In which package?

To be clear, let's say I'm in JUnit, testing some code in a module. This module is using some dependency (requires). I want to mock it using EasyMock. JUnit will patch the module. It needs to require EasyMock I suppose.

Then, we do ClassLoadingStrategy.UsingLookup.of(MethodHandles.privateLookupIn(SampleClass.class, MethodHandles.lookup())

This code is done in the EasyMock module. The lookup is from EasyMock. It then asks for a private lookup in the dependency module.

My understanding is that the dependency module should be open to EasyMock or it will fail.
Then the proxy can't be in the same package since I'm not patching the dependency.
It will then lose access to package-scope methods.
The class loader will be the one of the proxied class which is fine.

So, what am I missing here?

[raphw]

Byte Buddy will create the proxy class in the class loader that you specify. Typically that is the class loader of the proxied class.

You are right that this requires the proxied class's module to open and read the EasyMock module in this case. But with cglib, it requires to read and open the cglib module instead. With Byte Buddy's approach, at least you need to depend on the using class instead of depending on the underlying code generator.

Alternatively, you can load the class in a EasyMock-specific class loader where this mock class does not need to read the code generator's module either.

[henri-tremblay]

So, to be able to mock, any EasyMock user will need to add loads of --add-opens org.something/org.that=org.easymock?

I agree it's better (not that much better since cglib is shaded and so will be in the org.easymock package anyway).

To load in a specific class loader, it forces the user to use a JUnit rule or runner no?

I feel like going back in 2002...

[raphw]

If you are shading, you end up pretty much with the same result, that is right. Ideally, there would be a way to declare a test module-info but there is nothing like that in the moment. You could register a method handle lookup somewhere programmatically, like in a rule, too to accomplish this, this would be a work around.