Struts 2 s:fielderror not working in Java 8, WebSphere

[dougbreaux]

(Original thread in Struts mailing list)

Struts 2.5.14.1, WebSphere 8.5.5.13, IBM Java 8 (level 1.8.0_161) on AIX.

We have a JSP with a plain <s:fielderror /> element, that works without issue when we switch WebSphere back to Java 7, but in Java 8 produces the following error:

2018-05-22 12:43:24,384 ERROR [freemarker.runtime]: error - Error executing FreeMarker template
FreeMarker template error:
Java method "AbstractCollection.size()" threw an exception when invoked on LinkedHashMap$LinkedKeySet object "[]"; see cause exception in the Java stack trace.

----
FTL stack trace ("~" means nesting-related):
        - Failed at: #assign eKeysSize = eKeys.size()  [in template "template/simple/fielderror.ftl" at line 23, column 5]
----

Java stack trace (for programmers):
----
freemarker.core._TemplateModelException: [... Exception message was already printed; see it above ...]
        at freemarker.ext.beans._MethodUtil.newInvocationTemplateModelException(_MethodUtil.java:289)
        at freemarker.ext.beans._MethodUtil.newInvocationTemplateModelException(_MethodUtil.java:252)
        at freemarker.ext.beans.SimpleMethodModel.exec(SimpleMethodModel.java:74)
        at freemarker.core.MethodCall._eval(MethodCall.java:65)
        at freemarker.core.Expression.eval(Expression.java:81)
        at freemarker.core.Assignment.accept(Assignment.java:134)
        at freemarker.core.Environment.visit(Environment.java:326)
        at freemarker.core.Environment.visit(Environment.java:332)
        at freemarker.core.Environment.visit(Environment.java:332)
        at freemarker.core.Environment.process(Environment.java:305)
        at freemarker.template.Template.process(Template.java:378)
        at org.apache.struts2.components.template.FreemarkerTemplateEngine.renderTemplate(FreemarkerTemplateEngine.java:149)
        at org.apache.struts2.components.UIBean.mergeTemplate(UIBean.java:579)
        at org.apache.struts2.components.UIBean.end(UIBean.java:535)
        at org.apache.struts2.views.jsp.ComponentTagSupport.doEndTag(ComponentTagSupport.java:39)
.. my JSP
..
Caused by: java.lang.AbstractMethodError: java/util/AbstractCollection.size()I
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
        at java.lang.reflect.Method.invoke(Method.java:508)
        at freemarker.ext.beans.BeansWrapper.invokeMethod(BeansWrapper.java:1487)
        at freemarker.ext.beans.SimpleMethodModel.exec(SimpleMethodModel.java:69)

One suggestion was that it could be issue 1892, but nobody was certain, so it was suggested that I open another issue.

[DanHeidinga]

@dougbreaux Are you able to run this with the latest OpenJ9 binary? We need to know if the issue still reproduces and then we can begin to investigate

[dougbreaux]

The latest available for my version of WebSphere? That would be the most I could do. I can certainly ask my operations team to apply that update, but I think it's still SR5 FP15, which doesn't yet have the 1892 fix?

[DanHeidinga]

The Eclipse OpenJ9 and IBM SDKs are being confused in this discussion. This project creates & maintains the Eclipse OpenJ9 JVM. IBM consumes from this project to create their IBM Java SDK.

If you need an IBM Java SDK, you'll need to contact IBM. If you are able to patch your version of Websphere to run with a JDK built from the latest OpenJ9 source, we can see if the problem still reproduces and help debug it. AdoptOpenJDK has nightly binaries that be used.

[dougbreaux]

Ok, I did wonder why I was sent here, but I was just following instructions :-)

That said, I'm pretty sure I and my team won't be up for trying to make our WebSphere run with an OpenJ9 JDK.

[DanHeidinga]

Sorry for the confusion. I can sympathize with the pain of porting Websphere to another JDK.

Is there a smaller test case that demonstrates the problem?

Otherwise, I suggest generating a system core file using the -Xdump:system:events=throw,filter=java/lang/AbstractMethodError dump option so you can see the VM's state at the time of the exception using the jdmpview tool.

[dougbreaux]

Somebody in IBM was able to get us a test WAS iFix with FP 16 in it, which would include the 1892 fix, but regrettably, still has this problem.

/usr/IBM/WebSphere/AppServer/java_1.8_64/bin/java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 8.0.5.16 - pap6480sr5fp16-20180524_01(SR5 FP16))
IBM J9 VM (build 2.9, JRE 1.8.0 AIX ppc64-64-Bit Compressed References 20180523_387150 (JIT enabled, AOT enabled)
OpenJ9 - c81abfb
OMR - f3f8112
IBM - 4f87eeb)
JCL - 20180515_01 based on Oracle jdk8u171-b11

2018-06-05 09:22:28,083 ERROR [freemarker.runtime]: error - Error executing FreeMarker template
FreeMarker template error:
Java method "AbstractCollection.size()" threw an exception when invoked on LinkedHashMap$LinkedKeySet object "[]"; see cause exception in the Java stack trace.

----
FTL stack trace ("~" means nesting-related):
        - Failed at: #assign eKeysSize = eKeys.size()  [in template "template/simple/fielderror.ftl" at line 23, column 5]
----

Java stack trace (for programmers):
----
freemarker.core._TemplateModelException: [... Exception message was already printed; see it above ...]

[DanHeidinga]

@dougbreaux Are you able to add the following commandline to the your run?

-Xdump:system+java:events=throw,filter=java/lang/AbstractMethodError

This will generate both java cores and system cores when the AbstractMethodError is being thrown. The cores will allow us to understand what's happening in the VM at that point and hopefully find a solution.

The cores can be shared privately if you prefer - reach out to me either here or internally and we can work out how to look at them.

[dougbreaux]

I will do that, hopefully tonight.

[dougbreaux]

@DanHeidinga did the trace not reveal anything useful?

[dougbreaux]

Unsurprisingly, my simplistic test from a Struts 2 Maven "blank" archetype doesn't exhibit this behavior, FWIW. So it's not a trivial case.

[DanHeidinga]

@dougbreaux jdmpview can't find a JRE in the core and I've been unsuccessful manually putting things back together.

Can you generate a new core and confirm that jdmpview -core <corefile> is able to find the JRE in the core? If it fails, you'll see something like: *0 : PID: 8323098 : No JRE in the jdmpview output

[dougbreaux]

@DanHeidinga So... why would the next core file not have the same problem?

[dougbreaux]

Ok, in the process of trying to get that dump again, I decided to remove some other JVM arguments we have that are related to an agent installed in the the JVM, and voila, the problem ceased. So I think we can close this, and I'll have to pursue with that product support.