Unknown error in XPath.

[mateuszskorupa96]

Caused by: javax.xml.transform.TransformerException: Unknown error in XPath.
at com.gargoylesoftware.htmlunit.html.xpath.XPathAdapter.execute(XPathAdapter.java:188)
at com.gargoylesoftware.htmlunit.html.xpath.XPathUtils.evaluateXPath(XPathUtils.java:145)
at com.gargoylesoftware.htmlunit.html.xpath.XPathUtils.getByXPath(XPathUtils.java:77)
... 51 more
Caused by: java.lang.NullPointerException
at org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.nextNode(DOM2DTM.java:390)
at org.apache.xml.dtm.ref.DTMDefaultBase._nextsib(DTMDefaultBase.java:565)
at org.apache.xml.dtm.ref.DTMDefaultBase.getNextSibling(DTMDefaultBase.java:1142)
at org.apache.xml.dtm.ref.DTMDefaultBaseTraversers$ChildTraverser.next(DTMDefaultBaseTraversers.java:463)
at org.apache.xpath.axes.AxesWalker.getNextNode(AxesWalker.java:335)
at org.apache.xpath.axes.AxesWalker.nextNode(AxesWalker.java:363)
at org.apache.xpath.axes.WalkingIterator.nextNode(WalkingIterator.java:195)
at org.apache.xpath.axes.NodeSequence.nextNode(NodeSequence.java:335)
at org.apache.xpath.axes.NodeSequence.runTo(NodeSequence.java:494)
at org.apache.xpath.axes.NodeSequence.setRoot(NodeSequence.java:270)
at org.apache.xpath.axes.LocPathIterator.execute(LocPathIterator.java:212)
at com.gargoylesoftware.htmlunit.html.xpath.XPathAdapter.execute(XPathAdapter.java:167)
... 53 more

We are encountering the aforementioned issue randomly. Most of the time it works fine. We are assuming that this is caused by the fact that XPath object is not designed for multithreading access. I have tried searching for information about this issue however I did not find any solution/workaround, is there any way to fix this?

[rbri]

Sounds reasonable but maybe something else.... https://sourceforge.net/p/htmlunit/mailman/message/23898002/

[mateuszskorupa96]

I saw this thread, but we already have a higher version of the HTMLunit (v.2.29) and there the suggestion is to use 2.7 which is so much older.

[rbri]

the code looks like this (xalan)

next = pos.getNextSibling();

pos can't be null here because it is used before and a local var.

Can you please check your stack trace - maybe the NPE is inside getNextSibling?

[jometzner]

I have the same issue in the latest version 2.56.0. The stacktrace I'm seeing is:

Caused by: javax.xml.transform.TransformerException: Unknown error in XPath.
        at com.gargoylesoftware.htmlunit.html.xpath.XPathAdapter.execute(XPathAdapter.java:235)
        at com.gargoylesoftware.htmlunit.html.xpath.XPathHelper.evaluateXPath(XPathHelper.java:137)
        at com.gargoylesoftware.htmlunit.html.xpath.XPathHelper.getByXPath(XPathHelper.java:71)
        ... 102 more
Caused by: java.lang.NullPointerException
        at org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.addNode(DOM2DTM.java:215)
        at org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.nextNode(DOM2DTM.java:542)
        at org.apache.xml.dtm.ref.DTMDefaultBase._nextsib(DTMDefaultBase.java:565)
        at org.apache.xml.dtm.ref.DTMDefaultBase.getNextSibling(DTMDefaultBase.java:1142)
        at org.apache.xml.dtm.ref.DTMDefaultBaseTraversers$ChildTraverser.next(DTMDefaultBaseTraversers.java:463)
        at org.apache.xpath.axes.AxesWalker.getNextNode(AxesWalker.java:335)
        at org.apache.xpath.axes.AxesWalker.nextNode(AxesWalker.java:363)
        at org.apache.xpath.axes.WalkingIterator.nextNode(WalkingIterator.java:195)
        at org.apache.xpath.axes.NodeSequence.nextNode(NodeSequence.java:335)
        at org.apache.xpath.axes.NodeSequence.runTo(NodeSequence.java:494)
        at org.apache.xpath.axes.NodeSequence.setRoot(NodeSequence.java:270)
        at org.apache.xpath.axes.LocPathIterator.execute(LocPathIterator.java:212)
        at com.gargoylesoftware.htmlunit.html.xpath.XPathAdapter.execute(XPathAdapter.java:214)
        ... 104 more

Need to have a look at the xalan code base.

[Pldi23]

We are encountering the same issue intermittently with exactly the same stacktrace (HTMLunit v.2.62.0), are there any workaround/solution?

[Pldi23]

the code looks like this (xalan)

next = pos.getNextSibling();

pos can't be null here because it is used before and a local var.

Can you please check your stack trace - maybe the NPE is inside getNextSibling?

In our case pos = null what is triggered the error stacktrace, the issue is flaky.

[rbri]

@Pldi23

org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.addNode(DOM2DTM.java:215)

looks here like this

type = node.getNodeType();

Can you please post your stack trace.
Do you have xalan-2.7.2?

[Pldi23]

@rbri

Can you please post your stack trace.

java.lang.RuntimeException: Could not retrieve XPath >//input[lower-case(@type)='radio' and @name='tab-group-1656481419909']< on HtmlPage(http://localhost:42685/jenkins/)@398295463
	at com.gargoylesoftware.htmlunit.html.xpath.XPathHelper.getByXPath(XPathHelper.java:93)
	at com.gargoylesoftware.htmlunit.html.DomNode.getByXPath(DomNode.java:1584)
	at com.gargoylesoftware.htmlunit.html.HtmlRadioButtonInput.setCheckedForPage(HtmlRadioButtonInput.java:181)
	at com.gargoylesoftware.htmlunit.html.HtmlRadioButtonInput.setChecked(HtmlRadioButtonInput.java:139)
	at com.gargoylesoftware.htmlunit.html.HtmlRadioButtonInput.onAddedToPage(HtmlRadioButtonInput.java:279)
	at com.gargoylesoftware.htmlunit.html.HtmlPage.notifyNodeAdded(HtmlPage.java:1771)
	at com.gargoylesoftware.htmlunit.html.DomNode.fireAddition(DomNode.java:1060)
	at com.gargoylesoftware.htmlunit.html.DomNode.appendChild(DomNode.java:955)
	at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.appendChild(HtmlUnitNekoDOMBuilder.java:791)
	at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.addNodeToRightParent(HtmlUnitNekoDOMBuilder.java:475)
	at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.startElement(HtmlUnitNekoDOMBuilder.java:359)
	at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
	at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.startElement(HtmlUnitNekoDOMBuilder.java:291)
	at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)
	at net.sourceforge.htmlunit.cyberneko.HTMLTagBalancer.startElement(HTMLTagBalancer.java:812)
	at net.sourceforge.htmlunit.cyberneko.filters.DefaultFilter.startElement(DefaultFilter.java:140)
	at net.sourceforge.htmlunit.cyberneko.filters.NamespaceBinder.startElement(NamespaceBinder.java:278)
	at net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scanStartElement(HTMLScanner.java:2811)
	at net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2131)
	at net.sourceforge.htmlunit.cyberneko.HTMLScanner.scanDocument(HTMLScanner.java:937)
	at net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:443)
	at net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:394)
	at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
	at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.parse(HtmlUnitNekoDOMBuilder.java:768)
	at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoHtmlParser.parse(HtmlUnitNekoHtmlParser.java:204)
	at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:298)
	at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:218)
	at org.jvnet.hudson.test.HudsonPageCreator.createPage(HudsonPageCreator.java:54)
	at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:688)
	at com.gargoylesoftware.htmlunit.WebClient.loadDownloadedResponses(WebClient.java:2547)
	at com.gargoylesoftware.htmlunit.html.HtmlFormUtil.submit(HtmlFormUtil.java:82)
	at org.jvnet.hudson.test.JenkinsRule$WebClient.login(JenkinsRule.java:2463)
	at org.jvnet.hudson.test.JenkinsRule$WebClient.login(JenkinsRule.java:2329)
	at org.jvnet.hudson.test.JenkinsRule$WebClient.login(JenkinsRule.java:2475)
	at com.cloudbees.jenkins.plugins.bitbucket.Security2033Test.doClearWhenInvokedUsingGetMethodThenResourceNotFound(Security2033Test.java:157)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:606)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.xml.transform.TransformerException: Unknown error in XPath.
	at com.gargoylesoftware.htmlunit.html.xpath.XPathAdapter.execute(XPathAdapter.java:235)
	at com.gargoylesoftware.htmlunit.html.xpath.XPathHelper.evaluateXPath(XPathHelper.java:137)
	at com.gargoylesoftware.htmlunit.html.xpath.XPathHelper.getByXPath(XPathHelper.java:71)
	... 48 more
Caused by: java.lang.NullPointerException
	at org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.nextNode(DOM2DTM.java:390)
	at org.apache.xml.dtm.ref.DTMDefaultBase._nextsib(DTMDefaultBase.java:565)
	at org.apache.xml.dtm.ref.DTMDefaultBase.getNextSibling(DTMDefaultBase.java:1142)
	at org.apache.xml.dtm.ref.DTMDefaultBaseTraversers$ChildTraverser.next(DTMDefaultBaseTraversers.java:463)
	at org.apache.xpath.axes.AxesWalker.getNextNode(AxesWalker.java:335)
	at org.apache.xpath.axes.AxesWalker.nextNode(AxesWalker.java:363)
	at org.apache.xpath.axes.WalkingIterator.nextNode(WalkingIterator.java:195)
	at org.apache.xpath.axes.NodeSequence.nextNode(NodeSequence.java:335)
	at org.apache.xpath.axes.NodeSequence.runTo(NodeSequence.java:494)
	at org.apache.xpath.axes.NodeSequence.setRoot(NodeSequence.java:270)
	at org.apache.xpath.axes.LocPathIterator.execute(LocPathIterator.java:212)
	at com.gargoylesoftware.htmlunit.html.xpath.XPathAdapter.execute(XPathAdapter.java:214)
	... 50 more

Do you have xalan-2.7.2?

yes, we're using xalan-2.7.2

[rbri]

Ah ok, the location is

org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.nextNode(DOM2DTM.java:390)

Will think about it again - maybe we can get rid of xalan at all.
PR's are welcome....

[rbri]

@Pldi23 can imagine two possible scenarios

• your code (the XPath selection) works in parallel to some js code that manipulates the dom and the xpath engine works on some dom node in an intermediate state
• there is some code that produces a corrupt dom node that leads to this error

What do you think?

[rbri]

@Pldi23 can you debug the problem?

[Pldi23]

@Pldi23 can imagine two possible scenarios

your code (the XPath selection) works in parallel to some js code that manipulates the dom and the xpath engine works on some dom node in an intermediate state
there is some code that produces a corrupt dom node that leads to this error
What do you think?

Hard to say, I'll try to debug, but the issue happens very rarely and there no obvious way how to make it more reproducible.

[rbri]

I'll try to debug, but the issue happens very rarely and there no obvious way how to make it more reproducible.

I understand, but without having more info it is more or less luck to find this.
If i can make a xalan version with some debug code inside - will you be able to use this?

[Pldi23]

Hm, sounds good, I'm not sure if we could update somehow our ci infrastructure to use it, but at least I could use it locally I guess.
(By the way, you could also try to run one of our open source tests for example this one bitbucket-branch-source flaky test. It is flaky here on a simple login step when accessing DOM)

[Pldi23]

Ok I caught it, this is what I see on a debug right now (didn't analyze it yet)