Unknown error in XPath. #385
Comments
|
Sounds reasonable but maybe something else.... https://sourceforge.net/p/htmlunit/mailman/message/23898002/ |
|
I saw this thread, but we already have a higher version of the HTMLunit (v.2.29) and there the suggestion is to use 2.7 which is so much older. |
|
the code looks like this (xalan) pos can't be null here because it is used before and a local var. Can you please check your stack trace - maybe the NPE is inside getNextSibling? |
|
I have the same issue in the latest version 2.56.0. The stacktrace I'm seeing is: Need to have a look at the xalan code base. |
|
We are encountering the same issue intermittently with exactly the same stacktrace (HTMLunit v.2.62.0), are there any workaround/solution? |
In our case |
|
org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.addNode(DOM2DTM.java:215) looks here like this Can you please post your stack trace. |
yes, we're using xalan-2.7.2 |
|
Ah ok, the location is Will think about it again - maybe we can get rid of xalan at all. |
|
@Pldi23 can imagine two possible scenarios
What do you think? |
|
@Pldi23 can you debug the problem? |
Hard to say, I'll try to debug, but the issue happens very rarely and there no obvious way how to make it more reproducible. |
I understand, but without having more info it is more or less luck to find this. |
|
Hm, sounds good, I'm not sure if we could update somehow our ci infrastructure to use it, but at least I could use it locally I guess. |
|
Ok I caught it, this is what I see on a debug right now (didn't analyze it yet) |
|
Just as assumption, could there be a race condition between 2 |
I think not, they are processed in sequence and also the results are copied into list. |
|
From your screenshot it looks like some code is executed during page parsing; usually a script tag. My guess is more code like com.gargoylesoftware.htmlunit.html.DomNode.basicInsertBefore(DomNode) where the node is already attached to the dom before e.g. the parent is set. If you like i can try to reorder some of this and make a new snapshot build. |
If you have fix idea you could also make changes in a draft PR for example and I'll build it and substitute jar locally and setup a our test for a thousands run for the night. |
|
a new snapshot build 2.63.0-SNAPSHOT is available - let's cross our fingers |
|
@rbri I ran our test with a your new build 1500 times and it failed again on 1497 attempt. (As I see your fix improve the flakiness, because statistically before your fix failures happened ~ 1 per 250 attempts, but didn't solve it completely) |
|
@Pldi23 you mean it failed only for 3 runs out of the 1500 ones? Let me think a bit more about this - but for a first wild guess the results are not that bad :-D |
|
I mean it failed once in 1500 runs |
|
@Pldi23 and if you run it only 1496 times :-D Will check the code - maybe i can spot more locations to improve |
|
@Pldi23 made another snapshot build... |
|
@rbri Thanks, with the same name |
|
@Pldi23 yes same name |
|
Could you please check, (perhaps I'm paranoying ;)), but I don't see your latest commit in |
|
Really strange but it looks you are right. |
|
Checked again, weird, but latest commit are still not there. |
|
I looked inside and |
|
Thank you @rbri , you are right (for some reason maven reloaded htmlunit-2.63.0-20220704.141031-4 every time). Now I get latest build and I'll get back to you after next 1500 runs :) |
|
for maven the -U switch helps here but who knows what all the proxies and content delivery networks out there are doing |
|
@rbri Oh, unfortunately I'm observing a much more failures in this batch of runs, approximately 1 per 100 fails. |
|
@Pldi23 no idea why the results are bad based on the last change - maybe the problem depends on something else. |
|
@Pldi23 have reverted the DomElement changes and made a new snapshot build |
|
have another idea to reduce the problem.... working on it |
|
Good morning, From your stack trace we can see the xpath call (from HtmlRadioButton) is done during the parsing of the page. Usually this is the time where the dom is in manipulation and maybe some nodes are not in a consistent state. Because of this i think using XPath inside the HtmlUnit implementation is a bad idea and i have replaced the calls. I expect one of two possible results
@Pldi23 waiting for your results .... |
|
Hi @rbri , unfortunately my local night runs still fails (5 times in 1500 runs). |
|
@rbri good news, no failures in 1500 runs with latest build 🎉🎉🎉! You're a hero of the day! |
|
before starting the celebration i think you have to do some more tuns to be sure - maybe it was only luck this time |
|
Sure, I setup more runs |
|
btw any change in terms of performance? |
|
Hard to measure (every test running ~8-10 sec and depends of how much load my PC currently and I don't have any tools like dynatrace connected locally) but visually looking on timing I would say performance didn't get any worse |
|
As always i had some hope the changes making the monster faster :-) |
|
based on my code review of your changes (with my limited knowledges of your library) performance should be improved :) And yes, test is still consistently green! |
|
Ok time to switch of the computer and think about the next steps - do you need a release of this? |
|
I don't want to influence your release strategy, but yes, we need the fix in the nearest release. |
|
There is no real fixed strategy outside making one nearly every month. Will have a look if i can make one this weekend. |
|
Will close this now because it seems to be fixed for at least this situation. Please open a new one if you facing similar problems. |
Caused by: javax.xml.transform.TransformerException: Unknown error in XPath.
at com.gargoylesoftware.htmlunit.html.xpath.XPathAdapter.execute(XPathAdapter.java:188)
at com.gargoylesoftware.htmlunit.html.xpath.XPathUtils.evaluateXPath(XPathUtils.java:145)
at com.gargoylesoftware.htmlunit.html.xpath.XPathUtils.getByXPath(XPathUtils.java:77)
... 51 more
Caused by: java.lang.NullPointerException
at org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.nextNode(DOM2DTM.java:390)
at org.apache.xml.dtm.ref.DTMDefaultBase._nextsib(DTMDefaultBase.java:565)
at org.apache.xml.dtm.ref.DTMDefaultBase.getNextSibling(DTMDefaultBase.java:1142)
at org.apache.xml.dtm.ref.DTMDefaultBaseTraversers$ChildTraverser.next(DTMDefaultBaseTraversers.java:463)
at org.apache.xpath.axes.AxesWalker.getNextNode(AxesWalker.java:335)
at org.apache.xpath.axes.AxesWalker.nextNode(AxesWalker.java:363)
at org.apache.xpath.axes.WalkingIterator.nextNode(WalkingIterator.java:195)
at org.apache.xpath.axes.NodeSequence.nextNode(NodeSequence.java:335)
at org.apache.xpath.axes.NodeSequence.runTo(NodeSequence.java:494)
at org.apache.xpath.axes.NodeSequence.setRoot(NodeSequence.java:270)
at org.apache.xpath.axes.LocPathIterator.execute(LocPathIterator.java:212)
at com.gargoylesoftware.htmlunit.html.xpath.XPathAdapter.execute(XPathAdapter.java:167)
... 53 more
We are encountering the aforementioned issue randomly. Most of the time it works fine. We are assuming that this is caused by the fact that XPath object is not designed for multithreading access. I have tried searching for information about this issue however I did not find any solution/workaround, is there any way to fix this?
The text was updated successfully, but these errors were encountered: