feat: Support XPath2.0 to 3.1 #1774

Constantin1489 · 2023-09-07T01:10:58Z

Is XPath 3.1 100% upwardly compatible with XPath 2.0?
What are the differences between versions of XPath (1.0, 2.0, 3.1)

This update allows syntax like this below.(matches and position)
//*[matches(@id,"container")]/section[1]/article[2]/div[2]/table/tbody/tr[position()>3]/td[2]/a[1]

I tested with mine, works well. ~~However, exception catches and so on need to be added. Because I can't test every syntax or exception handling for this app.~~ ex)
(The issue in the image is fixed)

Also because of incompatibility between XPathes(1.0 vs 2.0), something will work differently.

Also, ~~I didn't do tests in the directory, because the tests already failed before I made changes.~~ BTW, I tested this patch with mine, works well.

Closes #1631
Closes #1674
#1773

[Is XPath 3.1 100% upwardly compatible with XPath 2.0?](https://stackoverflow.com/a/62438044/20307768) [What are the differences between versions of XPath (1.0, 2.0, 3.1)](https://stackoverflow.com/a/51377660/20307768)

changedetectionio/forms.py

dgtlmoon · 2023-09-07T06:25:21Z

very cool, running tests now

any chance you can add a test for something like //*[contains(@id,'post')]/div/div[2]/header/h2/a/text() | //*[contains(@id,'post')]/div/div[2]/header/h2/a/@href ?

just copy def test_check_with_prefix_include_filters to a new def test_xpath_20 in tests/test_xpath_selector.py

dgtlmoon · 2023-09-07T06:26:09Z

, because the tests already failed before I made changes.

That is not correct because it passes here on github, so yeah, super important to add a tests or it cant be merged

dgtlmoon · 2023-09-07T06:36:35Z

@Constantin1489 interesting

        res = client.post(
            url_for("edit_page", uuid="first"),
            data={"include_filters": "/something horrible", "url": test_url, "tags": "", "headers": "", 'fetch_backend': "html_requests"},
            follow_redirects=True
        )
>       assert b"is not a valid XPath expression" in res.data

So it should throw an exception if the xpath string is invalid right? I wonder

Constantin1489 · 2023-09-07T12:16:35Z

Okay, I do it. Please, give me few days

[elementpath exception classes](https://elementpath.readthedocs.io/en/latest/xpath_api.html#exception-classes)

Constantin1489 · 2023-09-07T20:28:15Z

@dgtlmoon e295a17 fixes this. You don't have to take any action.

Constantin1489 · 2023-09-07T20:48:46Z

Is there a method to run tests locally? Or is it just a port conflict?

dgtlmoon · 2023-09-07T20:50:49Z

Is there a method to run tests locally? Or is it just a port conflict?

cd changedetectionio
pytest tests/test_foobar.py

Constantin1489 · 2023-09-07T20:55:24Z

Yes, I did it before. I cannot configure it correctly.

FAILED tests/test_xpath_selector.py::test_setup - AttributeError: Can't pickle local object 'LiveServer.start.<locals>.worker'
FAILED tests/test_xpath_selector.py::test_check_xpath_text_function_utf8 - assert b'<div class="">Stock Alert (UK): RPi CM4' in b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>500 Internal Server Error</title>\n<h1>Internal Server ...
FAILED tests/test_xpath_selector.py::test_check_with_prefix_include_filters - assert b'Some text thats the same' in b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>500 Internal Server Error</title>\n<h1>Internal Server Error</h1>\n<p>...
FAILED tests/test_xpath_selector.py::test_xpath_20 - assert b'Some text thats the same' in b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>500 Internal Server Error</title>\n<h1>Internal Server Error</h1>\n<p>...
==================================================================== 4 failed, 3 passed, 5 warnings in 35.93s =====================================================================
Shutting down datastore thread

I tried it on Rocky os 9(RHEL) it also failed.

=================================================================== 111 failed, 10 passed, 8 warnings in 48.47s ===================================================================

dgtlmoon · 2023-09-07T21:01:44Z

I cant help there sorry, i never tried in mac or windows

Constantin1489 · 2023-09-07T21:46:07Z

@dgtlmoon I added a 09b6203 test for the syntax. (#1774 (comment)).

Explanations will be commented on there.

Constantin1489 · 2023-09-07T21:49:11Z

changedetectionio/tests/test_xpath_selector.py

+
+    res = client.post(
+        url_for("edit_page", uuid="first"),
+        data={"include_filters": "//*[contains(@class, 'sametext')]|//*[contains(@class, 'changetext')]",


Parsing with the union operator. The operator is supported since 1.0.

//*[contains(@class, 'sametext')] //*[contains(@class, 'changetext')]

will parse like

<div class="sametext">Some text thats the same</div> <div class="sametext">Some text thats the same</div> <div class="changetext">Some new text</div> <div class="changetext">Some new text</div>

But the operator//*[contains(@class, 'sametext')]|//*[contains(@class, 'changetext')] will parse like this

<div class="sametext">Some text thats the same</div> <div class="changetext">Some new text</div> <div class="sametext">Some text thats the same</div> <div class="changetext">Some new text</div>

Does this mean that it affects existing xpaths that people have saved?

@Constantin1489

No. It won't. It is just an xpath syntax explanation. I didn't explain here enough. I'm sorry for slabs.

Let's say modified.html contains the content below.

$cat modified.html <html> <body> <p>Which is across multiple lines</p> <div class="sametext">Some text thats the same</div> <div class="changetext">Some new text</div> <div class="sametext">Some text thats the same</div> <div class="changetext">Some new text</div> </body> </html>

basex is cli xpath parser.
If the user uses the XPath union syntax, the result is below.

$basex -i modified.html "//*[contains(@class, 'sametext')]|//*[matches(@class, 'changetext')]" <div class="sametext">Some text thats the same</div> <div class="changetext">Some new text</div> <div class="sametext">Some text thats the same</div> <div class="changetext">Some new text</div>

In your program, without the union syntax,

//*[contains(@class, 'sametext')] //*[contains(@class, 'changetext')]

will show

<div class="sametext">Some text thats the same</div> <div class="sametext">Some text thats the same</div> <div class="changetext">Some new text</div> <div class="changetext">Some new text</div>

Constantin1489 · 2023-09-07T21:50:59Z

changedetectionio/tests/test_xpath_selector.py

+    )
+
+    assert b"Some text thats the same" in res.data #in selector
+    assert b"Some text that will change" in res.data #in selector


Because the include_filters contains //*[contains(@class, 'changetext')]

changedetectionio/tests/test_xpath_selector.py

Constantin1489 · 2023-09-09T15:53:29Z

@dgtlmoon, ~~Do not merge this commit until further notice, please.~~
#1674

…ult early if it is not a list type.

Constantin1489 · 2023-09-09T21:42:31Z

@dgtlmoon I fixed problems only XPath2.0-3.1 related. If tests pass, it is okay to publish.

changedetectionio/templates/edit.html

changedetectionio/blueprint/tags/templates/edit-tag.html

changedetectionio/tests/test_xpath_selector.py

…e and other XPath2.0-3.1 functions.

changedetectionio/store.py

Constantin1489 · 2023-10-30T16:36:21Z

@dgtlmoon please, review the new commit!

dgtlmoon · 2023-11-06T08:32:22Z

@Constantin1489 I found a bug, when using xpath1: it does not validate the rule anymore

dgtlmoon · 2023-11-06T08:38:35Z

wachete uses xpath2 also in their export #1922

Constantin1489 · 2023-11-06T11:13:33Z

Oh, thank you for the bug! That is easy!
related with git show da923352371fe5e121bb321efb390501c4547871 changedetectionio/forms.py
2. I'm currently digging my PR with git log -p --author=Constantin.
We need except etree.XPathEvalError as e: should be back, too!
It will be except (elementpath.ElementPathError, etree.XPathEvalError) as e
https://stackoverflow.com/a/24338247/20307768

2-2.
[ ] fix exception handling except etree.XPathEvalError as e: in changedetectionio/forms.py

2-3.
Previous if line.strip()[0] == '/': in class ValidateCSSJSONXPATHInput(object): of changedetectionio/forms.py
will be changed to cover xpath1:, xpath: too in new my commit(I will add it soon). Is there a special reason you didn't do that?

We need to revive the tests for xpath1 too! Is it an easy way, without copy and paste all test code??

dgtlmoon · 2023-11-06T11:48:27Z

2-3. Previous if line.strip()[0] == '/': in class ValidateCSSJSONXPATHInput(object): of changedetectionio/forms.py will be changed to cover xpath1:, xpath: too in new my commit(I will add it soon). Is there a special reason you didn't do that?

Not sure I understand, xpath: should be now xpath2 and xpath1 should be xpath1

We need to revive the tests for xpath1 too! Is it an easy way, without copy and paste all test code??

please just copy test_xpath_validation to test_xpath1_validation and I will clean up the code later

Constantin1489 · 2023-11-06T12:08:00Z

for 2-3.
It will become
if line.strip()[0] == '/' or line.strip().startswith('xpath:'):
and new block is
if line.strip().statswith('xpath1:'):

to validate

for 3. Okay!

Constantin1489 · 2023-11-06T13:42:55Z

I added! Would you please check what I missed?

dgtlmoon · 2023-11-06T14:22:30Z

I just realised the VisualSelector needs checking if it still works with xpath1: style syntax also, might need a tweak

Constantin1489 · 2023-11-06T17:30:01Z

Sorry, I didn't understand. Would you explain it to me?

Constantin1489 · 2023-11-08T20:14:01Z

I considered why you wanted to add xpath1 on the visual selector.

I think you are worried about if a user selects an element on the visual selector tab but the result won't show the same as xpath1. I don't think that can happen.

The visual selector represents an element as an element or an element with a number.

Let's test it in ycombinator.

<td class="title"><span class="titleline"><a href="https://ooni.org/" rel="nofollow noreferrer">Open Observatory of Network Interference</a><span class="sitebit comhead"> (<a href="from?site=ooni.org"><span class="sitestr">ooni.org</span></a>)</span></span></td>

For this let's test /html/body/center/table/tbody/tr[3]/td/table/tbody/tr[31]/td[3].
this is the result

Open Observatory of Network Interference ( ooni.org )

when we test two kind of xpath,

/html/body/center/table/tbody/tr[3]/td/table/tbody/tr[31]/td[3]
/html/body/center/table/tbody/tr[3]/td/table/tbody/tr[31]/td
xpath1:/html/body/center/table/tbody/tr[3]/td/table/tbody/tr[31]/td[3]
xpath1:/html/body/center/table/tbody/tr[3]/td/table/tbody/tr[31]/td
/html/body/center/table/tbody/tr[3]/td/table/tbody/tr[31]/td[3]
/html/body/center/table/tbody/tr[3]/td/table/tbody/tr[31]/td

result is

Open Observatory of Network Interference ( ooni.org ) 11. 
Open Observatory of Network Interference ( ooni.org ) Open Observatory of Network Interference ( ooni.org ) 11.
Open Observatory of Network Interference ( ooni.org ) Open Observatory of Network Interference ( ooni.org ) 11.
Open Observatory of Network Interference ( ooni.org )

So I don't believe there will be a different result for the visual selection because it is just a basic syntax.

Just for sure,
//*/text() can be different.(I don't know where the reason came from between xpath1 vs xpath2 or elementpath) but not that significant.

Also, this kind of syntax won't show a difference.

xpath1://tr[*]/td[3]/a[1]
//tr[*]/td[3]/a[1]

Constantin1489 · 2023-11-09T12:37:22Z

See also "https://books.google.co.kr/books?id=zgV5MlhlyigC&pg=PA601&lpg=PA601&dq=%22redeﬁning+path+expressions+to+return+a+sequence+of+nodes+in+document+order+with+no+duplicates,+the+result+is+effectively+the+same.%22&source=bl&ots=9iIJBqzEEV&sig=ACfU3U3-RyFTwBjSnkisMUQ3am4aVOSDPw&hl=en&sa=X&ved=2ahUKEwi8sP6b97aCAxWRHnAKHcspAsIQ6AF6BAgKEAM#v=onepage&q=%22redeﬁning%20path%20expressions%20to%20return%20a%20sequence%20of%20nodes%20in%20document%20order%20with%20no%20duplicates%2C%20the%20result%20is%20effectively%20the%20same.%22&f=false"

Constantin1489 · 2023-11-09T14:54:40Z

See also, https://www.w3.org/TR/xpath20/#id-backwards-compatibility
In this ref, no mention related to CD's visual selector.

dgtlmoon · 2023-11-12T15:30:19Z

I think this is going to break functionality for atleast some people out there who have weird/tricky xPath1 selectors.. but CD absolutely needs xPath2/3 so we just have to soldier on :)

dgtlmoon · 2023-11-20T12:13:24Z

@Constantin1489

I just got this :)

Constantin1489 · 2023-11-20T13:23:08Z

@dgtlmoon that is a really great book. The author is one of the editors of XSLT2.0, 3.0 spec in W3C. He is also the editor of XPath2.0 reference of W3C!
He is an active user in StackOverFlow! If you ask a question related to XPath or XSLT, you will have a good chance to encounter him.
Yes, the book is written with authority.

dgtlmoon · 2023-12-05T08:15:42Z

@Constantin1489 I think we can say that this went well in the end! thanks! havent got any large bug reports about problems caused by this

Constantin1489 · 2023-12-05T10:19:46Z

@dgtlmoon the update_14 was a great choice! Thank you for your patience, too!

dgtlmoon · 2024-02-06T13:29:03Z

@Constantin1489 over at - #2164 something changed in elementpath, or one of the queries in your test was actually invalid but is not valid, or something... ( tests/test_xpath_selector_unit.py )

Constantin1489 · 2024-02-06T13:57:31Z

Thank you for the report. I will contact the repo.

Constantin1489 · 2024-02-06T14:57:29Z

I've just tested if (count(//hotel/branch/staff) = 5) then true() else false() and if (count(/hotel/branch/staff) = 5) then true() else false(), using other tools(basex and xidel), they returned true. I think I will create some test cases and report them to the repo to discuss. Thank you @dgtlmoon !

dgtlmoon · 2024-02-07T15:58:44Z

@Constantin1489 amazing thank you :)

feat: Support XPath2.0-3.1

da92335

[Is XPath 3.1 100% upwardly compatible with XPath 2.0?](https://stackoverflow.com/a/62438044/20307768) [What are the differences between versions of XPath (1.0, 2.0, 3.1)](https://stackoverflow.com/a/51377660/20307768)

Constantin1489 commented Sep 7, 2023

View reviewed changes

changedetectionio/forms.py Show resolved Hide resolved

Constantin1489 added 2 commits September 8, 2023 03:02

forms/fix: Catch elementpath lib exceptions.

e295a17

[elementpath exception classes](https://elementpath.readthedocs.io/en/latest/xpath_api.html#exception-classes)

test_xpath_selector/test: Add xpath union operator test(1.0-3.1)

09b6203

Constantin1489 commented Sep 7, 2023

View reviewed changes

Constantin1489 commented Sep 8, 2023

View reviewed changes

changedetectionio/tests/test_xpath_selector.py Outdated Show resolved Hide resolved

Constantin1489 mentioned this pull request Sep 8, 2023

Dockerfile/fix: Update builder and runner to Python 3.11 #1781

Merged

html_tools/fix: Support other type of elementpath results. return res…

d9f5000

…ult early if it is not a list type.

Constantin1489 force-pushed the element_lib branch 2 times, most recently from 4a75940 to 79cf984 Compare September 9, 2023 21:23

Constantin1489 commented Sep 9, 2023

View reviewed changes

changedetectionio/templates/edit.html Show resolved Hide resolved

dgtlmoon reviewed Sep 9, 2023

View reviewed changes

changedetectionio/blueprint/tags/templates/edit-tag.html Show resolved Hide resolved

dgtlmoon reviewed Sep 9, 2023

View reviewed changes

changedetectionio/tests/test_xpath_selector.py Outdated Show resolved Hide resolved

dgtlmoon reviewed Sep 9, 2023

View reviewed changes

changedetectionio/tests/test_xpath_selector.py Outdated Show resolved Hide resolved

Constantin1489 added 2 commits September 10, 2023 10:18

test_xpath_selector/test: Add elementpath tests for int type, str typ…

7c0d9db

…e and other XPath2.0-3.1 functions.

docs: Resolve syntax conflicts between CSS selector and XPath.

20e2cde

Constantin1489 force-pushed the element_lib branch from 79cf984 to 20e2cde Compare September 10, 2023 01:19

Constantin1489 commented Oct 30, 2023

View reviewed changes

changedetectionio/store.py Outdated Show resolved Hide resolved

Constantin1489 commented Oct 30, 2023

View reviewed changes

changedetectionio/store.py Outdated Show resolved Hide resolved

fix: tone and manner

f8bb4e4

Constantin1489 added 3 commits November 6, 2023 21:24

test: add test for test_xpath1_validation

c378d2c

forms/fix: Catch lxml xpath1 lib exceptions.

ec88d4d

test: add test for test_xpath23_prefix_validation

d0b7770

dgtlmoon changed the title ~~feat: Support XPath2.0-3.1~~ feat: Support XPath2.0 to 3.1 Nov 6, 2023

jgupta mentioned this pull request Nov 10, 2023

[feature] custom CSS/xPath filters, not made in visual selector, should work in visual selector #1826

Open

dgtlmoon merged commit 26931e0 into dgtlmoon:master Nov 13, 2023
3 checks passed

feat: Support XPath2.0 to 3.1 #1774

feat: Support XPath2.0 to 3.1 #1774

Conversation

Constantin1489 commented Sep 7, 2023 • edited

dgtlmoon commented Sep 7, 2023

dgtlmoon commented Sep 7, 2023

dgtlmoon commented Sep 7, 2023

Constantin1489 commented Sep 7, 2023

Constantin1489 commented Sep 7, 2023

Constantin1489 commented Sep 7, 2023 • edited

dgtlmoon commented Sep 7, 2023

Constantin1489 commented Sep 7, 2023 • edited

dgtlmoon commented Sep 7, 2023

Constantin1489 commented Sep 7, 2023

Constantin1489 Sep 7, 2023 • edited

Choose a reason for hiding this comment

dgtlmoon Sep 8, 2023

Choose a reason for hiding this comment

dgtlmoon Sep 8, 2023

Choose a reason for hiding this comment

Constantin1489 Sep 8, 2023 • edited

Choose a reason for hiding this comment

Constantin1489 Sep 7, 2023

Choose a reason for hiding this comment

Constantin1489 commented Sep 9, 2023 • edited

Constantin1489 commented Sep 9, 2023

Constantin1489 commented Oct 30, 2023

dgtlmoon commented Nov 6, 2023

dgtlmoon commented Nov 6, 2023

Constantin1489 commented Nov 6, 2023 • edited

dgtlmoon commented Nov 6, 2023

Constantin1489 commented Nov 6, 2023 • edited

Constantin1489 commented Nov 6, 2023

dgtlmoon commented Nov 6, 2023

Constantin1489 commented Nov 6, 2023

Constantin1489 commented Nov 8, 2023 • edited

Constantin1489 commented Nov 9, 2023

Constantin1489 commented Nov 9, 2023

dgtlmoon commented Nov 12, 2023 • edited

dgtlmoon commented Nov 20, 2023 • edited

Constantin1489 commented Nov 20, 2023 • edited

dgtlmoon commented Dec 5, 2023

Constantin1489 commented Dec 5, 2023

dgtlmoon commented Feb 6, 2024 • edited

Constantin1489 commented Feb 6, 2024

Constantin1489 commented Feb 6, 2024 • edited

dgtlmoon commented Feb 7, 2024

Constantin1489 commented Sep 7, 2023 •

edited

Constantin1489 commented Sep 7, 2023 •

edited

Constantin1489 commented Sep 7, 2023 •

edited

Constantin1489 Sep 7, 2023 •

edited

Constantin1489 Sep 8, 2023 •

edited

Constantin1489 commented Sep 9, 2023 •

edited

Constantin1489 commented Nov 6, 2023 •

edited

Constantin1489 commented Nov 6, 2023 •

edited

Constantin1489 commented Nov 8, 2023 •

edited

dgtlmoon commented Nov 12, 2023 •

edited

dgtlmoon commented Nov 20, 2023 •

edited

Constantin1489 commented Nov 20, 2023 •

edited

dgtlmoon commented Feb 6, 2024 •

edited

Constantin1489 commented Feb 6, 2024 •

edited