[BUG] conflict with lxml v4.9.1

Describe the bug
After upgrading lxml to v4.9.1 I am having the following issue.

Traceback (most recent call last):
  File "/Users/john/.pyenv/versions/3.10.5/lib/python3.10/site-packages/webchanges/filters.py", line 1220, in _get_filtered_elements
    root = etree.fromstring(self.data, self.parser)  # bandit B320: use defusedxml TODO
  File "src/lxml/etree.pyx", line 3254, in lxml.etree.fromstring
  File "src/lxml/parser.pxi", line 1908, in lxml.etree._parseMemoryDocument
ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/john/.pyenv/versions/3.10.5/bin/webchanges", line 8, in <module>
    sys.exit(main())
  File "/Users/john/.pyenv/versions/3.10.5/lib/python3.10/site-packages/webchanges/cli.py", line 307, in main
    urlwatch_command.run()
  File "/Users/john/.pyenv/versions/3.10.5/lib/python3.10/site-packages/webchanges/command.py", line 688, in run
    self.handle_actions()
  File "/Users/john/.pyenv/versions/3.10.5/lib/python3.10/site-packages/webchanges/command.py", line 626, in handle_actions
    self.test_job(self.urlwatch_config.test_job)
  File "/Users/john/.pyenv/versions/3.10.5/lib/python3.10/site-packages/webchanges/command.py", line 201, in test_job
    raise job_state.exception
  File "/Users/john/.pyenv/versions/3.10.5/lib/python3.10/site-packages/webchanges/handler.py", line 170, in process
    filtered_data = FilterBase.process(filter_kind, subfilter, self, filtered_data)
  File "/Users/john/.pyenv/versions/3.10.5/lib/python3.10/site-packages/webchanges/filters.py", line 240, in process
    return filtercls(job_state.job, job_state).filter(data, subfilter)
  File "/Users/john/.pyenv/versions/3.10.5/lib/python3.10/site-packages/webchanges/filters.py", line 1313, in filter
    return lxml_parser.get_filtered_data()
  File "/Users/john/.pyenv/versions/3.10.5/lib/python3.10/site-packages/webchanges/filters.py", line 1249, in get_filtered_data
    elements = self._get_filtered_elements()
  File "/Users/john/.pyenv/versions/3.10.5/lib/python3.10/site-packages/webchanges/filters.py", line 1227, in _get_filtered_elements
    root = etree.fromstring(self.data, self.parser)  # bandit B320: use defusedxml TODO
  File "src/lxml/etree.pyx", line 3254, in lxml.etree.fromstring
  File "src/lxml/parser.pxi", line 1913, in lxml.etree._parseMemoryDocument
  File "src/lxml/parser.pxi", line 1793, in lxml.etree._parseDoc
  File "src/lxml/parser.pxi", line 1082, in lxml.etree._BaseParser._parseUnicodeDoc
  File "src/lxml/parser.pxi", line 615, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 725, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 654, in lxml.etree._raiseParseError
  File "<string>", line 1
lxml.etree.XMLSyntaxError: Char 0x0 out of allowed range, line 1, column 2

To Reproduce

Expected behavior

Screen scrape/screenshots

Version info
Please run webchanges -v and paste the version information as follows (first 3 lines):

• webchanges 3.10.2
• Python 3.10.5 or 3.10.3
• System macOS-10.15.7-x86_64-i386-64bit

Additional context
Downgrading to lxml v4.8.0 or v4.9.0 resolves this issue for me.

[mborsetti]

Thanks for the bug report!

I am trying to track down what changed in lxml 4.9.1 and it's non-obvious.

What type of data are you passing to the filter?

If you can even share the YAML for the job that would be wonderful.

Thanks for your patience. This is the job. I have two for the same URL - so I have added #1 and #2 to the end of the respective jobs.

--- #### SwimSwam - working 4/10/22 ####
name: 26_SwimSwam - News [NASA Wildcat Aquatics]
url: "https://swimswam.com/feed/#1"
filter:
  - xpath:
      method: xml
      path: '//item/title/text()|//item/description/text()|//item/content:encoded/text()'
      namespaces:
        content: http://purl.org/rss/1.0/modules/content/
      exclude: 'a'
  - keep_lines_containing:
      re: '\bNASA\sWildcat'
  - html2text:
additions_only: true

[mborsetti]

Great, thanks!

Unfortunately it's not throwing the error locally (on Windows):

C:\>webchanges --jobs jobs-test --test 1
26_SwimSwam - News [NASA Wildcat Aquatics]
------------------------------------------



--
Job tested in 0.57 seconds with webchanges 3.10.2.

C:\>python --version
Python 3.10.5

C:\>pip list | findstr lxml
lxml                          4.9.1

I tried it on a Linux box and no errors there either... ???

I have no programming experience so can't really understand the error messages - does it provide any clues?

This was the last command I ran which brought on this issue. I also updated some homebrew packages the same day

pip install -U cffi charset-normalizer cryptography cssbeautifier importlib-metadata jsbeautifier lxml numpy pandas Pillow playwright redis requests setuptools tabulate typing_extensions websocket-client

The versions of the updated pacakges run by that command:

charset-normalizer 2.1.0
cryptography 37.0.4
cssbeautifier 1.14.4
importlib-metadata 4.12.0
jsbeautifier 1.14.4
lxml 4.9.1
numpy 1.23.1
pandas 1.4.3
playwright 1.23.0
redis 4.3.4
requests 2.28.1
setuptools 63.1.0
tabulate 0.8.10
typing_extensions 4.3.0
websocket-client 1.3.3

Maybe it's something else, but the error indicates the new version of lxml - and downgrading it resolves the issue... and upgrading it causes it to recur.

0-zsh% pip install -U lxml
Requirement already satisfied: lxml in ./.pyenv/versions/3.10.5/lib/python3.10/site-packages (4.9.0)
Collecting lxml
  Downloading lxml-4.9.1.tar.gz (3.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 9.9 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: lxml
  Building wheel for lxml (setup.py) ... done
  Created wheel for lxml: filename=lxml-4.9.1-cp310-cp310-macosx_10_15_x86_64.whl size=1695119 sha256=7f4cd80302ad308d604935721f4bfa95d2ab91fc9696dba6e5aa0f9d48b66c88
  Stored in directory: /Users/john/Library/Caches/pip/wheels/a4/ec/7b/8acde6da24b5aabeee049213d5bec12d1e9214d3cae276387b
Successfully built lxml
Installing collected packages: lxml
  Attempting uninstall: lxml
    Found existing installation: lxml 4.9.0
    Uninstalling lxml-4.9.0:
      Successfully uninstalled lxml-4.9.0
Successfully installed lxml-4.9.1

The is the output after re-installing lxml 4.9.1
Terminal Saved Output 4.9.1.txt
The is the output after installing lxml 4.9.0
Terminal Saved Output 4.9.0.txt

[mborsetti]

Thanks.

Nothing is obvious in the lxml changelog: https://lxml.de/4.9/changes-4.9.1.html.

However, I think I may have found something at https://lxml.de/FAQ.html#why-can-t-lxml-parse-my-xml-from-unicode-strings. The way XML is handled in webchanges (legacy code) does not seem to conform to the way lxml wants it, and maybe with this release they made other changes that broke this nonstandard way. Maybe, or maybe not.

In any case let me work on an interim release and see if my intuition is correct; hard to know since I still can't reproduce the issue.

Don't know if this is meaningful... since you are saying this is not an issue on Windows, one thing I notice in the lxml change log is that 4.90 is being built with, I think, older versions of libxml2 and libxslt on Windows, and newer versions of those binaries for other OS.

Is it possible that whatever changes happened in 4.9.1 were triggered by an initial change in 4.9.0?

[mborsetti]

Yeah, could be, but your job works both on Windows 11 and on Ubuntu 22.04 LTS, so ... who knows what's going on.

I've posted a release candidate in the unreleased branch; would you be so kind as to install it and see if my hunch is correct and it does fix your issue?

pip install https://github.com/mborsetti/webchanges/archive/unreleased.tar.gz

Thanks

Thank you so much for your efforts. Wish I could be of more help.

I installed the Release Candidate but before I could upgrade lxml to 4.9.1 to test-run the jobs I mention above ( #26, #27 ), three other jobs ( #30, #31, #32 ) ran and came back with errors... these did not have an issue before installing the RC.

ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.

Good news is that the initial issue with jobs 26 and 27 are now not reporting an error with the RC and lxml 4.9.1

--- #### SwimSwam - working 4/10/22 ####
name: 26_SwimSwam - News [NASA Wildcat Aquatics]
url: "https://swimswam.com/feed/#1"
filter:
  - xpath:
      method: xml
      path: '//item/title/text()|//item/description/text()|//item/content:encoded/text()'
      namespaces:
        content: http://purl.org/rss/1.0/modules/content/
      exclude: 'a'
  - keep_lines_containing:
      re: '\bNASA\sWildcat'
  - html2text:
additions_only: true
--- #### SwimSwam - working 4/10/22 ####
name: 27_SwimSwam - News [Prokos]
url: "https://swimswam.com/feed/#2"
filter:
  - xpath:
      method: xml
      path: '//item/title/text()|//item/description/text()|//item/content:encoded/text()'
      namespaces:
        content: http://purl.org/rss/1.0/modules/content/
      exclude: 'a'
  - keep_lines_containing:
      re: '\bProkos\b'
  - html2text:
additions_only: true

But now three other jobs are reporting an error with the RC and lxml 4.9.0 and 4.9.1. Those three jobs are all from the same website. (As a precaution, I removed all of the jobs from the jobs file, ran webchanges, then cleared cache, and then re-added all of the jobs. No luck)

--- #### NASA Registration working 4/17/22 ####
name: 30_NASA - Registration
url: "https://www.teamunify.com/team/isnasa/page/team-registration"
use_browser: false
filter:
  - xpath:
      path: '//*[@id="mainContent"]'
  - html2text:
      method: bs4
  - delete_lines_containing:
      re: '(?i)^(.*)Sign\sin|^(.*)Home|^Questions?|(?i)^Check\sStatus|^\sFacebook|Twitter|Instagram|^\© Copyright|Privacy\sStatement|^\[\]\(https'
  - remove_repeated
additions_only: true
--- #### NASA SC Records - working 4/18/22 ####
name: 31_NASA - SC Records
url: "https://www.teamunify.com/team/isnasa/page/team-info/short-course-yards"
filter:
  - xpath:
      path: '//*[@id="mainContent"]'
  - html2text:
      method: bs4
      parser: html5lib
  - re.sub: '(?m)^[ \t]*'
  - delete_lines_containing:
      re: '(?i)^(.*)Sign\sin|^(.*)Home|^\sFacebook|Twitter|Instagram|^\© Copyright|Privacy\sStatement|^\[\]\(https'
  - strip:
additions_only: true
--- #### NASA Records - working 4/18/22 ####
name: 32_NASA - LC Records
url: "https://www.teamunify.com/team/isnasa/page/team-info/long-course-meters"
filter:
  - xpath:
      path: '//*[@id="mainContent"]'
  - html2text:
      method: bs4
      parser: html5lib
  - re.sub: '(?m)^[ \t]*'
  - delete_lines_containing:
      re: '(?i)^(.*)Sign\sin|^(.*)Home|^\sFacebook|Twitter|Instagram|^\© Copyright|Privacy\sStatement|^\[\]\(https'
  - strip:
additions_only: true

Changing the job to exclude the path allows the job to run but I do get another error

--- #### NASA Registration working 4/17/22 ####
name: 30_NASA - Registration
url: "https://www.teamunify.com/team/isnasa/page/team-registration#1"
filter:
#   - xpath:
#       path: '//*[@id="mainContent"]'
  - html2text:
      method: bs4
  - delete_lines_containing:
      re: '(?i)^(.*)Sign\sin|^(.*)Home|^Questions?|(?i)^Check\sStatus|^\sFacebook|Twitter|Instagram|^\© Copyright|Privacy\sStatement|^\[\]\(https'
  - remove_repeated
additions_only: true

webchanges --test-filter 30
/Users/john/.pyenv/versions/3.10.5/lib/python3.10/site-packages/bs4/builder/init.py:545: XMLParsedAsHTMLWarning: It looks like you're parsing an XML document using an HTML parser. If this really is an HTML document (maybe it's XHTML?), you can ignore or filter this warning. If it's XML, you should know that using an XML parser will be more reliable. To parse this document as XML, make sure you have the lxml package installed, and pass the keyword argument features="xml" into the BeautifulSoup constructor.
warnings.warn(

Changing the job again:

--- #### NASA Registration working 4/17/22 ####
name: 30_NASA - Registration
url: "https://www.teamunify.com/team/isnasa/page/team-registration"
filter:
  - xpath:
      path: '//*[@id="mainContent"]'
  - html2text:
  - delete_lines_containing:
      re: '(?i)^(.*)Sign\sin|^(.*)Home|^Questions?|(?i)^Check\sStatus|^\sFacebook|Twitter|Instagram|^\© Copyright|Privacy\sStatement|^\[\]\(https'
  - remove_repeated
additions_only: true

webchanges --test-filter 30
ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.

So after installing the RC it seems that xpath is causing an error on those jobs for some reason.

I am adding settings from my config in case that is an issue? I believe I copied this from your website:

###
job_defaults:
  all:
    headers:
      Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
      Accept-Language: en-US,en
      DNT: '1'
      Pragma: no-cache
      Sec-Fetch-Dest: document
      Sec-Fetch-Mode: navigate
      Sec-Fetch-Site: same-origin
      Sec-Fetch-User: ?1
      Upgrade-Insecure-Requests: '1'
      User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15 Chrome/99.0.4844.0
  browser:
      _beta_use_playwright: true

RC and 4.9.1 error.txt
job runs with incorrect URL ???.txt

[mborsetti]

Wonderful, thanks for running this and for the excellent data and feedback! You have been of amazing help.

I tracked down the error in job 30 (and presumably 31 and 32) to the data from the server starting with an XML declaration (<?xml version="1.0" encoding="UTF-8"?>) (see https://stackoverflow.com/questions/37592045/is-an-xml-declaration-necessary-in-an-html-document). Now that the code is handling the lxml package correctly, the latter will generate an error as it was expecting "pure" HTML but got an XML header before the HTML: an unexpected corner case that I can fix.

I pushed a new release candidate (3.10.3rc1) to the unreleased branch; if you can test it that would be great:

pip install https://github.com/mborsetti/webchanges/archive/unreleased.tar.gz

That did it!

webchanges --errors
Reports no errors or issues from any of my current jobs.

Thank you for sticking with me to the "bitter end". Appreciate it.