UserWarning: Selector got both text and root, root is being ignored. WARNING

[gelodefaultbrain]

Description

UserWarning: Selector got both text and root, root is being ignored. Only happens on Ubuntu

Steps to Reproduce

1. Open a simple python shell
2. Paste code below
3. Run it

from scrapy.selector import Selector
link = "<a class='fsm fwn fcdg' target='_blank' href='/ta/id/340639-par-olaines-novada-domes-2009-gada-30-septembra-sedes-lemuma-apstiprinato-saistoso-noteikumu-nr-76-par-detalplanojuma-apstiprinasanu'>Par Olaines novada domes 2009. gada 30. septembra sdes lmum apstiprinto saistoo noteikumu Nr. 76 'Par detlplnojuma apstiprinanu nekustamajam paumam 'Vrpas' (kad. Nr. 8080 008 0095)', pielikuma Nr. 66 atcelanu</a>"
selector = Selector(text=link, type='html')

Warning output:

/home/iamuser/.local/share/virtualenvs/myenv-YA2UsV5L/lib/python3.11/site-packages/scrapy/selector/unified.py:83: UserWarning: Selector got both text and root, root is being ignored.
  super().__init__(text=text, type=st, root=root, **kwargs)

Expected behavior: No warnings? It's quite strange because it doesn't appear on MacOS and Windows

Actual behavior: Shows warning message above

Reproduces how often: 100% on Ubuntu

Versions

Please paste here the output of executing scrapy version --verbose in the command line.

Scrapy       : 2.8.0
lxml         : 4.9.2.0
libxml2      : 2.9.14
cssselect    : 1.2.0
parsel       : 1.8.1
w3lib        : 2.1.1
Twisted      : 22.10.0
Python       : 3.11.2 (main, Feb  7 2023, 13:52:42) [GCC 11.3.0]
pyOpenSSL    : 23.1.1 (OpenSSL 3.0.8 7 Feb 2023)
cryptography : 39.0.1
Platform     : Linux-5.15.0-69-generic-x86_64-with-glibc2.35

Additional context

Why does this warning appear on ubuntu? I tried running it on MacOS and Windows and it doesn't appear. Is this something of a concern? I believe this should be supressed. Thanks!

[GeorgeA92]

Most likely caused by this scrapy/parsel@746a0da commit on parsel.

This change appears only on parsel version 1.8.0 and above.

I tried running it on MacOS and Windows and it doesn't appear

According to docs parsel v. 1.8.0 released... only yesterday.
I suppose that Your other environments have earlier version of parsel.
@gelodefaultbrain Can You please post output of scrapy version --verbose from Your other(MacOS, Windows) environments?
If it is an issue for You I`d suggest to rollback parsel to earlier 1.7.0 as temporary solution.

cc @Gallaecio

[gelodefaultbrain]

Hi @GeorgeA92

This is the output of my scrapy version --verbose

Scrapy       : 2.8.0
lxml         : 4.9.2.0
libxml2      : 2.9.14
cssselect    : 1.2.0
parsel       : 1.8.1
w3lib        : 2.1.1
Twisted      : 22.10.0
Python       : 3.11.2 (main, Feb  7 2023, 13:52:42) [GCC 11.3.0]
pyOpenSSL    : 23.1.1 (OpenSSL 3.0.8 7 Feb 2023)
cryptography : 39.0.1
Platform     : Linux-5.15.0-69-generic-x86_64-with-glibc2.35

Thank you for responding. Just wondering, this isn't much of a concern right? The reason I'm doing this code
selector = Selector(text=link, type='html') is because I'm trying to convert an html code as a scrapy selector object. So yeah with that, I get this warning only on Linux/Ubuntu by the way. Thanks

PS: Sorry, I meant this is from my Linux one and yeah I can see from the commit indeed. I do have parsel 1.8.1

[wRAR]

It's unrelated to BeautifulSoup.

[gelodefaultbrain]

Most likely caused by this scrapy/parsel@746a0da commit on parsel.

This change appears only on parsel version 1.8.0 and above.

I tried running it on MacOS and Windows and it doesn't appear

According to docs parsel v. 1.8.0 released... only yesterday. I suppose that Your other environments have earlier version of parsel. @gelodefaultbrain Can You please post output of scrapy version --verbose from Your other environments? If it is an issue for You I`d suggest to rollback parsel to earlier 1.7.0 as temporary solution.

cc @Gallaecio

Just wondering, would it be possible if there's a way for someone at least to suppressed this warning? I'm not sure what it is for though but it's showing up whenever I do Selector(text=link, type='html'). Basically when we're trying to convert an html string into a scrapy selector object. If so, please feel free to enlighten me the purpose and importance of this warning message.

I'm just a bit confused to why it would have a warning wherein I'm just converting an html string to a scrapy selector object.

Basically this code Selector(text=link, type='html'). I'm just passing in the link to the text argument and it's quite confusing and unrelated to the root argument? The message "Selector got both text and root, root is being ignored." seems to be quite confusing 😕 cause it says that "selector got both text and root"? wherein I'm just using text argument and pass in the link variable (html string). I see that root is default to None so I'm really confused to what the warning is saying.

If that makes sense :D But for now yeah thanks for letting me know about the lower versions! 😄

[wRAR]

if there's a way for someone at least to suppressed this warning?

You can try https://docs.python.org/3/library/warnings.html assuming you are asking about doing this in the user code / on the project level.

I'm just a bit confused to why it would have a warning wherein I'm just converting an html string to a scrapy selector object.

Because the Scrapy code is not updated for the new Parsel code. The warning is harmless.

[gelodefaultbrain]

Show comment

Okay cool! Thank you very much for responding.

[gelodefaultbrain]

Before closing this issue, just a quick note that it wasn't because of the Platform. Whether Linux, Windows or Mac. It was because of my parsel version. As discussed here

#5903 (comment)
#5903 (comment)

[Gallaecio]

This was a valid issue to raise, I would have left it open, but we can handle this as part of #5894 as well, so no strong opinion.

[andersk]

FWIW, this caused Zulip’s test-documentation script to output 3031 copies of this warning:

2023-04-25 20:40:17 [py.warnings] WARNING: /srv/zulip-venv-cache/3ec1f5854a5e9afdb5fbc8904bfe106ed062ff56/zulip-py3-venv/lib/python3.8/site-packages/scrapy/selector/unified.py:83: UserWarning: Selector got both text and root, root is being ignored.
2023-04-25 20:40:17 [py.warnings] WARNING: /srv/zulip-venv-cache/3ec1f5854a5e9afdb5fbc8904bfe106ed062ff56/zulip-py3-venv/lib/python3.8/site-packages/scrapy/selector/unified.py:83: UserWarning: Selector got both text and root, root is being ignored.
2023-04-25 20:40:17 [py.warnings] WARNING: /srv/zulip-venv-cache/3ec1f5854a5e9afdb5fbc8904bfe106ed062ff56/zulip-py3-venv/lib/python3.8/site-packages/scrapy/selector/unified.py:83: UserWarning: Selector got both text and root, root is being ignored.
2023-04-25 20:40:17 [py.warnings] WARNING: /srv/zulip-venv-cache/3ec1f5854a5e9afdb5fbc8904bfe106ed062ff56/zulip-py3-venv/lib/python3.8/site-packages/scrapy/selector/unified.py:83: UserWarning: Selector got both text and root, root is being ignored.
2023-04-25 20:40:17 [py.warnings] WARNING: /srv/zulip-venv-cache/3ec1f5854a5e9afdb5fbc8904bfe106ed062ff56/zulip-py3-venv/lib/python3.8/site-packages/scrapy/selector/unified.py:83: UserWarning: Selector got both text and root, root is being ignored.
2023-04-25 20:40:17 [py.warnings] WARNING: /srv/zulip-venv-cache/3ec1f5854a5e9afdb5fbc8904bfe106ed062ff56/zulip-py3-venv/lib/python3.8/site-packages/scrapy/selector/unified.py:83: UserWarning: Selector got both text and root, root is being ignored.
2023-04-25 20:40:17 [py.warnings] WARNING: /srv/zulip-venv-cache/3ec1f5854a5e9afdb5fbc8904bfe106ed062ff56/zulip-py3-venv/lib/python3.8/site-packages/scrapy/selector/unified.py:83: UserWarning: Selector got both text and root, root is being ignored.
2023-04-25 20:40:17 [py.warnings] WARNING: /srv/zulip-venv-cache/3ec1f5854a5e9afdb5fbc8904bfe106ed062ff56/zulip-py3-venv/lib/python3.8/site-packages/scrapy/selector/unified.py:83: UserWarning: Selector got both text and root, root is being ignored.
…

drowning out all other output, so “harmless” is not the word I would have chosen. Can this be reopened?

[Gallaecio]

@andersk We plan to address it soon. as part of #5894, probably in time for the next release of Scrapy. It is harmless in the sense that you can silence it with standard Python warning configuration, and it should be safe to ignore.