IndexError: list index out of range

[CachingFoX]

If I run test cases with python setup.py lint I got an IndexError: list index out of range exception. The repository is up-to-date and without any changes. This appends since some days.

======================================================================
ERROR: test_get_cache (test.test_geocaching.TestShortcuts)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/andreas/PycharmProjects/pycaching_dev/test/test_geocaching.py", line 393, in test_get_cache
    self.assertEqual("Nekonecne ticho", c.name)
  File "/Users/andreas/PycharmProjects/pycaching_dev/pycaching/util.py", line 31, in wrapper
    self.load()
  File "/Users/andreas/PycharmProjects/pycaching_dev/pycaching/cache.py", line 741, in load
    self._logbook_token = re.findall("userToken\\s*=\\s*'([^']+)'", js_content)[0]
IndexError: list index out of range

I have debug the exception and I found following problem and a "solution":
js_content = "\n".join(map(lambda i: i.text, root.find_all("script")))

• js_content is full of newlines
• root.find_all("script") is okay
• map returns a empty set - I don't know way

Solution
I use a virtual environment and I down-grade the package beautifulsoup4 from version 4.9.0 to 4.8.2 and it works.

[FriedrichFroebel]

This is due to the following change (taken from the changelog):

Embedded CSS and Javascript is now stored in distinct Stylesheet and Script tags, which are ignored by methods like get_text() since most people don't consider this sort of content to be 'text'. This feature is not supported by the html5lib treebuilder. [bug=1868861]

The corresponding bug is available here. In fact we have been using the text attribute for getting the content of the script tags until now.

I just played around a bit and managed to find a solution for the problem:

# Requires at least bs4 version 4.9.0.
from bs4.element import Script
js_content = "\n".join(root.find_all(string=lambda i: isinstance(i, Script)))

or

# Works with bs4 version 4.8.2 as well.
js_content = []
for script in root.find_all("script"):
    js_content += script.contents
js_content = "\n".join(js_content)

# Short, but less readable version:
# js_content = "\n".join([i for s in root.find_all("script") for i in s.contents])

According to the changelog, the import for the first solution will be simplified with the next release of bs4.

@tomasbedrich Are you okay with requiring at least bs4 version 4.9.0, or should we maintain backwards compatibility and use the second approach?

[CachingFoX]

Thank you @FriedrichFroebel - yes you are right "map" works fine and the text property of script nodes are empty.

[tomasbedrich]

Nice catch, thanks for reporting. @FriedrichFroebel I think we should update to the latest version.