Extracting with include_images option causes subsequent extractions to fail

[TommiNieminen]

There's a bug with extraction using include_images option, it only works once and then extractions start to fail. The bug is in htmlprocessing.py, lines 46 to 50:

  if include_images is True:
      # Many websites have <img> inside <figure> or <picture> or <source> tag
      for element in ['figure', 'picture', 'source']:
          MANUALLY_CLEANED.remove(element)
      MANUALLY_STRIPPED.remove('img')

This bit of code will be repeated for all extractions, but since the tags have been removed on the first extraction, a ValueError will be thrown for all other extractions. I tested that adding existence checks solves the issue:

if include_images is True:
        # Many websites have <img> inside <figure> or <picture> or <source> tag
        for element in ['figure', 'picture', 'source']:
            if element in MANUALLY_CLEANED:
                MANUALLY_CLEANED.remove(element)
        if 'img' in MANUALLY_STRIPPED:
            MANUALLY_STRIPPED.remove('img')

[adbar]

Thanks, the way the lists were used was inconsistent.
It should be fixed in 8a5123f, can you confirm by running the last version from the repository?

[TommiNieminen]

Thanks, the cleaning_list removal works now, but the stripping_list removal will still cause a ValueError with repeated extractions, since the code tries to remove 'img' that was already removed during the first extraction.

        cleaning_list = [e for e in cleaning_list if e
                         not in ('figure', 'picture', 'source')]
        stripping_list.remove('img')

[adbar]

I got it wrong: without copy() the lists were linked to the same object and thus identical. It should be fixed now (33cd96b).
Can you please confirm?

[TommiNieminen]

Seems to work now, thanks.