LinkExtractor with Unique = False doesn't extract fully identical Links #3798

Ksianka · 2019-05-27T18:21:49Z

Case:
unexpected behavior identified for LinkExctractor with Unique = False if the page contains fully identical links (the same URL and text).
The current result returns one unique link instead of two or more identical ones.
For example:

Local http file

<html>
<body>
<a href='sample3.html'>sample 3 repetition</a>
<a href='sample3.html'>sample 3 repetition</a>
</body>
</html>```

2) Scrapy test code:
ignore the error about <GET file:///robots.txt>  as we use local file

`import scrapy
from scrapy.linkextractors import LinkExtractor

class QuotesSpider(scrapy.Spider):
    name = “test”
    start_urls = [
        'file:///<insert_your_local_path>/Test_file.html',
    ]

def __init__(self, *args, **kwargs):
    super(QuotesSpider, self).__init__(*args, **kwargs)
    self.le = LinkExtractor(unique=False)

def parse(self, response):
    links = self.le.extract_links(response)
    yield {'extract_link': links}`

3) run the code: scrapy crawl test, and get the result with one link:

{'extract_link': [Link(url='file:///<your_local_path>/sample3.html', text='sample 3 repetition', fragment='', nofollow=False)]}

Instead of the result with two links from HTML:

{'extract_link': [Link(url='file:///<your_local_path>/sample3.html', text='sample 3 repetition', fragment='', nofollow=False), 
Link(url='<your_local_path>/sample3.html', text='sample 3 repetition', fragment='', nofollow=False)]}

Ksianka · 2019-05-27T18:24:11Z

Pre-solution:
the issue is driven by the public method exctract_link of LxmlLinkExtractor(the same as LinkExctractor) which returns unique_list(all_links), see line 130 in module lxmlhtml.py, instead of all_links (similar to solution in sgml.py).

However, here we get another issue:
It looks like the original idea of this unique_list(all_links) was to avoid duplication of links if there is a repetition of tags in restrict_css and restrict_xpaths stated by user In LinkExctractor instance.
For example: self.le = ##LinkExtractor(unique=False, restrict_css=('a', 'a'), restrict_xpaths=('a', ‘a’)).

The duplication of restrict_css and restrict_xpaths is prosed to be corrected by applying function unique_list() to joined/combined list of css and x_paths in init.py module of class FilteringLinkExtractor(object).
Code for correction:

a) Initial code for joined css_x_paths:
scrapy/linkextractors/init.py

self.restrict_xpaths = tuple(arg_to_iter(restrict_xpaths))
self.restrict_xpaths += tuple(map(self._csstranslator.css_to_xpath,
arg_to_iter(restrict_css)))

b) Adjusted code for joined css_x_paths:
scrapy/linkextractors/init.py

_joined_xpath_css = arg_to_iter(restrict_xpaths)
_joined_xpath_css += tuple((map(self._csstranslator.css_to_xpath,
arg_to_iter(restrict_css))))
self.restrict_xpaths = tuple(unique_list(_joined_xpath_css))

…xtract fully identical Links

Fix bug of issue scrapy#3798. When use LinkExtractor with option 'unique=False', It still returns links with duplicates removed. I changed extract_links to reflect the unique option.

Ksianka changed the title ~~#3796 LinkExtractor with Unique = False doesn't extract fully identical Links~~ #3798 LinkExtractor with Unique = False doesn't extract fully identical Links May 27, 2019

Ksianka pushed a commit to Ksianka/scrapy that referenced this issue May 27, 2019

fix for issue scrapy#3798 LinkExtractor with Unique = False doesn't e…

f94739b

…xtract fully identical Links

Ksianka mentioned this issue May 27, 2019

Fix for issue #3798 LinkExtractor with Unique = False doesn't extract fully identical Links #3799

Closed

Gallaecio added the bug label Aug 19, 2019

qwlake mentioned this issue Jul 26, 2020

Fix unique option in LinkExtractor #4695

Closed

wRAR changed the title ~~#3798 LinkExtractor with Unique = False doesn't extract fully identical Links~~ LinkExtractor with Unique = False doesn't extract fully identical Links Jul 27, 2020

wRAR mentioned this issue Mar 23, 2022

fix: return unique_list only when link_extractor.unique is True #5458

Merged

wRAR closed this as completed in #5458 Jan 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LinkExtractor with Unique = False doesn't extract fully identical Links #3798

LinkExtractor with Unique = False doesn't extract fully identical Links #3798

Ksianka commented May 27, 2019 •

edited

Loading

Ksianka commented May 27, 2019

LinkExtractor with Unique = False doesn't extract fully identical Links #3798

LinkExtractor with Unique = False doesn't extract fully identical Links #3798

Comments

Ksianka commented May 27, 2019 • edited Loading

Ksianka commented May 27, 2019

Ksianka commented May 27, 2019 •

edited

Loading