[MRG] Add LxmlLinkExtractor class similar to SgmlLinkExtractor (#528) #559

redapple · 2014-01-23T16:30:54Z

~~I had to change the Selector HTML parse from lxml.etree.HTMLParser to lxml.html.HTMLParser to have those helpful .make_links_absolute() and .iterlinks()~~

~~There's still margin for some factorisation and cleanup (as much of the code is bluntly copied from SgmlLinkExtractor~~

implementation of #528

redapple · 2014-01-26T10:46:59Z

The last change makes the thing a bit more readable. But the self.base_url = None line in SgmlLinkExtractor.__init__() feels a bit awkward

redapple · 2014-01-27T22:37:21Z

This question on StackOverflow makes me wonder about nofollow https://stackoverflow.com/questions/21392222/scrapy-honor-rel-nofollow

redapple · 2014-02-01T22:47:05Z

@dangra @kmike @nramirezuy @pablohoffman @darkrho , any thoughts so far?

The extractor could even be renamed to SelectorLinkExtractor but users may expect that it takes a Selector parameter somehow...

kmike · 2014-02-01T22:54:39Z

scrapy/contrib/linkextractors/lxmlhtml.py

+                 deny_extensions=None):
+
+        tag_func = lambda x: x in tags
+        attr_func = lambda x: x in attrs


I think it is better to make sets from tags and attrs before creating lambdas. And maybe we can write tag_func=tags.__contains__ then, not sure about it. Micro-optimizing ftw.

kmike · 2014-02-01T23:02:28Z

scrapy/linkextractor.py

+            allowed &= not url_has_any_extension(parsed_url, self.deny_extensions)
+        if allowed and self.canonicalize:
+            link.url = canonicalize_url(parsed_url)
+        return allowed


I see that you've just copied this code, but it seems if allowed could be added to every if to avoid extra computations. Or it can be rewritten like

if self.allow_res and not _matches(link.url, self.allow_res): return False if self.deny_res and _matches(link.url, self.deny_res): return False # ...

Also, modifying link.url in _link_allowed method is bad.
Feel free to ignore this all because it is not relevant for this PR :)

IMO, that method needs to be revamped to allow easy extension and modification. But that is out of the focus of this PR.

kmike · 2014-02-01T23:05:45Z

to make it clear: I haven't really looked at this PR, just noticed a couple of suboptimal chunks of code.

rmax · 2014-02-02T00:10:43Z

The extractor could even be renamed to SelectorLinkExtractor but users may expect that it takes a Selector parameter somehow...

Just like the Selector class, I think this class can be named LinkExtractor and be importable from scrapy.contrib.linkextractors. This class is going to deprecate the sgml lx, right?

rmax · 2014-02-02T13:02:08Z

scrapy/contrib/linkextractors/lxmlhtml.py

-        for e, a, l, p in html.iterlinks():
+    def _extract_links(self, selector, response_url, response_encoding):
+        # hacky way to get the underlying lxml parsed document
+        for e, a, l, p in selector._root.iterlinks():


I have a spider which uses tags="div" and attrs="data-url" in the link extractor. I think it won't work here given that .iterlinks returns only a subset of predefined tags: http://lxml.de/lxmlhtml.html

Interesting @darkrho . Do you mind making up an example HTML snippet illustrating this?
We could rewrite a scrapy-specific .iterlinks, the common case not being that complicated:

for el in self.iter(): attribs = el.attrib tag = _nons(el.tag) if tag != 'object': for attrib in link_attrs: if attrib in attribs: yield (el, attrib, attribs[attrib], 0)

(https://github.com/lxml/lxml/blob/master/src/lxml/html/__init__.py#L363)

@redapple Here is a simplified example:

In [1]: from scrapy.http import HtmlResponse In [2]: from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor In [3]: response = HtmlResponse('http://example.com', body=""" <html><body> <div id="item1" data-url="get?id=1"><a href="#">Item 1</a></div> <div id="item2" data-url="get?id=2"><a href="#">Item 2</a></div> </body></html> """) In [4]: lx = SgmlLinkExtractor(tags='div', attrs='data-url') In [5]: lx.extract_links(response) Out[5]: [Link(url='http://example.com/get?id=1', text=u'Item 1', fragment='', nofollow=False), Link(url='http://example.com/get?id=2', text=u'Item 2', fragment='', nofollow=False)]

it lxml.html.iterlinks is not good for us, what do you think about basing the new link extractor on top of scrapy selectors. Additionally it avoids the switch of the default parser, now lxml.etree.HtmlParser, to lxml.html.HtmlParser.

who wants to give a shot at a Selector-based .extract_links()?

I personally think it's easier to implement using lxml.etree.Element.iter() and using element's attrib property than using .xpath() or .css() on Selector objects (it's probably more efficient also)

dangra · 2014-02-03T02:19:34Z

scrapy/selector/unified.py

@@ -16,7 +16,7 @@
 __all__ = ['Selector', 'SelectorList']

 _ctgroup = {
-    'html': {'_parser': etree.HTMLParser,
+    'html': {'_parser': html.HTMLParser,


I can be paranoid, but shouldn't we compare the parsing performance before/after making this change?
This is going to affect every parsed page even if not using linkextractors.

lxml.html.HTMLParser is specially meant for HTML and the main difference seems to be the helpers for links. I don't think it would make that much of a difference (but I have no proof).

But if we say that .iterlinks doesn't fit our needs in the end, we may not need it at all.

And @dangra as you mention parsing performance, I'd be thrilled to have some performance tests in scrapy, if only to validate/invalidate my beloved compiled XPath expressions :)

nramirezuy · 2014-02-03T16:55:43Z

@redapple These are my thoughts about the subject: #578

kmike · 2014-04-15T10:49:37Z

See also: #331

redapple · 2014-06-20T11:28:52Z

@kmike , @darkrho , @nramirezuy , @pablohoffman , probably room for cleanup but this should contain all fixes from your comments.
Question: go that route? or add another extractor based on Selector?

dangra · 2014-06-20T14:09:30Z

scrapy/selector/unified.py

@@ -2,7 +2,7 @@
 XPath selectors based on lxml
 """

-from lxml import etree
+from lxml import etree, html


unneeded import lxml.etree.html

dangra · 2014-06-20T19:13:37Z

scrapy/contrib/linkextractors/lxmlhtml.py

+        # hacky way to get the underlying lxml parsed document
+        for el, attr, attr_val in self._iter_links(selector._root):
+            if self.scan_tag(el.tag):
+                if self.scan_attr(attr):


looks like there is no need for nesting two "if"s, it can be reduced to if self.scan_tag(el.tag) and self.scan_attr(attr)

dangra · 2014-06-20T19:14:28Z

+1 to update docs and merge.

redapple · 2014-06-20T23:36:52Z

I can still squash of course

kmike · 2014-06-22T22:06:11Z

LGTM

redapple mentioned this pull request Jan 23, 2014

RegexLinkExtractor: encode URL unicode value when creating Links #561

Merged

dangra mentioned this pull request Jan 25, 2014

UnicodeEncodeError in SgmlLinkExtractor when using restrict_xpaths #562

Closed

kmike reviewed Feb 1, 2014
View changes

rmax reviewed Feb 2, 2014
View changes

dangra reviewed Feb 3, 2014
View changes

dangra mentioned this pull request Feb 3, 2014

Support relative urls better #548

Closed

redapple mentioned this pull request May 27, 2014

SGMLParseError when looking for links with SGMLLinkExtractor #731

Closed

dangra reviewed Jun 20, 2014
View changes

scrapy/selector/unified.py Outdated

@@ -2,7 +2,7 @@

XPath selectors based on lxml

"""

from lxml import etree

from lxml import etree, html

This comment has been minimized.

Sign in to view

dangra Jun 20, 2014
Member

unneeded import lxml.etree.html

dangra reviewed Jun 20, 2014
View changes

kmike mentioned this pull request Jun 22, 2014

Changed feed to HTMLParser to unicode instead of str. #756

Closed

redapple changed the title ~~[WIP] Add LxmlLinkExtractor class similar to SgmlLinkExtractor (#528)~~ [MRG] Add LxmlLinkExtractor class similar to SgmlLinkExtractor (#528) Jun 23, 2014

redapple added 2 commits Jan 23, 2014

Add LxmlLinkExtractor class similar to SgmlLinkExtractor (#528)

d34a57a

Add doc on LxmlLinkExtractor class

Loading status checks…

90e6914

dangra mentioned this pull request Jun 25, 2014

Promote LxmlLinkExtractor as LxmlExtractor #761

Merged

dangra merged commit 90e6914 into scrapy:master Jun 25, 2014
1 check passed

1 check passed

continuous-integration/travis-ci The Travis CI build passed
Details

redapple mentioned this pull request Jun 16, 2016

Selector.root is not an instance of lxml.html.HtmlElement even if parser is html scrapy/parsel#40

Closed

redapple deleted the redapple:lxmlextractor branch Jul 8, 2016

scrapy / scrapy

[MRG] Add LxmlLinkExtractor class similar to SgmlLinkExtractor (#528) #559

[MRG] Add LxmlLinkExtractor class similar to SgmlLinkExtractor (#528) #559

redapple commented Jan 23, 2014

redapple commented Jan 26, 2014

redapple commented Jan 27, 2014

redapple commented Feb 1, 2014

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

kmike commented Feb 1, 2014

rmax commented Feb 2, 2014

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nramirezuy commented Feb 3, 2014

kmike commented Apr 15, 2014

redapple commented Jun 20, 2014

This comment has been minimized.

This comment has been minimized.

dangra commented Jun 20, 2014

redapple commented Jun 20, 2014

kmike commented Jun 22, 2014

scrapy / scrapy

[MRG] Add LxmlLinkExtractor class similar to SgmlLinkExtractor (#528) #559

[MRG] Add LxmlLinkExtractor class similar to SgmlLinkExtractor (#528) #559

Conversation

redapple commented Jan 23, 2014

redapple commented Jan 26, 2014

redapple commented Jan 27, 2014

redapple commented Feb 1, 2014

This comment has been minimized.

kmike Feb 1, 2014 Member

This comment has been minimized.

kmike Feb 1, 2014 Member

This comment has been minimized.

rmax Feb 2, 2014 Contributor

kmike commented Feb 1, 2014

rmax commented Feb 2, 2014

This comment has been minimized.

rmax Feb 2, 2014 Contributor

This comment has been minimized.

redapple Feb 2, 2014 Author Contributor

This comment has been minimized.

rmax Feb 2, 2014 Contributor

This comment has been minimized.

dangra Feb 3, 2014 Member

This comment has been minimized.

redapple Feb 3, 2014 Author Contributor

This comment has been minimized.

dangra Feb 3, 2014 Member

This comment has been minimized.

redapple Feb 3, 2014 Author Contributor

nramirezuy commented Feb 3, 2014

kmike commented Apr 15, 2014

redapple commented Jun 20, 2014

This comment has been minimized.

dangra Jun 20, 2014 Member

This comment has been minimized.

dangra Jun 20, 2014 Member

dangra commented Jun 20, 2014

redapple commented Jun 20, 2014

kmike commented Jun 22, 2014

kmike Feb 1, 2014
Member

kmike Feb 1, 2014
Member

rmax Feb 2, 2014
Contributor

rmax Feb 2, 2014
Contributor

redapple Feb 2, 2014
Author Contributor

rmax Feb 2, 2014
Contributor

dangra Feb 3, 2014
Member

redapple Feb 3, 2014
Author Contributor

dangra Feb 3, 2014
Member

redapple Feb 3, 2014
Author Contributor

dangra Jun 20, 2014
Member

dangra Jun 20, 2014
Member