[MRG+1] update Scrapy to use parsel 1.5 #3390

kmike · 2018-08-15T11:26:39Z

Todo:

update Selector and SelectorList API documentation
revise Selectors tutorial: either bring more from the parsel's tutorial, or link to missing pieces (like has-class XPath extension); maybe rearrange some of the sections.
check .extract usages
find more places where .attrib makes code simpler
update after #3400.
fix "Removing namespaces" example - see scrapy/parsel#119

codecov · 2018-08-21T23:27:39Z

Codecov Report

Merging #3390 into master will not change coverage.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master    #3390   +/-   ##
=======================================
  Coverage   84.49%   84.49%           
=======================================
  Files         167      167           
  Lines        9376     9376           
  Branches     1392     1392           
=======================================
  Hits         7922     7922           
  Misses       1199     1199           
  Partials      255      255

Impacted Files	Coverage Δ
scrapy/linkextractors/sgml.py	`96.8% <ø> (ø)`	⬆️
scrapy/selector/unified.py	`93.75% <ø> (ø)`	⬆️
scrapy/loader/__init__.py	`94.52% <100%> (ø)`	⬆️

See #3373 and #3372 for motivation.

* remove unused link * fix ReST syntax * fix a link to regular expression docs

This change is backwards incompatible if ItemLoader is used with a custom Selector subclass which overrides .extract without overriding .getall.

Also, response.urljoin is added in a few places, for robustness.

… adjust wording in the introduction

… suggest get/getall

stummjr · 2018-09-12T19:52:14Z

@kmike just a heads-up that I started reviewing this today.

eliasdorneles · 2018-09-14T09:32:09Z

I love this PR so much <3 =)

See scrapy/parsel#119

stummjr

👏 wow, that's a huge revamp in the docs. Looks AWESOME to me! 👍

stummjr · 2018-09-17T20:36:29Z

docs/intro/tutorial.rst

-``None`` when it doesn't find any element matching the selection.
+However, using ``.get()`` directly on a :class:`~scrapy.selector.SelectorList`
+instance avoids an ``IndexError`` and returns ``None`` when it doesn't
+find any element matching the selection.

 There's a lesson here: for most scraping code, you want it to be resilient to
 errors due to things not being found on a page, so that even if some parts fail
 to be scraped, you can at least get **some** data.

 Besides the :meth:`~scrapy.selector.Selector.extract` and


I think we need to replace the .extract here with .getall.

stummjr · 2018-09-17T20:36:29Z

docs/intro/tutorial.rst

@@ -641,7 +649,7 @@ this time for scraping author information::

        def parse_author(self, response):
            def extract_with_css(query):
-                return response.css(query).extract_first().strip()
+                return response.css(query).get().strip()


Maybe we should promote the usage of the default parameter here to make this example less error prone:

return response.css(query).get(default='').strip()

I know that this is somehow unrelated, but still related to best practices.

TIL response.css('foo::text').get() returns None if foo is found, but there is no text inside.

stummjr · 2018-09-17T20:36:29Z

docs/topics/selectors.rst


-For convenience, response objects expose a selector on `.selector` attribute,
-it's totally OK to use this shortcut when possible::
+For convenience, response objects expose a selector on `.selector` attribute.


Is the .selector part supposed to be shown in monospaced font? I think we're missing an extra surrounding backtick in that case.

stummjr · 2018-09-17T20:36:29Z

docs/topics/selectors.rst

-    >>> Selector(text=body).xpath('//span/text()').extract()
-    [u'good']
+    >>> Selector(text=body).xpath('//span/text()').get()
+    'good'

 Constructing from response::


Maybe we should add a small observation here stating that this is usually not necessary, given that Response already contains a selector.

Even though it's explained later, I think we should be explicit that this could be a bad pattern.

stummjr · 2018-09-17T20:36:29Z

docs/topics/selectors.rst

+Notice that CSS selectors can select text or attribute nodes using CSS3
+pseudo-elements::
+
+    >>> selector.css('title::text').get()


should it be response.css('title::text').get()?

stummjr · 2018-09-17T20:36:29Z

docs/topics/selectors.rst

+
+.. _topics-selectors-css-extensions:
+
+Extensions to CSS Selectors


this is awesome! we were missing this section :)

stummjr · 2018-09-17T20:36:29Z

tests/test_utils_iterators.py

+            attrs.append((
+                x.attrib['id'],
+                x.xpath("name/text()").extract(),
+                x.xpath("./type/text()").extract()))


.getall() here too?

* fixed several small issues * re-written "Creating Selectors" section * fixed remaining .extract usage in tests

also, move "Selecting attributes" reference closer to `a::atr(href)` example

kmike · 2018-09-18T00:04:26Z

Thanks for a careful review @stummjr! How does it look now?

stummjr

Looks great to me! 👏

stummjr · 2018-09-18T13:22:39Z

docs/topics/selectors.rst

+shortcuts. By using ``response.selector`` or one of these shortcuts
+you can also ensure the response body is parsed only once.
+
+But if required, it is possible to use ``Selector`` directly.


I liked how you changed the order and rephrased this part! 🤘

kmike changed the title ~~[WIP] update docs to use parsel 1.5~~ [WIP] update Scrapy to use parsel 1.5 Aug 21, 2018

kmike force-pushed the parsel-1.5 branch from 05c3be4 to 133ca3a Aug 22, 2018

kmike mentioned this pull request Aug 23, 2018

[MRG+1] Added general guide for developer tools #3400

Merged

kmike requested a review from stummjr Aug 23, 2018

kmike force-pushed the parsel-1.5 branch from 133ca3a to 0794f5d Aug 23, 2018

kmike added 14 commits Aug 8, 2018

require parsel 1.5+

25ac469

DOC more Python 3 in examples

0ccead6

add pytest temp files to gitignore

395d9d0

DOC .extract_first() -> .get()

ca27010

DOC update Scrapy selectors tutorial to match parsel's tutorial better

d32c4de

DOC unlink Firefox & Firebug sections from the tutorial for now.

09fd6c2

See #3373 and #3372 for motivation.

DOC cleanup references in tutorials:

2c48d15

* remove unused link * fix ReST syntax * fix a link to regular expression docs

TST update tests to use get/getall/attrib instead of extract

53da56c

update spider templates to use .get

8c29be6

[backwards incompatible] switch ItemLoader from .extract to .getall.

460f0f0

This change is backwards incompatible if ItemLoader is used with a custom Selector subclass which overrides .extract without overriding .getall.

switch SgmlLinkExtractor to .getall

12e42bb

DOC mention .attrib in the tutorial

afce971

DOC switch from .extract to get/getall API in docs

bdcc045

Also, response.urljoin is added in a few places, for robustness.

DOC reorganize selectors tutorial, port more topics from parsel docs,…

Loading status checks…

7fdfdb7

… adjust wording in the introduction

kmike force-pushed the parsel-1.5 branch from 0794f5d to 7fdfdb7 Sep 12, 2018

DOC use autodocs for selectors; document more methods and attributes;…

Loading status checks…

dc95ecb

… suggest get/getall

kmike changed the title ~~[WIP] update Scrapy to use parsel 1.5~~ update Scrapy to use parsel 1.5 Sep 12, 2018

kmike added this to the v1.6 milestone Sep 12, 2018

DOC fix remove_namespaces example

Loading status checks…

9db21e5

See scrapy/parsel#119

stummjr approved these changes Sep 17, 2018

View changes

kmike added 2 commits Sep 18, 2018

DOC address @stummjr's review comments

2c3b215

* fixed several small issues * re-written "Creating Selectors" section * fixed remaining .extract usage in tests

DOC mention gotcha with `foo::text` selector and empty `foo` elements

Loading status checks…

ffbd33e

also, move "Selecting attributes" reference closer to `a::atr(href)` example

stummjr approved these changes Sep 18, 2018

View changes

stummjr changed the title ~~update Scrapy to use parsel 1.5~~ [MRG+1] update Scrapy to use parsel 1.5 Sep 18, 2018

dangra merged commit bafd174 into master Sep 18, 2018
3 checks passed

3 checks passed

codecov/patch 100% of diff hit (target 84.49%)
Details

continuous-integration/appveyor/pr AppVeyor build succeeded
Details

continuous-integration/travis-ci/pr The Travis CI build passed
Details

dangra deleted the parsel-1.5 branch Sep 18, 2018

kmike added the docs label Dec 26, 2018

scrapy / scrapy

[MRG+1] update Scrapy to use parsel 1.5 #3390

[MRG+1] update Scrapy to use parsel 1.5 #3390

kmike commented Aug 15, 2018 •

edited

codecov bot commented Aug 21, 2018 •

edited

stummjr commented Sep 12, 2018

eliasdorneles commented Sep 14, 2018

stummjr left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

kmike commented Sep 18, 2018

stummjr left a comment

This comment has been minimized.


		.. _topics-selectors-css-extensions:

		Extensions to CSS Selectors

scrapy / scrapy

[MRG+1] update Scrapy to use parsel 1.5 #3390

[MRG+1] update Scrapy to use parsel 1.5 #3390

Conversation

kmike commented Aug 15, 2018 • edited

codecov bot commented Aug 21, 2018 • edited

Codecov Report

stummjr commented Sep 12, 2018

eliasdorneles commented Sep 14, 2018

stummjr left a comment

This comment has been minimized.

stummjr Sep 17, 2018 Member

This comment has been minimized.

stummjr Sep 17, 2018 Member

This comment has been minimized.

kmike Sep 17, 2018 Author Member

This comment has been minimized.

stummjr Sep 17, 2018 Member

This comment has been minimized.

stummjr Sep 17, 2018 Member

This comment has been minimized.

stummjr Sep 17, 2018 Member

This comment has been minimized.

stummjr Sep 17, 2018 Member

This comment has been minimized.

stummjr Sep 17, 2018 Member

kmike commented Sep 18, 2018

stummjr left a comment

This comment has been minimized.

stummjr Sep 18, 2018 Member

kmike commented Aug 15, 2018 •

edited

codecov bot commented Aug 21, 2018 •

edited

stummjr Sep 17, 2018
Member

stummjr Sep 17, 2018
Member

kmike Sep 17, 2018
Author Member

stummjr Sep 17, 2018
Member

stummjr Sep 17, 2018
Member

stummjr Sep 17, 2018
Member

stummjr Sep 17, 2018
Member

stummjr Sep 17, 2018
Member

stummjr Sep 18, 2018
Member