Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG+1] docs: update overview spider code to use toscrape.com and minor changes #2249

Merged
merged 3 commits into from Sep 16, 2016

Conversation

@eliasdorneles
Copy link
Member

@eliasdorneles eliasdorneles commented Sep 15, 2016

So, this will replace the spider example code from the overview that
scrapes questions from StackOverflow by a spider scraping quotes (much
like the one in the tutorial), and upates the text around it to be
consistent.

There are also minor wording changes plus a small Sphinx/reST syntax fix
on the features list at the bottom (it was creating a definition list,
causing one line to be bold).

Does this look good?
Thanks!

So, this will replace the spider example code from the overview that
scrapes questions from StackOverflow by a spider scraping quotes (much
like the one in the tutorial), and upates the text around it to be
consistent.

There are also minor wording changes plus a small Sphinx/reST syntax fix
on the features list at the bottom (it was creating a definition list,
causing one line to be bold).
of them as they finish.
an argument. In the ``parse`` callback we loop through the quote elements
using a CSS Selector, yield a Python dict with the extracted quote text and author,
look for a link to the next page and schedules another request using the same

This comment has been minimized.

@stummjr

stummjr Sep 15, 2016
Member

and schedules another request using the same: schedules should be in 1st person of plural.

This comment has been minimized.

@eliasdorneles

eliasdorneles Sep 15, 2016
Author Member

fixed!

@codecov-io
Copy link

@codecov-io codecov-io commented Sep 15, 2016

Current coverage is 83.36% (diff: 100%)

Merging #2249 into master will not change coverage

Powered by Codecov. Last update 2f60f2a...75531e4

Copy link
Contributor

@redapple redapple left a comment

Nice!

'link': response.url,
}
next_page = response.css('li.next a::attr("href")').extract_first()
if next_page:

This comment has been minimized.

@kmike

kmike Sep 15, 2016
Member

I think next_page is None could be slightly better because an empty string is a valid relative URL

This comment has been minimized.

@eliasdorneles

eliasdorneles Sep 15, 2016
Author Member

fixed!

@redapple redapple changed the title docs: update overview spider code to use toscrape.com and minor changes [MRG+1] docs: update overview spider code to use toscrape.com and minor changes Sep 16, 2016
@eliasdorneles
Copy link
Member Author

@eliasdorneles eliasdorneles commented Sep 16, 2016

hey @kmike, I'll merge this to move things forward, feel free to point out if you have any other concern.
thanks for reviewing!

@eliasdorneles eliasdorneles merged commit de1a6ac into master Sep 16, 2016
2 of 3 checks passed
2 of 3 checks passed
codecov/project 83.36% (target 95.00%)
Details
codecov/patch Coverage not affected when comparing 2f60f2a...75531e4
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@redapple redapple deleted the fix-overview-spider branch Oct 25, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants