Episode on scraping UNSC data with requests/lxml #47
Conversation
_episodes/04-lxml.md
Outdated
| - "How can I download web pages' HTML in Python?" | ||
| - "How can I evaluate XPath or CSS selectors in Python?" | ||
| - "How can I format scraped data as a spreadsheet?" | ||
| - "How do I build a scraper that is resilient to change and aberration?" |
There was a problem hiding this comment.
Suggest using a different word than "aberration" in the spirit of reducing the use of jargon, especially in the episode questions. Or just something like "How do I build a scraper that will keep working even if the page structure changes?"
_episodes/04-lxml.md
Outdated
| - "An element tree's `cssseelct` and `xpath` methods extract elements of interest." | ||
| - "A scraper can be divided into: identifying the set of URLs to scrape; extracting some elements from a page; and transforming them into a useful output format." | ||
| - "It is important but challenging to be resilient to variation in page structure: one should automatically validate and manually inspect their extractions." | ||
| - "A framework like [Scrapy](http://scrapy.org) may help to build robust scrapers, but may be harder to learn." |
There was a problem hiding this comment.
Link to the scrapy lesson if moved to _extras?
_episodes/04-lxml.md
Outdated
| * We can look at the HTML source code of a page to find how target elements are structured and | ||
| how to select them. | ||
| * We can use the browser console to try out XPath or CSS selectors on a live site. | ||
| * We can use visual scrapers to handle some basic scraping tasks. These help determine an appropriate selector, and may also perform spidering. |
There was a problem hiding this comment.
Was spidering defined in the previous episode? If not, I suggest explaining the term.
There was a problem hiding this comment.
It is mentioned in the intro, but in passing. Might be worthwhile expanding on that definition here.
|
|
||
| # Introducing Requests and lxml | ||
|
|
||
| We make use of two tools that are not specifically developed for scraping, but are very useful for that purpose (among others). |
There was a problem hiding this comment.
I suggest moving up the paragraph about Python before introducing the libraries. First, clarify that we will be coding the scraper in Python (link to the base Python lesson) and that the following tools are Python libraries that we will need.
_episodes/04-lxml.md
Outdated
| > its HTML (using the `html.parser` backend). In some ways, Beautiful | ||
| > Soup may have a more friendly design for web scraping (e.g. its handling | ||
| > of text). | ||
| {: .callout} |
There was a problem hiding this comment.
I like the comments about alternatives (also in the visual scraper episode). But I wonder if it wouldn't be better to group them into a discussion section. I think one of the principles of the Carpentries is that we try to focus on one set of tools, teach them, and then mention alternatives, rather than risk confusing people while they are learning?
The bs4 comments are not really necessary here at all
|
Thanks!
…On 10 July 2017 at 04:01, Thomas Guignard ***@***.***> wrote:
Merged #47 <#47>.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#47 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAEz6638-B81U_Gfc_fCwlW6-rVubfDVks5sMRWNgaJpZM4OM9sX>
.
|
Closes #6, #11, #12 .
This episode is untested in real life.
The motivation (as far as I'm concerned) for presenting requests/lxml instead of scrapy is that:
Issues: