Some improvements for Scrapy tutorial

[eliasdorneles]

So, here are some improvements for Scrapy tutorial, please let me know what you think.

Summary of the changes:

• adds an example and explanation about following links (fixes #608 )
• added note about CSS vs XPath, idea is to encourage people who know CSS to dig XPath too
• updated recommend XPath tutorials to the ones I felt were most helpful as a XPath beginner
• add a watch target to docs Makefile (note: depends on watchdog)
• a few other small changes

Does this look good?

[kmike]

Selector works with CSS as well

[kmike]

It may be good to mention response.urljoin because it is shorter, and because it handles <base> HTML tag.

[eliasdorneles]

oh right!! sorry, forgot about it again! :)

[kmike]

It may worths it to mention a callback method can yield both Items and Requests at the same time, there are no separate "item callbacks" and "request callbacks".

[eliasdorneles]

+1, will do it in a jiffy

[kmike]

👍

[kmike]

One more common pain point for beginners is creating a single item from multiple web pages.
Maybe add a link to http://doc.scrapy.org/en/latest/topics/request-response.html#topics-request-response-ref-request-callback-arguments for now, or even discuss it briefly in the tutorial?

In future other solutions could appear, e.g. #1138 or #1144.

[eliasdorneles]

@kmike +1, I added some examples and attended your other comments.

Let me know if you think of another thing. =)

[kmike]

in github UI extra new lines look too sparse to my taste, but maybe they are more readable at RTFD

[eliasdorneles]

yeah, I tried without the new lines and it looked a bit confusing.

[eliasdorneles]

So, with the risk of overthinking this... :P
The main change here was to avoid the indirection (before it was showing the structure, and detailing the type of content below).

Yes, it is a bit more sparse now, but this is nicer for a reader trying to understand the structure, because 1) she can see the descriptions directly in the structure tree, and 2) she can scan quickly the vertically-aligned comments to know the kinds of files there are.

The only thing I'm not so sure about here is the vertical alignment.

[kmike]

I like inline comments more.

[kmike]

This example can be confusing because in the tutorial we were not yielding dicts, only Items.

[kmike]

There is "Spiders are expected to return their scraped data inside :class:~scrapy.item.Item objects." statement earlier, which doesn't show a full picture now.

[eliasdorneles]

Hmmm, right.
I've already added a bit about being able to use Python dicts in the beginning of "Defining our Item" section above, lemme just fix that statement then.

[eliasdorneles]

So, I settled for just removing the incorrect statement and changing the example for using Item, to avoid the different item type being a distraction here.

[kmike]

is -> valid CSS?

[eliasdorneles]

ooops, no! 😁

[kmike]

A couple more ideas/issues/suggestions for the tutorial, this time more radical:

1. It may focus on start_requests instead of start_urls. This way users will see more clearly how Scrapy works; parse method will be less 'magical', and from start_requests is easy to explain that start_urls is a shortcut for a common use case.
2. Transition from dicts (used in overview) to items (used here) could be more gradual; to do that we may introduce Items later in the tutorial, and continue using dicts at the beginning. I.e. start writing spider using dicts, get something working, and then switch to Items. The jump from overview to tutorial could become more smooth this way.

I think tutorial should explain all the points displayed in overview again, but in more details, expanding from basic features (dicts, single file spiders, css selectors) to more advanced (items, scrapy projects, xpath selectors). Some repetition will help users learn. IMHO showing a feature in overview and then using something completely different in a tutorial is not enough, it is confusing. After an overview users already know something; we should reinforce their knowledge and provide extra details.