Add a page which explains important Scrapy concepts in a single page

[kmike]

As @plafl said: "Scrapy is very extensible but that has a cost too. There are too many concepts: spiders, items, middlewares, pipelines, exporters, extensions, signals, settings. As a newcomer I would like to know which problem they solve."

+1 :) I think we should add a page which will explain all this in a single place - what are these concepts, when and why to use them.

[Granitosaurus]

Isn't that pretty much http://doc.scrapy.org/en/latest/topics/architecture.html?highlight=scrapy%20architecture ? Which is my favorite page in the docs.

[kmike]

@Granitas yeah, it is close, a good catch.

This page is not targeted for beginners though, e.g. both for downloader and for spider middleware it says just "They provide a convenient mechanism for extending Scrapy functionality by plugging custom code." without explaining which custom code should go to a spider mw and which should go to downloader mw. You can figure it out by meditating over the architecture overview picture, but it is not an easy task if you're just starting. Also, it doesn't explain extensions at all.

[darshanime]

I can write this page, I have a few questions to get started:

1. is the page is aimed at the extensions developer
2. should it read like an article or should it contain code
3. what is the desired length of the article (how many words)
4. what concepts should be focused on

[Gallaecio]

This seems to go in the lines of a question I recently tried to answer on StackOverflow: https://stackoverflow.com/q/54421455/939364