startproject and override command line tool for Page Objects development

CHANGELOG.rst

@@ -7,7 +7,6 @@ TBR:
 ------------------
 
 * Cache mechanism using ``SCRAPY_POET_CACHE`` setting
-
 * We also have these **backward incompatible** changes since the
   rules follow a different structure:
 
@@ -16,8 +15,12 @@ TBR:
       for better URL matching.
     * This resuls in a newer format in the ``SCRAPY_POET_OVERRIDES`` setting.
 
-* removed support for Python 3.6
-* added support for Python 3.10
+* Modified the `scrapy startproject` command to include boilerplates for
+  Page Objects and its tests.
+* Added a new `scrapy override` command easily create new Page Objects
+  with accompanying tests over a given webpage.
+* Removed support for Python 3.6
+* Added support for Python 3.10
 
 0.2.1 (2021-06-11)
 ------------------

MANIFEST.in

@@ -1,5 +1,7 @@
-include CHANGES.rst
+include CHANGELOG.rst
 include LICENSE
 include README.rst
 
-recursive-include tests *
+recursive-include scrapy_poet/templates *.tmpl
+
+global-exclude __pycache__ *.py[cod]

docs/index.rst

@@ -49,6 +49,7 @@ To get started, see :ref:`intro-install` and :ref:`intro-tutorial`.
    :maxdepth: 1
 
    settings
+   project_commands
    api_reference
    contributing
    changelog

docs/project_commands.rst

@@ -0,0 +1,61 @@
+.. _`project_commands`:
+
+================
+Project Commands
+================
+
+``scrapy-poet`` extends some of Scrapy's commands for developer productivity.
+
+startproject
+============
+
+The original ``scrapy startproject`` command would produce a project structure like:
+
+.. code-block::
+
+    my_project
+    ├── my_project
+    │   ├── spiders
+    │   │   └── __init__.py
+    │   ├── __init__.py
+    │   ├── items.py
+    │   ├── middlewares.py
+    │   ├── pipelines.py
+    │   └── settings.py
+    └── scrapy.cfg
+
+``scrapy-poet`` extends this further to setup:
+
+    * a Page Objects directory in ``po/``
+    * a test suite for Page Objects in ``tests/po/``
+
+Invoking the same ``scrapy startproject`` would produce:
+
+.. code-block::
+
+    my_project
+    ├── my_project
+    │   ├── po
+    │   │   ├── templates
+    │   │   │   └── __init__.py
+    │   │   └── __init__.py
+    │   ├── spiders
+    │   │   └── __init__.py
+    │   ├── __init__.py
+    │   ├── items.py
+    │   ├── middlewares.py
+    │   ├── pipelines.py
+    │   └── settings.py
+    ├── tests
+    │   └── po
+    │       ├── fixtures
+    │       │   └── __init__.py
+    │       └── __init__.py
+    └── scrapy.cfg
+
+override
+========
+
+Following the structure above, we can then use ``scrapy override`` to produce:
+
+

example/example/settings.py

@@ -8,6 +8,9 @@
 #     https://docs.scrapy.org/en/latest/topics/spider-middleware.html
 from example.autoextract import AutoextractProductProvider
 
+from web_poet import default_registry
+
+
 BOT_NAME = 'example'
 
 SPIDER_MODULES = ['example.spiders']
@@ -22,3 +25,6 @@
    'scrapy_poet.InjectionMiddleware': 543,
 }
 
+PO_PACKAGE = "example.po"
+PO_TESTS_PACKAGE = "tests.po"
+SCRAPY_POET_OVERRIDES = default_registry.get_overrides(filters=PO_PACKAGE)