You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Let's also check the other spiders that inherit from LinksSpider, in case any return the total count of pages/results.
In doing so, we can create a new superclass for all spiders that can generate requests based on a total count (maybe name it IndexSpider). These include at least:
canada_montreal
chile_base (maybe – pagination is done in the URL path instead of the query string)
kenya_makueni
mexico_administracion_publica_federal
mexico_quien_es_quien
uganda_releases
uk_contracts_finder
We can add a note to either the docstring or writing-spiders.rst to say that we prefer using IndexSpider instead of LinksSpider where possible, since generating all requests at once will lead to a faster collection.
The Honduras API returns a json like:
We can use the
pages
field to build the next url in a loop and make the request in parallelThe text was updated successfully, but these errors were encountered: