Upgrade scrapy to v2.3.0 by removing the NotImplementedError

[curquiza]

Since this scrapy upgrading, we got an error when running:

$ pipenv run ./docs_scraper config.json
> Docs-Scraper: https://docs.meilisearch.com 27 records)
2020-09-10 14:42:09 [scrapy.core.scraper] ERROR: Spider error processing <GET https://docs.meilisearch.com> (referer: None)
Traceback (most recent call last):
  File "/Users/curquiza/.local/share/virtualenvs/docs-scraper-ao5z5akx/lib/python3.8/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/Users/curquiza/Documents/docs-scraper/scraper/src/documentation_spider.py", line 184, in parse_from_start_url
    return self.parse(response)
  File "/Users/curquiza/.local/share/virtualenvs/docs-scraper-ao5z5akx/lib/python3.8/site-packages/scrapy/spiders/__init__.py", line 93, in parse
    raise NotImplementedError('{}.parse callback is not defined'.format(self.__class__.__name__))
NotImplementedError: DocumentationSpider.parse callback is not defined

Nb hits: 27

Only the master branch is concerned, the latest release (v0.10.1 does not contain this error).

Edit

I reverted the concerned PR (manually because GitHub wasn't able to revert it automatically) to make the master branch working again. See #65.
The new goal of this issue is to upgrade Scrapy from v2.2.1 to v2.3.0 by fixing the NotImplementedError at the same time.

[renehernandez]

@curquiza I'll take a look later today

[renehernandez]

@curquiza I am not able to reproduce the error. Could you share the config.json file you are using to test this? Nevermind, I hadn't properly updated scrapy 😂

[renehernandez]

So the problem is that the Spider base class now returns NotImplemented error for the parse method invocation. We are invoking the method as a callback in the DocumentationSpider class at:

• start_requests
• parse_from_start_url

The solution would be to provide our implementation of the def parse(self, response) method