Having problem to run docs-scraper 0.10.1 with Python 3.6/3.8.0

[kappa-wingman]

I had machines running Fedora 32 and Centos 7/8.

On the development machine (Fedora 32). It has Python 3.8.5.
I could run the docs-scraper 0.10.1 without problem.

On the production machine Centos 7 (python 3.6) and Centos 8.2 (python 3.6 and python 3.8.0)
It could compile the docs_scraper package but having problem when it run, below is the error message.

$ pipenv run ./docs_scraper docs_scraper-config.json
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
2020-08-07 10:02:57 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.kappawingman.com> (referer: None)
Traceback (most recent call last):
File "/home/username/venv/docs-scraper-0.10.1/lib64/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/home/username/venv/docs-scraper-0.10.1/lib64/python3.6/site-packages/scrapy/spiders/init.py", line 93, in parse
raise NotImplementedError('{}.parse callback is not defined'.format(self.class.name))
NotImplementedError: DocumentationSpider.parse callback is not defined

All python 3.6/3.8.0 have warning about the 'twisted' package'.
Please take a look, thanks.

[curquiza]

Hello @kappa-wingman! Nice to see you again 😁 (and sorry that's to report an issue...)

It's indeed weird... I don't know why it works on MacOs, Ubuntu, Debian, Fedora but not on CentOS. I will take a look as soon as possible.

If other users have the same problem and would add precision or know how to solve that, it would be great 😁

[kappa-wingman]

Hello @curquiza!

Got an update. I could use my old python virtual environment (venv) for doc scraper (0.9.3) to run the new doc scraper python program (0.10.1).

Here is the diff of the package version of the virtual venv:

-cffi                1.14.0
+cffi                1.14.1
-cryptography        2.9.2
+cryptography        3.0
-hyperlink           19.0.0
-idna                2.9
+hyperlink           20.0.1
+idna                2.10
-importlib-resources 2.0.1
+importlib-resources 3.0.0
-lxml                4.5.1
-meilisearch         0.11.1
+iniconfig           1.0.1
+itemadapter         0.1.0
+itemloaders         1.0.1
+jmespath            0.10.0
+lxml                4.5.2
+meilisearch         0.12.0
-pip                 20.1.1
+pip                 20.2.1
-py                  1.8.2
+Protego             0.1.16
+py                  1.9.0
-pytest              5.4.3
+pytest              6.0.0
-Scrapy              1.7.4
+Scrapy              2.3.0
+toml                0.10.1
-urllib3             1.25.9
-virtualenv          20.0.25
+urllib3             1.25.10
+virtualenv          20.0.30

Then I downgrade Scrapy 2.3.0 to 1.7.4, I can use docs-scraper with meilisearch 0.12 now..
After that I checked my Fedora venv, it's using Scrapy 2.2.1.
I downgrade Scrapy to 2.2.1 in Centos 8 venv, then I can use with meilisearch 0.13 now :)

[curquiza]

Great! Glad to read you solved the issue!
Should we upgrade Scrapy into v2.2.3 from your POV? Would it avoid the issue you had?

[kappa-wingman]

The latest version of Scrapy (v2.3.0) would cause problem.
So installing a version lower than (v2.3.0) should be fine.

[curquiza]

Ok, if I understand well, I can close the issue then?

[kappa-wingman]

Yes, please close it. Thanks.