Pinning docutils in setup.py does not fix TypeError: 'generator' object is not reversible

[anitagraser]

This is a follow-up of #8616 (comment)

I tried to pin docutils in setup.py: movingpandas/movingpandas@1b6a47e

But the build keeps failing: https://readthedocs.org/projects/movingpandas/builds/15140289/

Is it recommended to use requirements.txt instead of setup.py?

[astrojuanlu]

@anitagraser I still didn't have time to reproduce the issue and try to understand why pinning on setup.py doesn't seem to work. I see mentions to easy_install coming from setuptools in the logs, which makes me think that the dependencies listed in install_requires are not getting properly installed. That is all I know for now.

As a workaround while we try to give a better solution (next week), if you're in a hurry, you can try using a requirements.txt, since that should use pip rather than setuptools-deprecated mechanisms.

[astrojuanlu]

It turns out that pinning docutils in setup.py and calling setup.py install does not remove the 0.18 version, because this deprecated mechanism uses easy_install. Therefore, the solution is to replace the docs/setup.py with a docs/requirements.txt file with proper pinnings. I will send a PR shortly.

I forked the project, and built exactly the same commit astrojuanlu/movingpandas@3900e38 and in my fork it works.

❌ Upstream https://readthedocs.org/projects/movingpandas/builds/15205455/ (Sphinx v1.8.5)
✔️ Fork https://readthedocs.org/projects/movingpandas-astrojuanlu/builds/15223145/ (Sphinx v4.2.0)

Since movingpandas is not pinning Sphinx and it was created earlier, it gets 1.8.5: https://docs.readthedocs.io/en/stable/builds.html#python

Projects created before Oct 20, 2020 use 1.8.x. New projects use the latest version.

That explains why my fork works. However, it still doesn't explain why, even after pinning docutils in setup.py with method: setuptools on the config file, 0.18 gets installed in your fork.

---

To understand more, I enabled LIST_PACKAGES_INSTALLED_ENV on your project and triggered a master build https://readthedocs.org/projects/movingpandas/builds/15223240/, where we have this:

/home/docs/checkouts/readthedocs.org/user_builds/movingpandas/envs/master/bin/python docs/setup.py install --force
...
Processing dependencies for movingpandas===master
Searching for docutils<0.18,>=0.14
Reading https://pypi.org/simple/docutils/
Downloading https://files.pythonhosted.org/packages/4c/5e/6003a0d1f37725ec2ebd4046b657abb9372202655f96e76795dca8c0063c/docutils-0.17.1-py2.py3-none-any.whl#sha256=cf316c8370a737a022b72b56874f6602acf974a37a9fba42ec2876387549fc61
Best match: docutils 0.17.1
Processing docutils-0.17.1-py2.py3-none-any.whl
Installing docutils-0.17.1-py2.py3-none-any.whl to /home/docs/checkouts/readthedocs.org/user_builds/movingpandas/envs/master/lib/python3.7/site-packages
Adding docutils 0.17.1 to easy-install.pth file
...
/home/docs/checkouts/readthedocs.org/user_builds/movingpandas/envs/master/bin/python -m pip list --pre
Package                       Version
----------------------------- ---------
...
docutils                      0.18

And the problem is that dependencies brought by setup.py install get installed with the ancient, deprecated, soon-to-be-removed easy_install, and have a lower precedence. A simpler way of seeing it:

docs@69329f85c908:~$ python3.7 -m venv .venv
docs@69329f85c908:~$ source .venv/bin/activate
(.venv) docs@69329f85c908:~$ pip install docutils
Collecting docutils
  Downloading docutils-0.18-py2.py3-none-any.whl (569 kB)
     |████████████████████████████████| 569 kB 513 kB/s 
Installing collected packages: docutils
Successfully installed docutils-0.18
WARNING: You are using pip version 20.1.1; however, version 21.3.1 is available.
You should consider upgrading via the '/home/docs/.venv/bin/python3.7 -m pip install --upgrade pip' command.
(.venv) docs@69329f85c908:~$ easy_install "docutils<0.18"
WARNING: The easy_install command is deprecated and will be removed in a future version.
Searching for docutils<0.18
Reading https://pypi.org/simple/docutils/
Downloading https://files.pythonhosted.org/packages/4c/5e/6003a0d1f37725ec2ebd4046b657abb9372202655f96e76795dca8c0063c/docutils-0.17.1-py2.py3-none-any.whl#sha256=cf316c8370a737a022b72b56874f6602acf974a37a9fba42ec2876387549fc61
Best match: docutils 0.17.1
Processing docutils-0.17.1-py2.py3-none-any.whl
Installing docutils-0.17.1-py2.py3-none-any.whl to /home/docs/.venv/lib/python3.7/site-packages
Adding docutils 0.17.1 to easy-install.pth file
Installing rst2man.py script to /home/docs/.venv/bin
Installing rst2latex.py script to /home/docs/.venv/bin
Installing rst2html5.py script to /home/docs/.venv/bin
Installing rst2odt_prepstyles.py script to /home/docs/.venv/bin
Installing rst2html4.py script to /home/docs/.venv/bin
Installing rstpep2html.py script to /home/docs/.venv/bin
Installing rst2odt.py script to /home/docs/.venv/bin
Installing rst2html.py script to /home/docs/.venv/bin
Installing rst2xml.py script to /home/docs/.venv/bin
Installing rst2pseudoxml.py script to /home/docs/.venv/bin
Installing rst2s5.py script to /home/docs/.venv/bin
Installing rst2xetex.py script to /home/docs/.venv/bin

Installed /home/docs/.venv/lib/python3.7/site-packages/docutils-0.17.1-py3.7.egg
Processing dependencies for docutils<0.18
Finished processing dependencies for docutils<0.18
(.venv) docs@69329f85c908:~$ pip list
Package    Version
---------- -------
docutils   0.18
pip        20.1.1
setuptools 47.1.0
WARNING: You are using pip version 20.1.1; however, version 21.3.1 is available.
You should consider upgrading via the '/home/docs/.venv/bin/python3.7 -m pip install --upgrade pip' command.
(.venv) docs@69329f85c908:~$ python
Python 3.7.12 (default, Nov  9 2021, 13:15:46) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docutils
>>> docutils.__version__
'0.18'
>>> 

[astrojuanlu]

movingpandas/movingpandas#154

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[humitos]

It seems this is not a problem related to Read the Docs or a solution that we can provide from our side. If I'm wrong, please comment here and re-open the issue explaining what we can do from our side. Thanks!