Integrate tox

[stephenfin]

tox is the automation tool of choice within the Python ecosystem. Integrate it
and avoid the need to manually create a virtualenv to test things.

[akrabat]

Can you point me at some good resources to learn about and use tox?

[stephenfin]

Hmm, it's kind of second nature to me at this point but this looks like a good article to get you started https://opensource.com/article/19/5/python-tox. In short, tox creates a virtual environment (virtualenv) and installs the package plus whatever other dependencies you specify in it, before running the command. You use it with tox -e $env, e.g. tox -e py27.

[akrabat]

https://tox.readthedocs.io also helps.

We should update DEVELOPERS.rst too.

[stephenfin]

I can do that in a follow-up to the doc cleanup series. Would you be okay with me just replacing all references to the manual script with tox invocations?

[akrabat]

Interestingly, I get failures when running on the Mac. Will investigate why.

[stephenfin]

What are the failures? I suspect you might be missing some interpreters. Does tox -e py27 fail? If so, can you paste the error here?

[akrabat]

What are the failures? I suspect you might be missing some interpreters. Does tox -e py27 fail? If so, can you paste the error here?

Full output here: https://gist.github.com/akrabat/0dfc4beb4d160549db1a5a2cde820322

[stephenfin]

There looks to be two issues here. Firstly, you're missing a Python 3 interpreter. If you run tox --skip-missing-interpreters instead, it'll skip those ones you can't handle. For the others though, that's the exact same errors I'm seeing locally. How were you running tests without tox?

[akrabat]

Currently I have two Pyenv environments: rst2pdf-py2 which has Python 2.7.16 installed in it and rst2pdf-py3 which has Python 3.7.4 installed. I run nosetests -v -i regulartest -I sphinxtest in each environment to run the tests for Py2 and then Py3.

[stephenfin]

Can you try the following?

source rst2pdf-py2/bin/activate # or however you activate it
pip freeze
nosetests -v -i regulartest -I sphinxtest
deactivate

source .tox/py27/bin/activate
pip freeze
nosetests -v -i regulartest -I sphinxtest
deactivate

And share the output of each.

[stephenfin]

In case it's not clear, I'm trying to figure out what the difference between the two environments. In theory, there shouldn't be one, but clearly something different is happening.

[akrabat]

Thanks for the help. Will have to be Sunday as I've been up since 6am and have stepped away from the laptop & won't be back home until then.

[johnthagen]

No idea if this is the issue (please pardon the drive-by comment), but tox caches the virtual environments it creates by default. If you both want to do a clean tox run, run with the same command line options and also pass --recreate.

$ tox --recreate -e py37

[akrabat]

Resolved the tox Python 3 problem. Seems that you need tox for Python 3, not tox for Python 2...

[akrabat]

pip freeze differences (side-by-side list):

# My pyenv (rst2pdf-py2)                                                                         | # tox py27
aafigure==0.6                                                                                    | aafigure==0.6
alabaster==0.7.12                                                                                | alabaster==0.7.12
Babel==2.7.0                                                                                     | Babel==2.7.0
backports.functools-lru-cache==1.5                                                               | backports.functools-lru-cache==1.5
certifi==2019.3.9                                                                                | certifi==2019.9.11
chardet==3.0.4                                                                                   | chardet==3.0.4
configparser==3.8.1                                                                              |
contextlib2==0.5.5                                                                               |
coverage==4.5.4                                                                                  | coverage==4.5.4
cssselect2==0.2.1                                                                                | cssselect2==0.2.2
cycler==0.10.0                                                                                   | cycler==0.10.0
docutils==0.14                                                                                   | docutils==0.15.2
filelock==3.0.12                                                                                 |
html5lib==1.0.1                                                                                  | html5lib==1.0.1
idna==2.8                                                                                        | idna==2.8
imagesize==1.1.0                                                                                 | imagesize==1.1.0
importlib-metadata==0.19                                                                         |
Jinja2==2.10.1                                                                                   | Jinja2==2.10.1
kiwisolver==1.1.0                                                                                | kiwisolver==1.1.0
lxml==4.4.1                                                                                      | lxml==4.4.1
MarkupSafe==1.1.1                                                                                | MarkupSafe==1.1.1
matplotlib==2.2.4                                                                                | matplotlib==2.2.4
nose==1.3.7                                                                                      | nose==1.3.7
numpy==1.16.4                                                                                    | numpy==1.16.5
packaging==19.0                                                                                  | packaging==19.2
pathlib2==2.3.4                                                                                  |
pdfrw==0.4                                                                                       | pdfrw==0.4
Pillow==6.0.0                                                                                    | Pillow==6.1.0
pluggy==0.12.0                                                                                   |
py==1.8.0                                                                                        |
Pygments==2.4.2                                                                                  | Pygments==2.4.2
pyparsing==2.4.0                                                                                 | pyparsing==2.4.2
PyPDF2==1.26.0                                                                                   | PyPDF2==1.26.0
python-dateutil==2.8.0                                                                           | python-dateutil==2.8.0
pytz==2019.1                                                                                     | pytz==2019.2
reportlab==3.5.23                                                                                | reportlab==3.5.26
requests==2.22.0                                                                                 | requests==2.22.0
-e git+git@github.com:akrabat/rst2pdf.git@1fece67bf256de3e101a955de9c37dce43d3e794#egg=rst2pdf   | -e git+git@github.com:akrabat/rst2pdf.git@1fece67bf256de3e101a955de9c37dce43d3e794#egg=rst2pdf
scandir==1.10.0                                                                                  |
six==1.12.0                                                                                      | six==1.12.0
smartypants==2.0.1                                                                               | smartypants==2.0.1
snowballstemmer==1.2.1                                                                           | snowballstemmer==1.9.1
Sphinx==1.8.5                                                                                    | Sphinx==1.8.5
sphinxcontrib-websupport==1.1.2                                                                  | sphinxcontrib-websupport==1.1.2
subprocess32==3.5.4                                                                              | subprocess32==3.5.4
svg2rlg==0.3                                                                                     |
svglib==0.9.2                                                                                    | svglib==0.9.2
tinycss2==0.6.1                                                                                  | tinycss2==0.6.1
toml==0.10.0                                                                                     |
tox==3.14.0                                                                                      |
typing==3.6.6                                                                                    | typing==3.7.4.1
urllib3==1.25.3                                                                                  | urllib3==1.25.6
virtualenv==16.7.3                                                                               |
webencodings==0.5.1                                                                              | webencodings==0.5.1
xhtml2pdf==0.2.3                                                                                 | xhtml2pdf==0.2.3
zipp==0.5.2                                                                                      |

Items in my env that aren't in tox:

configparser==3.8.1
contextlib2==0.5.5
filelock==3.0.12
importlib-metadata==0.19
pathlib2==2.3.4
pluggy==0.12.0
py==1.8.0
scandir==1.10.0
svg2rlg==0.3
toml==0.10.0
tox==3.14.0
virtualenv==16.7.3
zipp==0.5.2

[stephenfin]

Snip

Items in my env that aren't in tox:

configparser==3.8.1
contextlib2==0.5.5
filelock==3.0.12
importlib-metadata==0.19
pathlib2==2.3.4
pluggy==0.12.0
py==1.8.0
scandir==1.10.0

Most of these are dependencies of tox.

svg2rlg==0.3

This looks unnecessary now. We replaced this with svglib in #745.

toml==0.10.0
tox==3.14.0
virtualenv==16.7.3
zipp==0.5.2

Again, most of these are tox-related.

[stephenfin]

This should be good to merge once #854 has landed and got rid of the based JSON testing, though it will need the PyMuPDF added as a dependency.

[stephenfin]

@akrabat This is good to go now, I think

[akrabat]

I think tox hates me. This is a clean venv with 3.7.7 installed.

All I have done in the venv is:

pip install --upgrade pip
pip install --upgrade setuptools
pip install tox

pip list looks like this:

Package            Version
------------------ -------
appdirs            1.4.4
distlib            0.3.0
filelock           3.0.12
importlib-metadata 1.6.1
packaging          20.4
pip                20.1.1
pluggy             0.13.1
py                 1.8.1
pyparsing          2.4.7
setuptools         47.1.1
six                1.15.0
toml               0.10.1
tox                3.15.2
virtualenv         20.0.21
zipp               3.1.0

Running tox -e py37, I get this:

rob@swiftsure rst2pdf {rst2pdf-tox} (stephenfin/tox-integration) $ tox -e py37
py37 create: /Users/rob/Projects/python/rst2pdf/.tox/py37
py37 installdeps: -crequirements.txt, pytest, pytest-xdist, coverage
py37 develop-inst: /Users/rob/Projects/python/rst2pdf
py37 installed: aafigure==0.6,alabaster==0.7.12,apipkg==1.5,attrs==19.3.0,Babel==2.8.0,certifi==2020.4.5.2,chardet==3.0.4,coverage==5.1,cssselect2==0.3.0,cycler==0.10.0,docutils==0.16,execnet==1.7.1,html5lib==1.0.1,idna==2.9,imagesize==1.2.0,importlib-metadata==1.6.1,Jinja2==2.11.2,kiwisolver==1.2.0,lxml==4.5.1,MarkupSafe==1.1.1,matplotlib==3.2.1,more-itertools==8.3.0,numpy==1.18.5,packaging==19.0,pdfrw==0.4,Pillow==7.1.2,pluggy==0.13.1,py==1.8.1,Pygments==2.6.1,PyMuPDF==1.17.0,pyparsing==2.4.0,PyPDF2==1.26.0,pytest==5.4.3,pytest-forked==1.1.3,pytest-xdist==1.32.0,python-dateutil==2.8.1,pytz==2020.1,reportlab==3.5.42,requests==2.23.0,-e git+git@github.com:akrabat/rst2pdf.git@3e14c22b57b8e8bc7133dffde6eeaa741d40a47e#egg=rst2pdf,six==1.12.0,smartypants==2.0.1,snowballstemmer==2.0.0,Sphinx==3.0.4,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==1.0.3,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.4,svglib==1.0.0,tinycss2==1.0.2,urllib3==1.25.9,wcwidth==0.2.3,webencodings==0.5.1,xhtml2pdf==0.2.4,zipp==3.1.0
py37 run-test-pre: PYTHONHASHSEED='881834738'
py37 run-test: commands[0] | pytest -n auto
=============================================================== test session starts ===============================================================
platform darwin -- Python 3.7.7, pytest-5.4.3, py-1.8.1, pluggy-0.13.1
cachedir: .tox/py37/.pytest_cache
rootdir: /Users/rob/Projects/python/rst2pdf
plugins: forked-1.1.3, xdist-1.32.0
gw0 [308] / gw1 [308] / gw2 [308] / gw3 [308] / gw4 [308] / gw5 [308] / gw6 [308] / gw7 [308]
................................F........s.....F...........................s...ss.s..........F....................................ss.s..... [ 45%]
...s...s.....s...........s.......s.......s...ss..........s..........ssss..................s.....................s...............s.....s.... [ 90%]
....s..F.....F.....s.FF.....F.                                                                                                              [100%]
==================================================================== FAILURES =====================================================================

...

============================================================= short test summary info =============================================================
FAILED rst2pdf/tests/input/test_fancytitles.txt::test_fancytitles - AssertionError: File test_fancytitles.pdf not generated
FAILED rst2pdf/tests/input/test_inkscape.txt::test_inkscape - AssertionError: File test_inkscape.pdf not generated
FAILED rst2pdf/tests/input/test_issue_239_2.txt::test_issue_239_2 - AssertionError: File test_issue_239_2.pdf not generated
FAILED rst2pdf/tests/input/sphinx-issue172/conf.py::sphinx-issue172
FAILED rst2pdf/tests/input/sphinx-issue318/conf.py::sphinx-issue318
FAILED rst2pdf/tests/input/sphinx-issue320/conf.py::sphinx-issue320
FAILED rst2pdf/tests/input/sphinx-issue319/conf.py::sphinx-issue319
FAILED rst2pdf/tests/input/sphinx-repeat-table-rows/conf.py::sphinx-repeat-table-rows
============================================== 8 failed, 273 passed, 27 skipped in 98.51s (0:01:38) ===============================================
ERROR: InvocationError for command /Users/rob/Projects/python/rst2pdf/.tox/py37/bin/pytest -n auto (exited with code 1)
_____________________________________________________________________ summary _____________________________________________________________________
ERROR:   py37: commands failed
rob@swiftsure rst2pdf {rst2pdf-tox} (stephenfin/tox-integration) $

Full output is here: https://gist.github.com/akrabat/433b30a5fcdccd1b182d8afce8ea92df
Also, my Inkscape crashes: https://i.19ft.com/cd2a52cf.png

These are the log files found in .tox/py37/log:

• py37-0.log
• py37-1.log
• py37-2.log
• py37-3.log

The output of:

source .tox/py37/bin/activate
pip freeze

is: https://gist.github.com/akrabat/9283281ec6243b600911dc97c753d27a

For my pyenv virtualenv, I have this pip freeze: https://gist.github.com/akrabat/1fc1486a70f2e955eade2b332c7a1c5c

A cursory glance shows many differences. The most two suspicious are that tox has installed ReportLab 3.5.42 and Sphinx 3.0.4 even though requirements.txt specifies ReportLab 3.5.23 and Sphinx 1.8.5.

Any idea why tox doesn't honour the requirements.txt?

[johnthagen]

Please try tox --recreate -e py37. Tox caches it's internal venvs by default.

[akrabat]

Please try tox --recreate -e py37. Tox caches it's internal venvs by default.

Didn't help. I'm going to try again with pyenv out of the equation…

[akrabat]

If I do:

source .tox/py37/bin/activate
pip install Sphinx==1.8.5
pytest -n auto

it works.

i.e. Inkscape doesn't crash and the right Sphinx is used.

[stephenfin]

tl;dr: Sorry about that. Fixed.

Instead of installing everything from requirements.txt, I was using it as a constraints file. I did it this way because I wanted to test the extras machinery (i.e. the stuff in the square bracket in e.g. pip install .[tests]) but while you can specify -c in the [testenv] deps list to apply constraints against stuff installed that way, this doesn't apply to stuff installed via the [testenv] extras. Put another way, the -c requirement.txt in [testenv] deps was ignored when installing the dependencies indicated by [testenv] extras.

The solution here is to either (a) use requirements.txt as a requirements file rather than a constraints file, or (b) override the install_command setting so that we always use constraints. Since I want to rework how we do requirements in the future (tl;dr: we should be using ranges of constraints rather than specific versions for a "library" like rst2pdf), I've gone with (a) for now.

As for why I didn't spot this before? I had loads of failing tests before #854 and #870 merged and I missed the additional ones being added to the pile. 🙈

EDIT: You'll note I'm using the constraints approach in .travis.yml. We can probably keep that there for now.

EDIT: I've probably linked this before, but good reading background reading here on my note above about reworking requirements to use ranges. See also discussion on Pipfile vs Pipfile.lock.

[johnthagen]

Does this package need setuptools as a runtime dependancy? Generally this wouldn't be in install_requires.

[stephenfin]

We use pkg_resources to get version information. To the best of my knowledge, that is not packaged separately and would require setuptools. Right?

[johnthagen]

Ah, okay. I've usually seen it better to use one of these techniques than require setuptools, though it's probably fine: https://packaging.python.org/guides/single-sourcing-package-version

Maybe importlib.metadata package would be a more modern alternative? (Built into Python 3.8+).

[stephenfin]

I'd be happy to switch to importlib.metadata for Python 3.8+ with the importlib_metadata package included as a dependency for Python < 3.8. If we did this, we should probably also integrate setuptools_scm to automate versioning based on Git tag. I'd also be happy to just hard code the version and just use that (dumb approaches work too). Probably should do whatever we do settle on in a separate PR though as I'm mostly just cleaning up the setup.py file in this PR.

[akrabat]

tox -e py37 works for me now! Thanks @stephenfin

[akrabat]

tox -e docs and tox -e style both fail for me.

I think tox -e style is expected to fail as clearly the codebase isn't consistently following any style yet.

However, I think that tox -e docs probably should succeed.

$ tox -e docs
docs create: /Users/rob/Projects/python/rst2pdf/.tox/docs
docs installdeps: -rrequirements.txt, pytest, pytest-xdist, pymupdf, coverage
docs develop-inst: /Users/rob/Projects/python/rst2pdf
docs installed: aafigure==0.6,alabaster==0.7.12,apipkg==1.5,attrs==19.3.0,Babel==2.7.0,backports.functools-lru-cache==1.5,certifi==2019.3.9,chardet==3.0.4,coverage==5.1,cssselect2==0.3.0,cycler==0.10.0,docutils==0.14,execnet==1.7.1,html5lib==1.0.1,idna==2.8,imagesize==1.1.0,importlib-metadata==1.6.1,Jinja2==2.10.1,kiwisolver==1.1.0,lxml==4.5.1,MarkupSafe==1.1.1,matplotlib==2.2.4,more-itertools==8.3.0,numpy==1.16.4,packaging==19.0,pdfrw==0.4,Pillow==6.2.2,pluggy==0.13.1,py==1.8.1,Pygments==2.4.2,PyMuPDF==1.17.0,pyparsing==2.4.0,PyPDF2==1.26.0,pytest==5.4.3,pytest-forked==1.1.3,pytest-xdist==1.32.0,python-dateutil==2.8.0,pytz==2019.1,reportlab==3.5.23,requests==2.22.0,-e git+git@github.com:akrabat/rst2pdf.git@283ed007145b19b33ab7ff0400a65a585b78c298#egg=rst2pdf,six==1.12.0,smartypants==2.0.1,snowballstemmer==1.2.1,Sphinx==1.8.5,sphinxcontrib-websupport==1.1.2,subprocess32==3.5.4,svg2rlg==0.3,svglib==0.9.3,tinycss2==1.0.2,typing==3.6.6,urllib3==1.25.3,wcwidth==0.2.3,webencodings==0.5.1,xhtml2pdf==0.2.3,zipp==3.1.0
docs run-test-pre: PYTHONHASHSEED='2807965404'
docs run-test: commands[0] | rst2pdf manual.rst --custom-cover=assets/cover.tmpl -o output/manual.pdf -s assets/manual.style -b1
manual.rst:1873: (SEVERE/4) Problem with "start-after" option of "include" directive:
Text not found.
Traceback (most recent call last):
  File "/Users/rob/Projects/python/rst2pdf/.tox/docs/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2130, in run_directive
    result = directive_instance.run()
  File "/Users/rob/Projects/python/rst2pdf/.tox/docs/lib/python3.7/site-packages/docutils/parsers/rst/directives/misc.py", line 103, in run
    'directive:\nText not found.' % self.name)
docutils.parsers.rst.DirectiveError

(and then lots more errors)

[akrabat]

I'll read up on the constraints thing, but I agree with you about ranges. Hopefully, the updated test system will make that easier to do.

[stephenfin]

tox -e docs and tox -e style both fail for me.

I think tox -e style is expected to fail as clearly the codebase isn't consistently following any style yet.

Correct. I have a large PR waiting in the wings. Just need your py3 series to merge first.

However, I think that tox -e docs probably should succeed.

(and then lots more errors)

tox isn't to blame here. Rather my changes in commit 48305e3 are. We're using the include directive and looking for multiquotes using ''' but I changed these to """ without realizing. I'll fix that separately now but it's not a blocker for this.

EDIT nvm, fixed here.

[akrabat]

With a minor fix to the cp command in the docs environment, I think this now works for me too.