Tox.ini does not work out of the box

[pganssle]

Bug report

Running tox on both my local environment and apparently on Travis (see #11360) causes an issue because "Matplotlib test data is not installed".

As part of #11360, I also tried to create a custom pytz testenv that invokes pytest -m pytz and it failed during test discovery because it could not find mpl_toolkits. Ideally, running tox out of the box should work, and it should also be possible to use it to run specific subsets of the tests.

Code for reproduction

$ tox -e py36

Actual outcome
Output is spammy, hidden under a <details> tag:

$ tox -e py36
GLOB sdist-make: LOCATION/matplotlib/setup.py
py36 inst-nodeps: LOCATION/matplotlib/.tox/dist/matplotlib-2.0.2+8083.g391c0cb30.zip
py36 installed: atomicwrites==1.1.5,attrs==18.1.0,cycler==0.10.0,kiwisolver==1.0.1,matplotlib==2.0.2+8083.g391c0cb30,more-itertools==4.2.0,numpy==1.14.3,pluggy==0.6.0,py==1.5.3,pyparsing==2.2.0,pytest==3.6.0,python-dateutil==2.7.3,six==1.11.0
py36 runtests: PYTHONHASHSEED='735566734'
py36 runtests: commands[0] | sh -c rm -f $HOME/.matplotlib/fontList*
WARNING:test command found but not installed in testenv
  cmd: /usr/bin/sh
  env: LOCATION/matplotlib/.tox/py36
Maybe you forgot to specify a dependency? See also the whitelist_externals envconfig setting.
py36 runtests: commands[1] | LOCATION/matplotlib/.tox/py36/bin/python LOCATION/matplotlib/tests.py --processes=-1 --process-timeout=300
Python byte-compilation optimization level: 0
LOCATION/matplotlib/.tox/py36/lib/python3.6/site-packages/matplotlib/__init__.py:1440: UserWarning: Matplotlib is not built with the correct FreeType version to run tests.  Set local_freetype=True in setup.cfg and rebuild. Expect many image comparison failures below. Expected freetype version 2.6.1. Found freetype version 2.9.1. Freetype build type is not local
  "" if ft2font.__freetype_build_type__ == 'local' else "not "
Traceback (most recent call last):
  File "LOCATION/matplotlib/tests.py", line 56, in <module>
    recursionlimit=args.recursionlimit)
  File "LOCATION/matplotlib/.tox/py36/lib/python3.6/site-packages/matplotlib/__init__.py", line 1456, in test
    raise ImportError("Matplotlib test data is not installed")
ImportError: Matplotlib test data is not installed
ERROR: InvocationError for command 'LOCATION/matplotlib/.tox/py36/bin/python LOCATION/matplotlib/tests.py --processes=-1 --process-timeout=300' (exited with code 1)

Expected outcome
Should run the tests properly.

Matplotlib version

• Operating system: Linux
• Matplotlib version: master (git checkout 391c0cb3089ac9336b7ca80)
• Python version: 3.6.5
• Other libraries:

$ pip freeze
pluggy==0.6.0
py==1.5.3
six==1.11.0
tox==3.0.0
virtualenv==16.0.0

[jklymak]

I guess I don't know what tox is, but it looks like you didn't setup the setup.cfg in your environment for tests? https://matplotlib.org/devel/contributing.html#building-matplotlib-for-image-comparison-tests

[pganssle]

@jklymak That may indeed be the case, though since tox is only used for tests and creates its own environment, it should probably be setting those values correctly itself.

[anntzer]

diff --git a/tox.ini b/tox.ini
index 9407b70d6..e3bb93c7d 100644
--- a/tox.ini
+++ b/tox.ini
@@ -7,10 +7,13 @@
 envlist = py35, py36

 [testenv]
-changedir = /tmp
+setenv =
+    PIP_ISOLATED = 1
+    PIP_CONFIG_FILE = /dev/null
+    MPLLOCALFREETYPE = 1
+usedevelop = True
+changedir = {envtmpdir}
 commands =
-    sh -c 'rm -f $HOME/.matplotlib/fontList*'
-    {envpython} {toxinidir}/tests.py --processes=-1 --process-timeout=300
+    pytest --pyargs matplotlib
 deps =
-    numpy
     pytest

appears to make tox work. Perhaps also set MPLCONFIGDIR to e.g. {envtmpdir}/mplconfigdir or similar, and bikeshed the pytest default args; and also define the [test] extras_require (sphinx, pillow, etc.)