Add docs for new test suite (#39)

docs/manual/installation.rst

@@ -28,7 +28,7 @@ correctly install following software.
 
   Generally, you can install it just from your distrubution's repository::
     
-    $ sudo apt-get install nvidia-cuda-toolkit
+    sudo apt-get install nvidia-cuda-toolkit
 
   Although, default packages are often out of date, so in case you
   have one of those latest cool GPU, you may want to upgrade to the
@@ -40,20 +40,20 @@ correctly install following software.
 
   Your distribution should already have all you need::
 
-    $ sudo apt-get install python3 python3-dev python3-pip wheel
+    sudo apt-get install python3 python3-dev python3-pip wheel
 
 - `NumPy`_
 
   Once Python3 is correctly installed, you can install NumPy by::
 
-    $ pip3 install numpy
+    pip3 install numpy
 
 - `PyCUDA`_
 
   If CUDA is correctly installed, you again can simply install PyCUDA
   with ``pip``::
 
-    $ pip3 install pycuda
+    pip3 install pycuda
 
 Other pre-requisites should be transparently installed with the main
 Xentica package.
@@ -66,7 +66,7 @@ to install `Kivy framework`_.
 
 Its pre-requisites could be installed by::
 
-  $ sudo apt-get install \
+  sudo apt-get install \
     build-essential \
     git \
     ffmpeg \
@@ -82,13 +82,13 @@ Its pre-requisites could be installed by::
 
 Then, to install stable Kivy::
 
-  $ pip3 install Cython==0.25 Kivy==1.10.0
+  pip3 install Cython==0.25 Kivy==1.10.0
 
 On latest Debian distributions you can meet conflicts with
 ``libsdl-mixer``. Then, try to install latest developer version,
 like::
 
-  $ pip3 install Cython==0.27.3 git\+https://github.com/kivy/kivy.git
+  pip3 install Cython==0.27.3 git\+https://github.com/kivy/kivy.git
 
 If you have any other troubles with that, please refer to the official
 Kivy installation instructions.
@@ -98,7 +98,7 @@ Main package
 
 Xentica package could be installed with::
 
-  $ pip3 install xentica
+  pip3 install xentica
 
 Note, it does not depends on pre-requisites described above, but you
 still need to install them properly, or Xentica will not run.
@@ -108,29 +108,87 @@ Run Xentica examples
 
 In order to run Game of Life model built with Xentica::
 
-  $ pip3 install moire
-  $ wget https://raw.githubusercontent.com/a5kin/xentica/master/examples/game_of_life.py
-  $ python3 game_of_life.py
+  pip3 install moire
+  wget https://raw.githubusercontent.com/a5kin/xentica/master/examples/game_of_life.py
+  python3 game_of_life.py
 
 Or, if you are using ``optirun``::
 
-  $ optirun python3 game_of_life.py
+  optirun python3 game_of_life.py
 
 Run tests
 ---------
 
-In order to run Xentica tests::
+Our test suite is based on `tox`_. It allows us to run tests in all
+supported Python environments with a single ``tox`` command, and also
+automates checks for package build, docs, coverage, code style and
+performance.
 
-  $ git clone https://github.com/a5kin/xentica.git
-  $ cd xentica/tests
-  $ ./run_tests.sh
+So, in order to run Xentica tests you at minimum have to set up CUDA
+environment properly (as described above), and install ``tox``::
+
+  pip3 install tox
+
+If you are planning to run full test suite, you also need to install
+all necessary Python interpreters: 3.5-3.7 and pypy3, along with dev
+headers to build numpy and pycuda.
+
+On Ubuntu, regular Python interpreters are available with amazing
+`deadsnakes`_ repo::
+
+  sudo add-apt-repository ppa:deadsnakes/ppa
+  sudo apt-get update
+  sudo apt-get install python3.5 python3.5-dev
+  sudo apt-get install python3.6 python3.6-dev
+  sudo apt-get install python3.7 python3.7-dev
+
+Pypy3 however comes `in binaries`_ (make sure you download latest)::
+
+  wget -q -P /tmp https://bitbucket.org/pypy/pypy/downloads/pypy3.5-v7.0.0-linux64.tar.bz2
+  sudo tar -x -C /opt -f /tmp/pypy3.5-v7.0.0-linux64.tar.bz2
+  rm /tmp/pypy3.5-v7.0.0-linux64.tar.bz2
+  sudo mv /opt/pypy3.5-v7.0.0-linux64 /opt/pypy3
+  sudo ln -s /opt/pypy3/bin/pypy3 /usr/local/bin/pypy3
+
+Then, to run full tests::
+
+  git clone https://github.com/a5kin/xentica.git
+  cd xentica
+  tox
 
 Or, if you are using ``optirun``::
 
-  $ optirun python3 run_tests.py
+  optirun tox
+
+For the first time, it would take an amount of time to download /
+install environtments and all its dependencies. Tox will automatically
+set up all necessary stuff for you, including numpy and
+pycuda. Subsequent runs should be much quicker, as everything is
+already set up. In developer's environment (Ubuntu 18.04) it takes ~42
+sec to finish the full test suite.
+
+If you run tests often, it would also be helpful to get less verbose
+output. For that, you could execute a strict version of tox::
+
+  tox -q
+
+Or if you'd like to skip all uninstalled interpreters::
+
+  tox -s
+
+Or even quicker, for immediate test purposes, you could run your
+default Python3 interpreter tests only with codestyle and coverage::
+
+  tox -q -e flake8,coverage
+
+You could also check the full list of available environments with::
+
+  tox -l -v
 
-Tests will also give you performance metrics. Please update us with
-them, along with the info about you GPU and Xentica version.
+If you don't mind, please update us with the metrics under "Benchmark"
+section, along with the info about your GPU, environment and Xentica
+version. It would help us analyze performance and plan for future
+optimizations.
 
 .. _list of CUDA-enabled cards: https://developer.nvidia.com/cuda-gpus
 .. _NVIDIA CUDA Toolkit: http://docs.nvidia.com/cuda/index.html
@@ -144,3 +202,6 @@ them, along with the info about you GPU and Xentica version.
 .. _opening an issue: https://github.com/a5kin/xentica/issues/new
 .. _good article: http://www.pradeepadiga.me/blog/2017/03/22/installing-cuda-toolkit-8-0-on-ubuntu-16-04/
 .. _official NVIDIA source: https://developer.nvidia.com/cuda-downloads
+.. _tox: https://https://launchpad.net/~deadsnakes/+archive/ubuntu/ppatox.readthedocs.io/en/latest/
+.. _deadsnakes: https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa
+.. _in binaries: https://pypy.org/download.html

tox.ini

@@ -1,9 +1,10 @@
 [tox]
 envlist =
     py{35,36,37,py3},
-    sanity,docs,pylint,flake8,coverage,benchmark
+    sanity,docs,flake8,pylint,coverage,benchmark
 
 [testenv]
+description = "Environment: {envname}"
 commands_pre =
     python3 tests/title.py "Environment: {envname}"
 commands =
@@ -16,7 +17,8 @@ deps =
     pytest
 
 [testenv:coverage]
-basepython = python3.6
+description = "Coverage"
+basepython = python3
 commands_pre =
     python3 tests/title.py "Coverage"
 commands =
@@ -31,9 +33,10 @@ setenv =
     COVERAGE_FILE=.coverage
 
 [testenv:docs]
+description = "Documentation"
 commands_pre =
     python3 tests/title.py "Documentation"
-basepython = python3.6
+basepython = python3
 whitelist_externals = make
 commands =
     make -C docs html
@@ -44,37 +47,41 @@ deps =
     sphinx_rtd_theme
 
 [testenv:flake8]
+description = "Codestyle Check"
 commands_pre =
     python3 tests/title.py "Codestyle Check"
-basepython = python3.6
+basepython = python3
 deps =
     flake8
     flake8-docstrings
 commands =
     flake8 .
 
 [testenv:pylint]
+description = "Static Analysis"
 commands_pre =
     python3 tests/title.py "Static Analysis"
-basepython = python3.6
+basepython = python3
 deps =
     pylint
 commands =
     pylint xentica examples docs tests
 
 [testenv:benchmark]
+description = "Benchmark"
 commands_pre =
     python3 tests/title.py "Benchmark"
-basepython = python3.6
+basepython = python3
 commands =
     python -m pip -q install numpy wheel
     python -m pip -q install pycuda
     python3 tests/benchmark.py
 
 [testenv:sanity]
+description = "Sanity (Build) Checks"
 commands_pre =
     python3 tests/title.py "Sanity Checks"
-basepython = python3.6
+basepython = python3
 commands =
     check-manifest --ignore tox*.ini,tests*,data*
     python setup.py check -m -r -s