Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

update testing guide of contirbution guide #671

Merged
merged 2 commits into from Nov 4, 2017
Merged

update testing guide of contirbution guide #671

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2976fb4
update testing guide of contirbution guide
kmaehashi Oct 27, 2017
151e552
we assume all CuPy developers have GPU and CUDA
kmaehashi Nov 3, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
56 changes: 23 additions & 33 deletions docs/source/contribution.rst
View file
Open in desktop
Expand Up @@ -147,9 +147,10 @@ Testing Guidelines

Testing is one of the most important part of your code.
You must test your code by unit tests following our testing guidelines.
Note that we are using the nose package and the mock package for testing, so install nose and mock before writing your code::

$ pip install nose mock
Note that we are using pytest and mock package for testing, so install them before writing your code::

$ pip install pytest mock

In order to run unit tests at the repository root, you first have to build Cython files in place by running the following command::

Expand All @@ -174,20 +175,22 @@ In order to run unit tests at the repository root, you first have to build Cytho
If you want to use ccache for nvcc, please install ccache v3.3 or later.
You also need to set environment variable ``NVCC='ccache nvcc'``.

Once the Cython modules are built, you can run unit tests simply by running ``nosetests`` command at the repository root::
Once Cython modules are built, you can run unit tests by running the following command at the repository root::

$ nosetests
$ python -m pytest

It requires CUDA by default.
In order to run unit tests that do not require CUDA, pass ``--attr='!gpu'`` option to the ``nosetests`` command::
It requires CUDA and cuDNN by default.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, I think if a developer does not have CUDA, he cannot develop CuPy...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly... fixed.

In order to run unit tests that do not require CUDA and cuDNN, use ``CUPY_TEST_GPU_LIMIT=0`` environment variable and ``-m='not cudnn'`` option::

$ nosetests path/to/your/test.py --attr='!gpu'
$ export CUPY_TESITNG_GPU_LIMIT=0
$ python -m pytest path/to/your/test.py -m='not cudnn'

Some GPU tests involve multiple GPUs.
If you want to run GPU tests with insufficient number of GPUs, specify the number of available GPUs by ``--eval-attr='gpu<N'`` where ``N`` is a concrete integer.
For example, if you have only one GPU, launch ``nosetests`` by the following command to skip multi-GPU tests::
If you want to run GPU tests with insufficient number of GPUs, specify the number of available GPUs to ``CUPY_TEST_GPU_LIMIT``.
For example, if you have only one GPU, launch ``pytest`` by the following command to skip multi-GPU tests::

$ nosetests path/to/gpu/test.py --eval-attr='gpu<2'
$ export CUPY_TEST_GPU_LIMIT=1
$ python -m pytest path/to/gpu/test.py

Tests are put into the ``tests/cupy_tests`` and ``tests/install_tests`` directories.
These have the same structure as that of ``cupy`` and ``install`` directories, respectively.
Expand All @@ -196,27 +199,17 @@ In order to enable test runner to find test scripts correctly, we are using spec
* The name of each subdirectory of ``tests`` must end with the ``_tests`` suffix.
* The name of each test script must start with the ``test_`` prefix.

Following this naming convention, you can run all the tests by just typing ``nosetests`` at the repository root::
Following this naming convention, you can run all the tests by running the following command at the repository root::

$ nosetests
$ python -m pytest

Or you can also specify a root directory to search test scripts from::

$ nosetests tests/cupy_tests # to just run tests of CuPy
$ nosetests tests/install_tests # to just run tests of installation modules
$ python -m pytest tests/cupy_tests # to just run tests of CuPy
$ python -m pytest tests/install_tests # to just run tests of installation modules

If you modify the code related to existing unit tests, you must run appropriate commands.

.. note::
CuPy tests include type-exhaustive test functions which take long time to execute.
If you are running tests on a multi-core machine, you can parallelize the tests by following options::

$ nosetests --processes=12 --process-timeout=1000 tests/cupy_tests

The magic numbers can be modified for your usage.
Note that some tests require many CUDA compilations, which require a bit long time.
Without the ``process-timeout`` option, the timeout is set shorter, causing timeout failures for many test cases.

There are many examples of unit tests under the ``tests`` directory.
They simply use the ``unittest`` package of the standard library.

Expand All @@ -233,8 +226,9 @@ Test functions that require CUDA must be tagged by the ``cupy.testing.attr.gpu``
def test_my_gpu_func(self):
...

The functions tagged by the ``gpu`` decorator are skipped if ``--attr='!gpu'`` is given.
We also have the ``cupy.testing.attr.cudnn`` decorator to let ``nosetests`` know that the test depends on cuDNN.
The functions tagged by the ``gpu`` decorator are skipped if ``CUPY_TEST_GPU_LIMIT=0`` environment variable is set.
We also have the ``cupy.testing.attr.cudnn`` decorator to let ``pytest`` know that the test depends on cuDNN.
The test functions decorated by ``cudnn`` are skipped if ``-m='not cudnn'`` is given.

The test functions decorated by ``gpu`` must not depend on multiple GPUs.
In order to write tests for multiple GPUs, use ``cupy.testing.attr.multi_gpu()`` or ``cupy.testing.attr.multi_gpu()`` decorators instead::
Expand All @@ -249,15 +243,11 @@ In order to write tests for multiple GPUs, use ``cupy.testing.attr.multi_gpu()``
def test_my_two_gpu_func(self):
...

Once you send a pull request, your code is automatically tested by `Travis-CI <https://travis-ci.org/cupy/cupy/>`_ **with --attr='!gpu,!slow' option**.
Since Travis-CI does not support CUDA, we cannot check your CUDA-related code automatically.
The reviewing process starts after the test passes.
Once you send a pull request, `Travis-CI <https://travis-ci.org/cupy/cupy/>`_ automatically checks if your code meets our coding guidelines described above.
Since Travis-CI does not support CUDA, we cannot run unit tests automatically.
The reviewing process starts after the automatic check passes.
Note that reviewers will test your code without the option to check CUDA-related code.

.. note::
Some of numerically unstable tests might cause errors irrelevant to your changes.
In such a case, we ignore the failures and go on to the review process, so do not worry about it.

We leverage doctest as well. You can run doctest by typing ``make doctest`` at the ``docs`` directory::

$ cd docs
Expand Down