Tidy up tests (#120)

- Remove pylint disables in benchmarks - Move tests out of core and extras - Update info about tests in CONTRIBUTING
icaros-usc · Apr 4, 2021 · e6add5e · e6add5e
1 parent b925995
commit e6add5e
Show file tree

Hide file tree

Showing 63 changed files with 45 additions and 60 deletions.
diff --git a/.gitignore b/.gitignore
@@ -119,7 +119,7 @@ ENV/
 *.pth
 MNIST/
 dask-worker-space/
-!tests/extras/baseline_images/visualize_test/*
+!tests/baseline_images/visualize_test/*
 !docs/_static/**/*
 !examples/tutorials/_static/**/*
 !readme_assets/*

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -34,10 +34,10 @@ Ready to contribute? Here's how to set up pyribs for local development.
 2. Clone the fork locally:
 
    ```bash
-   # If you have SSH set up:
+   # With SSH:
    git clone git@github.com:USERNAME/pyribs.git
 
-   # Or, if you do not have SSH set up:
+   # Without SSH:
    git clone https://github.com/USERNAME/pyribs.git
    ```
 
@@ -74,21 +74,26 @@ Ready to contribute? Here's how to set up pyribs for local development.
 5. After making changes, check that the changes pass the tests:
 
    ```bash
-   pylint ribs tests
    pytest tests/
    make test # ^ same as above
-   tox  # Don't worry if this does not run; we will run it in CI/CD
    ```
 
-   To run the tests without benchmarks (benchmarks can take a while), run:
+   And to run the benchmarks:
 
    ```bash
-   pytest -c pytest_no_benchmark.ini
-   make test-only # ^ same as above, but shorter
+   pytest -c pytest_benchmark.ini
+   make benchmark # ^ same as above
    ```
 
-   To get pytest, pylint, and tox, pip install them into the environment.
-   However, they should already install with `pip install -e .[dev]`.
+   Finally, to lint the code:
+
+   ```bash
+   pylint ribs tests benchmarks examples
+   make lint # ^ same as above
+   ```
+
+   To get pytest and pylint, pip install them into the environment. However,
+   they should already install with `pip install -e .[dev]`.
 
 6. Commit the changes and push the branch to GitHub:
 
@@ -151,9 +156,9 @@ Tutorials are created in Jupyter notebooks that are stored under
 1. Before the main loop of the QD algorithm, include a line like
    `total_itrs = 500` (any other integer will work). This line will be replaced
    during testing (see `tests/tutorials.sh`) in order to test that the notebook
-   runs end-to-end. By default, the tests run the notebook with `total_itrs =
-   5`. If this tutorial needs more (or less), modify the switch-case statement
-   in `tests/tutorials.sh`.
+   runs end-to-end. By default, the tests run the notebook with
+   `total_itrs = 5`. If this tutorial needs more (or less), modify the
+   switch-case statement in `tests/tutorials.sh`.
 1. Make sure that the only level 1 heading (e.g. `# Awesome Tutorial`) is the
    title at the top of the notebook. Subsequent titles should be level 2 (e.g.
    `## Level 2 Heading`) or higher.

diff --git a/Makefile b/Makefile
@@ -49,14 +49,6 @@ test: ## run tests with the default Python
 	pytest tests
 .PHONY: test
 
-test-core: ## only test the core of ribs
-	pytest tests/core
-.PHONY: test-core
-
-test-extras: ## only test the extras of ribs
-	pytest tests/extras
-.PHONY: test-extras
-
 test-coverage: ## get better test coverage by running without numba on
 	NUMBA_DISABLE_JIT=1 pytest tests
 .PHONY: test-coverage

diff --git a/tests/README.md b/tests/README.md
@@ -1,15 +1,9 @@
 # Tests
 
-This directory contains tests and micro-benchmarks for ribs. The tests are
-separated into:
-
-1. `core`: tests for the part of ribs that works with `pip install ribs`
-2. `extras`: tests for the extra parts of ribs that only work when one installs
-   the full ribs with `pip install ribs[all]`
-
-To run these tests, install the dev dependencies for ribs with
-`pip install ribs[dev]` or `pip install -e .[dev]` (from the root directory of
-the repo).
+This directory contains tests and micro-benchmarks for ribs. The tests mirror
+the directory structure of `ribs`. To run these tests, install the dev
+dependencies for ribs with `pip install ribs[dev]` or `pip install -e .[dev]`
+(from the root directory of the repo).
 
 For information on running tests, see [CONTRIBUTING.md](../CONTRIBUTING.md).
 

diff --git a/tests/core/archives/README.md → tests/archives/README.md b/tests/core/archives/README.md → tests/archives/README.md
@@ -1,12 +1,11 @@
 # Steps for Testing an Archive
 
 1. Add the name of the archive to `conftest.py` under `ARCHIVE_NAMES`.
-1. Create data for it in `get_archive_data()`.
+1. Create data for it in `get_archive_data()` in `conftest.py`.
 1. Parts of the archive should now be automatically tested in
    `archive_base_test.py`.
-1. To test the rest of the archive, in particular the `add()` and `as_pandas()`
-   methods, start a new test file for it. For reference, see
-   `grid_archive_test.py`.
+1. To test the rest of the archive, in particular the `add()` method, start a
+   new test file for it. For reference, see `grid_archive_test.py`.
 1. Just like the `_data` fixture in `grid_archive_test.py`, create a
    fixture that uses `get_archive_data()` to retrieve test data. Feed that
    fixture into all the tests.

diff --git a/tests/core/__init__.py → tests/archives/__init__.py b/tests/core/__init__.py → tests/archives/__init__.py
diff --git a/tests/core/archives/add_status_test.py → tests/archives/add_status_test.py b/tests/core/archives/add_status_test.py → tests/archives/add_status_test.py
diff --git a/tests/core/archives/archive_base_test.py → tests/archives/archive_base_test.py b/tests/core/archives/archive_base_test.py → tests/archives/archive_base_test.py
@@ -52,8 +52,7 @@ def test_solution_dim_requires_init(_data):
 
 
 #
-# Test the dtypes of all classes, as some classes use the dtype in their
-# constructor.
+# Test the dtypes of all classes.
 #
 
 

diff --git a/tests/core/archives/conftest.py → tests/archives/conftest.py b/tests/core/archives/conftest.py → tests/archives/conftest.py
diff --git a/tests/core/archives/cvt_archive_benchmark.py → tests/archives/cvt_archive_benchmark.py b/tests/core/archives/cvt_archive_benchmark.py → tests/archives/cvt_archive_benchmark.py
@@ -3,19 +3,18 @@
 
 from ribs.archives import CVTArchive
 
-# pylint: disable = unused-variable
-
 
 def benchmark_init(use_kd_tree, benchmark):
     """Construction includes k-means clustering and building a kd-tree."""
 
-    @benchmark
     def init():
         archive = CVTArchive(1000, [(-1, 1), (-1, 1)],
                              samples=20_000,
                              use_kd_tree=use_kd_tree)
         archive.initialize(solution_dim=2)
 
+    benchmark(init)
+
 
 def benchmark_add_10k(use_kd_tree, benchmark, benchmark_data_10k):
     n, solutions, objective_values, behavior_values = benchmark_data_10k
@@ -47,11 +46,12 @@ def benchmark_get_10k_random_elites(use_kd_tree, benchmark, benchmark_data_10k):
     for i in range(n):
         archive.add(solutions[i], objective_values[i], behavior_values[i])
 
-    @benchmark
     def get_elites():
-        for i in range(n):
+        for _ in range(n):
             archive.get_random_elite()
 
+    benchmark(get_elites)
+
 
 def benchmark_as_pandas_2000_items(benchmark):
     bins = 2000

diff --git a/tests/core/archives/cvt_archive_test.py → tests/archives/cvt_archive_test.py b/tests/core/archives/cvt_archive_test.py → tests/archives/cvt_archive_test.py
diff --git a/...s/core/archives/grid_archive_benchmark.py → tests/archives/grid_archive_benchmark.py b/...s/core/archives/grid_archive_benchmark.py → tests/archives/grid_archive_benchmark.py
@@ -3,8 +3,6 @@
 
 from ribs.archives import GridArchive
 
-# pylint: disable = unused-variable
-
 
 def benchmark_add_10k(benchmark, benchmark_data_10k):
     n, solutions, objective_values, behavior_values = benchmark_data_10k
@@ -32,11 +30,12 @@ def benchmark_get_10k_random_elites(benchmark, benchmark_data_10k):
     for i in range(n):
         archive.add(solutions[i], objective_values[i], behavior_values[i])
 
-    @benchmark
     def get_elites():
-        for i in range(n):
+        for _ in range(n):
             archive.get_random_elite()
 
+    benchmark(get_elites)
+
 
 def benchmark_as_pandas_2025_items(benchmark):
     dim = 45

diff --git a/tests/core/archives/grid_archive_test.py → tests/archives/grid_archive_test.py b/tests/core/archives/grid_archive_test.py → tests/archives/grid_archive_test.py
diff --git a/...s/sliding_boundaries_archive_benchmark.py → ...s/sliding_boundaries_archive_benchmark.py b/...s/sliding_boundaries_archive_benchmark.py → ...s/sliding_boundaries_archive_benchmark.py
@@ -3,8 +3,6 @@
 
 from ribs.archives import SlidingBoundariesArchive
 
-# pylint: disable = unused-variable
-
 
 def benchmark_add_10k(benchmark, benchmark_data_10k):
     n, solutions, objective_values, behavior_values = benchmark_data_10k
@@ -37,11 +35,12 @@ def benchmark_get_10k_random_elites(benchmark, benchmark_data_10k):
     for i in range(n):
         archive.add(solutions[i], objective_values[i], behavior_values[i])
 
-    @benchmark
     def get_elites():
-        for i in range(n):
+        for _ in range(n):
             archive.get_random_elite()
 
+    benchmark(get_elites)
+
 
 def benchmark_as_pandas_2048_elements(benchmark):
     # TODO (btjanaka): Make this size smaller so that we do a remap.

diff --git a/...chives/sliding_boundaries_archive_test.py → ...chives/sliding_boundaries_archive_test.py b/...chives/sliding_boundaries_archive_test.py → ...chives/sliding_boundaries_archive_test.py
diff --git a/...es/visualize_test/cvt_archive_heatmap.png → ...es/visualize_test/cvt_archive_heatmap.png b/...es/visualize_test/cvt_archive_heatmap.png → ...es/visualize_test/cvt_archive_heatmap.png
diff --git a/...sualize_test/cvt_archive_heatmap_long.png → ...sualize_test/cvt_archive_heatmap_long.png b/...sualize_test/cvt_archive_heatmap_long.png → ...sualize_test/cvt_archive_heatmap_long.png
diff --git a/..._test/cvt_archive_heatmap_long_square.png → ..._test/cvt_archive_heatmap_long_square.png b/..._test/cvt_archive_heatmap_long_square.png → ..._test/cvt_archive_heatmap_long_square.png
diff --git a/...st/cvt_archive_heatmap_long_transpose.png → ...st/cvt_archive_heatmap_long_transpose.png b/...st/cvt_archive_heatmap_long_transpose.png → ...st/cvt_archive_heatmap_long_transpose.png
diff --git a/...vt_archive_heatmap_with_coolwarm_cmap.png → ...vt_archive_heatmap_with_coolwarm_cmap.png b/...vt_archive_heatmap_with_coolwarm_cmap.png → ...vt_archive_heatmap_with_coolwarm_cmap.png
diff --git a/..._test/cvt_archive_heatmap_with_limits.png → ..._test/cvt_archive_heatmap_with_limits.png b/..._test/cvt_archive_heatmap_with_limits.png → ..._test/cvt_archive_heatmap_with_limits.png
diff --git a/.../cvt_archive_heatmap_with_listed_cmap.png → .../cvt_archive_heatmap_with_listed_cmap.png b/.../cvt_archive_heatmap_with_listed_cmap.png → .../cvt_archive_heatmap_with_listed_cmap.png
diff --git a/...test/cvt_archive_heatmap_with_samples.png → ...test/cvt_archive_heatmap_with_samples.png b/...test/cvt_archive_heatmap_with_samples.png → ...test/cvt_archive_heatmap_with_samples.png
diff --git a/...s/visualize_test/grid_archive_heatmap.png → ...s/visualize_test/grid_archive_heatmap.png b/...s/visualize_test/grid_archive_heatmap.png → ...s/visualize_test/grid_archive_heatmap.png
diff --git a/...ualize_test/grid_archive_heatmap_long.png → ...ualize_test/grid_archive_heatmap_long.png b/...ualize_test/grid_archive_heatmap_long.png → ...ualize_test/grid_archive_heatmap_long.png
diff --git a/...test/grid_archive_heatmap_long_square.png → ...test/grid_archive_heatmap_long_square.png b/...test/grid_archive_heatmap_long_square.png → ...test/grid_archive_heatmap_long_square.png
diff --git a/...t/grid_archive_heatmap_long_transpose.png → ...t/grid_archive_heatmap_long_transpose.png b/...t/grid_archive_heatmap_long_transpose.png → ...t/grid_archive_heatmap_long_transpose.png
diff --git a/.../grid_archive_heatmap_with_boundaries.png → .../grid_archive_heatmap_with_boundaries.png b/.../grid_archive_heatmap_with_boundaries.png → .../grid_archive_heatmap_with_boundaries.png
diff --git a/...id_archive_heatmap_with_coolwarm_cmap.png → ...id_archive_heatmap_with_coolwarm_cmap.png b/...id_archive_heatmap_with_coolwarm_cmap.png → ...id_archive_heatmap_with_coolwarm_cmap.png
diff --git a/...test/grid_archive_heatmap_with_limits.png → ...test/grid_archive_heatmap_with_limits.png b/...test/grid_archive_heatmap_with_limits.png → ...test/grid_archive_heatmap_with_limits.png
diff --git a/...grid_archive_heatmap_with_listed_cmap.png → ...grid_archive_heatmap_with_listed_cmap.png b/...grid_archive_heatmap_with_listed_cmap.png → ...grid_archive_heatmap_with_listed_cmap.png
diff --git a/...mages/visualize_test/parallel_axes_2d.png → ...mages/visualize_test/parallel_axes_2d.png b/...mages/visualize_test/parallel_axes_2d.png → ...mages/visualize_test/parallel_axes_2d.png
diff --git a/...mages/visualize_test/parallel_axes_3d.png → ...mages/visualize_test/parallel_axes_3d.png b/...mages/visualize_test/parallel_axes_3d.png → ...mages/visualize_test/parallel_axes_3d.png
diff --git a/...ualize_test/parallel_axes_3d_coolwarm.png → ...ualize_test/parallel_axes_3d_coolwarm.png b/...ualize_test/parallel_axes_3d_coolwarm.png → ...ualize_test/parallel_axes_3d_coolwarm.png
diff --git a/...ze_test/parallel_axes_3d_custom_names.png → ...ze_test/parallel_axes_3d_custom_names.png b/...ze_test/parallel_axes_3d_custom_names.png → ...ze_test/parallel_axes_3d_custom_names.png
diff --git a/...allel_axes_3d_custom_objective_limits.png → ...allel_axes_3d_custom_objective_limits.png b/...allel_axes_3d_custom_objective_limits.png → ...allel_axes_3d_custom_objective_limits.png
diff --git a/...ze_test/parallel_axes_3d_custom_order.png → ...ze_test/parallel_axes_3d_custom_order.png b/...ze_test/parallel_axes_3d_custom_order.png → ...ze_test/parallel_axes_3d_custom_order.png
diff --git a/...isualize_test/parallel_axes_3d_sorted.png → ...isualize_test/parallel_axes_3d_sorted.png b/...isualize_test/parallel_axes_3d_sorted.png → ...isualize_test/parallel_axes_3d_sorted.png
diff --git a/...e_test/parallel_axes_3d_vertical_cbar.png → ...e_test/parallel_axes_3d_vertical_cbar.png b/...e_test/parallel_axes_3d_vertical_cbar.png → ...e_test/parallel_axes_3d_vertical_cbar.png
diff --git a/...e_test/parallel_axes_3d_width2_alpha2.png → ...e_test/parallel_axes_3d_width2_alpha2.png b/...e_test/parallel_axes_3d_width2_alpha2.png → ...e_test/parallel_axes_3d_width2_alpha2.png
diff --git a/...alize_test/sliding_boundaries_heatmap.png → ...alize_test/sliding_boundaries_heatmap.png b/...alize_test/sliding_boundaries_heatmap.png → ...alize_test/sliding_boundaries_heatmap.png
diff --git a/..._test/sliding_boundaries_heatmap_long.png → ..._test/sliding_boundaries_heatmap_long.png b/..._test/sliding_boundaries_heatmap_long.png → ..._test/sliding_boundaries_heatmap_long.png
diff --git a/...liding_boundaries_heatmap_long_square.png → ...liding_boundaries_heatmap_long_square.png b/...liding_boundaries_heatmap_long_square.png → ...liding_boundaries_heatmap_long_square.png
diff --git a/...ing_boundaries_heatmap_long_transpose.png → ...ing_boundaries_heatmap_long_transpose.png b/...ing_boundaries_heatmap_long_transpose.png → ...ing_boundaries_heatmap_long_transpose.png
diff --git a/...ng_boundaries_heatmap_with_boundaries.png → ...ng_boundaries_heatmap_with_boundaries.png b/...ng_boundaries_heatmap_with_boundaries.png → ...ng_boundaries_heatmap_with_boundaries.png
diff --git a/...boundaries_heatmap_with_coolwarm_cmap.png → ...boundaries_heatmap_with_coolwarm_cmap.png b/...boundaries_heatmap_with_coolwarm_cmap.png → ...boundaries_heatmap_with_coolwarm_cmap.png
diff --git a/...liding_boundaries_heatmap_with_limits.png → ...liding_boundaries_heatmap_with_limits.png b/...liding_boundaries_heatmap_with_limits.png → ...liding_boundaries_heatmap_with_limits.png
diff --git a/...g_boundaries_heatmap_with_listed_cmap.png → ...g_boundaries_heatmap_with_listed_cmap.png b/...g_boundaries_heatmap_with_listed_cmap.png → ...g_boundaries_heatmap_with_listed_cmap.png
diff --git a/tests/core/optimizers/__init__.py b/tests/core/optimizers/__init__.py
diff --git a/tests/core/archives/__init__.py → tests/emitters/__init__.py b/tests/core/archives/__init__.py → tests/emitters/__init__.py
diff --git a/tests/core/emitters/cma_me_emitter_test.py → tests/emitters/cma_me_emitter_test.py b/tests/core/emitters/cma_me_emitter_test.py → tests/emitters/cma_me_emitter_test.py
diff --git a/tests/core/emitters/conftest.py → tests/emitters/conftest.py b/tests/core/emitters/conftest.py → tests/emitters/conftest.py
diff --git a/tests/core/emitters/emitter_base_test.py → tests/emitters/emitter_base_test.py b/tests/core/emitters/emitter_base_test.py → tests/emitters/emitter_base_test.py
diff --git a/...re/emitters/gaussian_emitter_benchmark.py → tests/emitters/gaussian_emitter_benchmark.py b/...re/emitters/gaussian_emitter_benchmark.py → tests/emitters/gaussian_emitter_benchmark.py
@@ -4,8 +4,6 @@
 
 from ribs.emitters import GaussianEmitter
 
-# pylint: disable = unused-variable
-
 
 def benchmark_ask_tell_100k(benchmark, fake_archive_fixture):
     archive, x0 = fake_archive_fixture
@@ -26,10 +24,11 @@ def benchmark_ask_tell_100k(benchmark, fake_archive_fixture):
     obj_vals = np.random.rand(n, batch_size)
     behavior_vals = np.random.rand(n, batch_size, 2)
 
-    @benchmark
     def ask_and_tell():
         for i in range(n):
             solutions = emitter.ask()
             objective_values = obj_vals[i]
             behavior_values = behavior_vals[i]
             emitter.tell(solutions, objective_values, behavior_values)
+
+    benchmark(ask_and_tell)
diff --git a/tests/core/emitters/gaussian_emitter_test.py → tests/emitters/gaussian_emitter_test.py b/tests/core/emitters/gaussian_emitter_test.py → tests/emitters/gaussian_emitter_test.py
diff --git a/...re/emitters/iso_line_emitter_benchmark.py → tests/emitters/iso_line_emitter_benchmark.py b/...re/emitters/iso_line_emitter_benchmark.py → tests/emitters/iso_line_emitter_benchmark.py
@@ -4,8 +4,6 @@
 
 from ribs.emitters import IsoLineEmitter
 
-# pylint: disable = unused-variable
-
 
 def benchmark_ask_tell_100k(benchmark, fake_archive_fixture):
     archive, x0 = fake_archive_fixture
@@ -25,10 +23,11 @@ def benchmark_ask_tell_100k(benchmark, fake_archive_fixture):
     obj_vals = np.random.rand(n, batch_size)
     behavior_vals = np.random.rand(n, batch_size, 2)
 
-    @benchmark
     def ask_and_tell():
         for i in range(n):
             solutions = emitter.ask()
             objective_values = obj_vals[i]
             behavior_values = behavior_vals[i]
             emitter.tell(solutions, objective_values, behavior_values)
+
+    benchmark(ask_and_tell)
diff --git a/tests/core/emitters/iso_line_emitter_test.py → tests/emitters/iso_line_emitter_test.py b/tests/core/emitters/iso_line_emitter_test.py → tests/emitters/iso_line_emitter_test.py
diff --git a/tests/extras/__init__.py b/tests/extras/__init__.py
diff --git a/tests/core/factory_test.py → tests/factory_test.py b/tests/core/factory_test.py → tests/factory_test.py
diff --git a/tests/core/emitters/__init__.py → tests/optimizers/__init__.py b/tests/core/emitters/__init__.py → tests/optimizers/__init__.py
diff --git a/tests/core/optimizers/optimizer_test.py → tests/optimizers/optimizer_test.py b/tests/core/optimizers/optimizer_test.py → tests/optimizers/optimizer_test.py
diff --git a/tests/extras/visualize_test.py → tests/visualize_test.py b/tests/extras/visualize_test.py → tests/visualize_test.py
diff --git a/tox.ini b/tox.ini
@@ -31,9 +31,9 @@ deps =
     benchmarks: .[all]
 commands =
     pip install -U pip
-    ribs_core: pytest --basetemp={envtmpdir} tests/core
-    ribs_extras: pytest --basetemp={envtmpdir} tests/extras
-    pin_core: pytest --basetemp={envtmpdir} tests/core
-    pin_extras: pytest --basetemp={envtmpdir} tests/extras
+    ribs_core: pytest --basetemp={envtmpdir} tests/archives tests/emitters tests/optimizers tests/factory_test.py
+    ribs_extras: pytest --basetemp={envtmpdir} tests/visualize_test.py
+    pin_core: pytest --basetemp={envtmpdir} tests/archives tests/emitters tests/optimizers tests/factory_test.py
+    pin_extras: pytest --basetemp={envtmpdir} tests/visualize_test.py
     coverage: pytest --basetemp={envtmpdir} tests
     benchmarks: pytest --basetemp={envtmpdir} -c pytest_benchmark.ini tests