Better tests

.github/PULL_REQUEST_TEMPLATE.md

@@ -27,5 +27,5 @@
 - [] My PR was done against the `dev` branch, not `master`.
 - [] My code follows the code style of this project.
 - [] My change includes a change to the documentation, if required.
-- [] My change adds tests as appropriate.
+- [] If my change adds new code, [adequate tests](docs/testing.md) have been added.
 - [] I have read and agree to the **CONTRIBUTING** document.

.gitignore

@@ -1,7 +1,7 @@
 *.out
 *.pyc
 TAGS
-*/__pycache__/
+__pycache__
 tests/*.pyc
 tests/pylint.html
 tests/pylint.txt

Makefile

@@ -11,21 +11,15 @@ TARGET := $(shell lscpu | head -1 | sed -e 's/Architecture:\s*//g')
 COVERAGE_DIR ?= /tmp/cov
 GEF_PATH ?= $(shell readlink -f gef.py)
 TMPDIR ?= /tmp
-PYTEST_PARAMETERS := --verbose -n $(NB_CORES)
-ifdef DEBUG
-	PYTEST_PARAMETERS += --pdb
-endif
+PYTEST_PARAMETERS := --verbose --forked --numprocesses=$(NB_CORES)
 
 .PHONY: test test_% Test% testbins clean lint
 
 test: $(TMPDIR) testbins
-	TMPDIR=$(TMPDIR) python3 -m pytest $(PYTEST_PARAMETERS) tests/runtests.py
-
-Test%: $(TMPDIR) testbins
-	TMPDIR=$(TMPDIR) python3 -m pytest $(PYTEST_PARAMETERS) tests/runtests.py::$@
+	TMPDIR=$(TMPDIR) python3 -m pytest $(PYTEST_PARAMETERS) -k "not benchmark"
 
 test_%: $(TMPDIR) testbins
-	TMPDIR=$(TMPDIR) python3 -m pytest $(PYTEST_PARAMETERS) tests/runtests.py -k $@
+	TMPDIR=$(TMPDIR) python3 -m pytest $(PYTEST_PARAMETERS) -k $@
 
 testbins: $(TMPDIR) $(wildcard tests/binaries/*.c)
 	@TMPDIR=$(TMPDIR) $(MAKE) -j $(NB_CORES) -C tests/binaries TARGET=$(TARGET) all
@@ -36,7 +30,7 @@ clean:
 
 lint:
 	python3 -m pylint $(PYLINT_GEF_PARAMETERS) $(GEF_PATH)
-	python3 -m pylint $(PYLINT_TEST_PARAMETERS) $(wildcard tests/*.py)
+	python3 -m pylint $(PYLINT_TEST_PARAMETERS) $(wildcard tests/*.py tests/*/*.py)
 
 coverage:
 	@! ( [ -d $(COVERAGE_DIR) ] && echo "COVERAGE_DIR=$(COVERAGE_DIR) exists already")

docs/testing.md

@@ -0,0 +1,122 @@
+## Testing GEF
+
+This page describes how GEF testing is done. Any new command/functionality must receive adequate testing to be merged. Also PR failing CI (test + linting) won't be merged either.
+
+
+### Prerequisites
+
+All the prerequisite packages are in `requirements.txt` file at the root of the project. So running
+
+```bash
+python -m pip install -r requirements.txt --user -U
+```
+
+is enough to get started.
+
+
+### Running tests
+
+
+#### Using `make tests`
+
+For testing GEF on the architecture on the host running the tests (most cases), simply run
+
+```bash
+cd /root/of/gef
+make test
+```
+
+Otherwise, `make` can be adjusted to :
+ - run one test/one group of tests specifically
+   - eg. `make test_cmd_heap` will trigger all the test functions for the `heap` command
+ - run for a different architecture
+   - eg. on x86, you can test x86 too with `TARGET=i686 make TARGET=i686 test`
+
+Unless a specific test is mentioned, the command `make test` will run all the non-benchmark related tests.
+At the end, a summary of explanation will be shown, clearly indicating the tests that have failed: for instance:
+
+```
+=================================== short test summary info ==================================
+FAILED tests/commands/heap.py::HeapCommand::test_cmd_heap_bins_large - AssertionError: 'siz...
+FAILED tests/commands/heap.py::HeapCommand::test_cmd_heap_bins_small - AssertionError: 'siz...
+FAILED tests/commands/heap.py::HeapCommand::test_cmd_heap_bins_unsorted - AssertionError: '...
+======================== 3 failed, 4 passed, 113 deselected in 385.77s (0:06:25)==============
+```
+
+You can then use `pytest` directly to help you fix each error specifically.
+
+
+#### Using `pytest`
+
+GEF entirely relies on [`pytest`](https://pytest.org) for its testing. Refer to the project documentation for details.
+
+Adding a new command __requires__ for extensive testing in a new dedicated test module that should be located in `/root/of/gef/tests/commands/my_new_command.py`
+
+A skeleton of a test module would look something like:
+
+```python
+"""
+`my-command` command test module
+"""
+
+
+from tests.utils import GefUnitTestGeneric, gdb_run_cmd, gdb_start_silent_cmd
+
+
+class MyCommandCommand(GefUnitTestGeneric):
+    """`my-command` command test module"""
+
+    def test_cmd_my_command(self):
+        # `my-command` is expected to fail if the session is not active
+        self.assertFailIfInactiveSession(gdb_run_cmd("my-command"))
+
+        # `my-command` should never throw an exception in GDB when running
+        res = gdb_start_silent_cmd("my-command")
+        self.assertNoException(res)
+
+        # it also must print out a "Hello World" message
+        self.assertIn("Hello World", res)
+```
+
+When running your test, you can summon `pytest` with the `--pdb` flag to enter the python testing environment to help you get more information about the reason of failure.
+
+### Linting GEF
+
+You can use the Makefile at the root of the project to get the proper linting settings. For most cases, the following command is enough:
+
+```bash
+cd /root/of/gef
+make lint
+```
+
+
+### Benchmarking GEF
+
+Benchmarking relies on `pytest-benchmark` and is experimental for now.
+
+You can run all benchmark test cases as such:
+
+```bash
+cd /root/of/gef
+pytest -k benchmark
+```
+
+which will return (after some time) an execution summary
+
+```
+tests/perf/benchmark.py ..                                                               [100%]
+
+
+---------------------------------------- benchmark: 3 tests -----------------------------------
+Name (time in ms)          Min                 Max                Mean            StdDev              Median                IQR            Outliers     OPS            Rounds  Iterations
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+time_baseline         612.2325 (1.0)      630.3416 (1.01)     623.7984 (1.01)     7.2848 (1.64)     626.1485 (1.01)      9.9971 (1.81)          1;0  1.6031 (0.99)          5           1
+time_cmd_context      613.8124 (1.00)     625.8964 (1.0)      620.1908 (1.0)      4.4532 (1.0)      619.8831 (1.0)       5.5109 (1.0)           2;0  1.6124 (1.0)           5           1
+time_elf_parsing      616.5053 (1.01)     638.6965 (1.02)     628.1588 (1.01)     8.2465 (1.85)     629.0099 (1.01)     10.7885 (1.96)          2;0  1.5920 (0.99)          5           1
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+
+Legend:
+  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
+  OPS: Operations Per Second, computed as 1 / Mean
+============================================== 3 passed, 117 deselected in 14.78s =============================================
+```