Better tests (#813)

* first shot at refactoring ci testing

* - Added skeleton for most missing command test modules
- Fixed a logic error in `NopCommand`

* [ci] added tests for `nop`

* [ci] extra test for nop for memory check

* added benchmarking capability, can be triggered directly from `pytest`

* - fixed `pcustom` command test for 32b
- fixed calculation of tcache in gef
- added tests for `gef.heap`
- improved tests for `heap` command
- fixes #641

* [ci] cmd/heap - adjusted tcachebins indexes for 32b

* damnit

* - fixed linting
- fixed test `highlight` for 32b
- fixed test `pattern_search` for x86

* last fixes for tonight

* fixed `pattern` and `heap` tests for good

* - add 3rd party module check for `capstone`, `keystone`, `unicorn` and `ropper`

* added `test_func_update_gef`

* `make test` doesn't execute benchmark

* - fixed errors in the `pie` subcommands
- added tests for `pie`

* `theme` added more tests

* - improved tests for `pattern` and `edit-flags`

* [ci] created cases for all arches for bin tests in `tests/heap.py`

* fixed `heap` tests for good

* - added ci test for `glibcarena`
- fixed `theme` missing comma (original PR #808 by @mrshu )
- added missing values for `theme`

* - added tests for deprecated API
- added test for smart eval
- more function tests

* started `gef` test module

* - added tests for `syscall-args` and `is-syscall`

* - fixed `syscall-args` to also get catchpoints + tests

* - test `show_last_exception`

* make sure `syscall-args` test collects the ABI files from `gef-extras`

* linting

* only enable `syscall-args` test for x86

* `syscall-args` fixed typo in i686 test

* Fix RISCV arch detection (#790)

* Add RISCV alias so arch can be determined by ELF
* Add ptrsize property to RISCV arch
* Allow riscv tests to run

* Update tests/api/gef_arch.py

Co-authored-by: Grazfather <grazfather@gmail.com>

* fix: make shebang lines portable (#814)

* fix: make shebang lines portable

* fix: SC2006, SC2086, SC2016, SC2059

* make `heap` tests work universally

* disabling capstone/keystone/unicorn for some arches for now

* - fixed tests for ppc64
- added static `ptrsize` for ppc & ppc64 in gef

* - `BIN_LS` -> `_target("default")`
- removed auto demangle for now

* - disable pytest `--pdb` from makefile
- added doc for testing

* Apply suggestions from code review

Co-authored-by: Grazfather <grazfather@gmail.com>

* fixing ci

* [tests] use camel case for format string helper test class

* [tests] added docstring to `GefFuncDeprecatedApi`

* [tests] `edit-flags` are only for known arches for now

* PR review changes

* PR review last batch

Co-authored-by: Grazfather <grazfather@gmail.com>
Co-authored-by: theguy147 <37738506+theguy147@users.noreply.github.com>

.github/PULL_REQUEST_TEMPLATE.md

@@ -27,5 +27,5 @@
 - [] My PR was done against the `dev` branch, not `master`.
 - [] My code follows the code style of this project.
 - [] My change includes a change to the documentation, if required.
-- [] My change adds tests as appropriate.
+- [] If my change adds new code, [adequate tests](docs/testing.md) have been added.
 - [] I have read and agree to the **CONTRIBUTING** document.

.gitignore

@@ -1,7 +1,7 @@
 *.out
 *.pyc
 TAGS
-*/__pycache__/
+__pycache__
 tests/*.pyc
 tests/pylint.html
 tests/pylint.txt

Makefile

@@ -11,21 +11,15 @@ TARGET := $(shell lscpu | head -1 | sed -e 's/Architecture:\s*//g')
 COVERAGE_DIR ?= /tmp/cov
 GEF_PATH ?= $(shell readlink -f gef.py)
 TMPDIR ?= /tmp
-PYTEST_PARAMETERS := --verbose -n $(NB_CORES)
-ifdef DEBUG
-	PYTEST_PARAMETERS += --pdb
-endif
+PYTEST_PARAMETERS := --verbose --forked --numprocesses=$(NB_CORES)
 
 .PHONY: test test_% Test% testbins clean lint
 
 test: $(TMPDIR) testbins
-	TMPDIR=$(TMPDIR) python3 -m pytest $(PYTEST_PARAMETERS) tests/runtests.py
-
-Test%: $(TMPDIR) testbins
-	TMPDIR=$(TMPDIR) python3 -m pytest $(PYTEST_PARAMETERS) tests/runtests.py::$@
+	TMPDIR=$(TMPDIR) python3 -m pytest $(PYTEST_PARAMETERS) -k "not benchmark"
 
 test_%: $(TMPDIR) testbins
-	TMPDIR=$(TMPDIR) python3 -m pytest $(PYTEST_PARAMETERS) tests/runtests.py -k $@
+	TMPDIR=$(TMPDIR) python3 -m pytest $(PYTEST_PARAMETERS) -k $@
 
 testbins: $(TMPDIR) $(wildcard tests/binaries/*.c)
 	@TMPDIR=$(TMPDIR) $(MAKE) -j $(NB_CORES) -C tests/binaries TARGET=$(TARGET) all
@@ -36,7 +30,7 @@ clean:
 
 lint:
 	python3 -m pylint $(PYLINT_GEF_PARAMETERS) $(GEF_PATH)
-	python3 -m pylint $(PYLINT_TEST_PARAMETERS) $(wildcard tests/*.py)
+	python3 -m pylint $(PYLINT_TEST_PARAMETERS) $(wildcard tests/*.py tests/*/*.py)
 
 coverage:
 	@! ( [ -d $(COVERAGE_DIR) ] && echo "COVERAGE_DIR=$(COVERAGE_DIR) exists already")

docs/testing.md

@@ -0,0 +1,122 @@
+## Testing GEF
+
+This page describes how GEF testing is done. Any new command/functionality must receive adequate testing to be merged. Also PR failing CI (test + linting) won't be merged either.
+
+
+### Prerequisites
+
+All the prerequisite packages are in `requirements.txt` file at the root of the project. So running
+
+```bash
+python -m pip install -r requirements.txt --user -U
+```
+
+is enough to get started.
+
+
+### Running tests
+
+
+#### Using `make tests`
+
+For testing GEF on the architecture on the host running the tests (most cases), simply run
+
+```bash
+cd /root/of/gef
+make test
+```
+
+Otherwise, `make` can be adjusted to :
+ - run one test/one group of tests specifically
+   - eg. `make test_cmd_heap` will trigger all the test functions for the `heap` command
+ - run for a different architecture
+   - eg. on x86, you can test x86 too with `TARGET=i686 make TARGET=i686 test`
+
+Unless a specific test is mentioned, the command `make test` will run all the non-benchmark related tests.
+At the end, a summary of explanation will be shown, clearly indicating the tests that have failed: for instance:
+
+```
+=================================== short test summary info ==================================
+FAILED tests/commands/heap.py::HeapCommand::test_cmd_heap_bins_large - AssertionError: 'siz...
+FAILED tests/commands/heap.py::HeapCommand::test_cmd_heap_bins_small - AssertionError: 'siz...
+FAILED tests/commands/heap.py::HeapCommand::test_cmd_heap_bins_unsorted - AssertionError: '...
+======================== 3 failed, 4 passed, 113 deselected in 385.77s (0:06:25)==============
+```
+
+You can then use `pytest` directly to help you fix each error specifically.
+
+
+#### Using `pytest`
+
+GEF entirely relies on [`pytest`](https://pytest.org) for its testing. Refer to the project documentation for details.
+
+Adding a new command __requires__ for extensive testing in a new dedicated test module that should be located in `/root/of/gef/tests/commands/my_new_command.py`
+
+A skeleton of a test module would look something like:
+
+```python
+"""
+`my-command` command test module
+"""
+
+
+from tests.utils import GefUnitTestGeneric, gdb_run_cmd, gdb_start_silent_cmd
+
+
+class MyCommandCommand(GefUnitTestGeneric):
+    """`my-command` command test module"""
+
+    def test_cmd_my_command(self):
+        # `my-command` is expected to fail if the session is not active
+        self.assertFailIfInactiveSession(gdb_run_cmd("my-command"))
+
+        # `my-command` should never throw an exception in GDB when running
+        res = gdb_start_silent_cmd("my-command")
+        self.assertNoException(res)
+
+        # it also must print out a "Hello World" message
+        self.assertIn("Hello World", res)
+```
+
+When running your test, you can summon `pytest` with the `--pdb` flag to enter the python testing environment to help you get more information about the reason of failure.
+
+### Linting GEF
+
+You can use the Makefile at the root of the project to get the proper linting settings. For most cases, the following command is enough:
+
+```bash
+cd /root/of/gef
+make lint
+```
+
+
+### Benchmarking GEF
+
+Benchmarking relies on `pytest-benchmark` and is experimental for now.
+
+You can run all benchmark test cases as such:
+
+```bash
+cd /root/of/gef
+pytest -k benchmark
+```
+
+which will return (after some time) an execution summary
+
+```
+tests/perf/benchmark.py ..                                                               [100%]
+
+
+---------------------------------------- benchmark: 3 tests -----------------------------------
+Name (time in ms)          Min                 Max                Mean            StdDev              Median                IQR            Outliers     OPS            Rounds  Iterations
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+time_baseline         612.2325 (1.0)      630.3416 (1.01)     623.7984 (1.01)     7.2848 (1.64)     626.1485 (1.01)      9.9971 (1.81)          1;0  1.6031 (0.99)          5           1
+time_cmd_context      613.8124 (1.00)     625.8964 (1.0)      620.1908 (1.0)      4.4532 (1.0)      619.8831 (1.0)       5.5109 (1.0)           2;0  1.6124 (1.0)           5           1
+time_elf_parsing      616.5053 (1.01)     638.6965 (1.02)     628.1588 (1.01)     8.2465 (1.85)     629.0099 (1.01)     10.7885 (1.96)          2;0  1.5920 (0.99)          5           1
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+
+Legend:
+  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
+  OPS: Operations Per Second, computed as 1 / Mean
+============================================== 3 passed, 117 deselected in 14.78s =============================================
+```