-
-
Notifications
You must be signed in to change notification settings - Fork 719
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Better tests #813
Merged
Merged
Better tests #813
Changes from 43 commits
Commits
Show all changes
45 commits
Select commit
Hold shift + click to select a range
cb85ef0
first shot at refactoring ci testing
ca51d43
- Added skeleton for most missing command test modules
f96f4ad
[ci] added tests for `nop`
4d1dc46
[ci] extra test for nop for memory check
c739c68
added benchmarking capability, can be triggered directly from `pytest`
cbe0a03
- fixed `pcustom` command test for 32b
27881bf
[ci] cmd/heap - adjusted tcachebins indexes for 32b
14be53e
damnit
dc66f9b
- fixed linting
7d77734
last fixes for tonight
fc31d63
fixed `pattern` and `heap` tests for good
a99a6cb
- add 3rd party module check for `capstone`, `keystone`, `unicorn` an…
41d44fd
added `test_func_update_gef`
10d1025
`make test` doesn't execute benchmark
9d655ae
- fixed errors in the `pie` subcommands
f870920
`theme` added more tests
072f5d7
- improved tests for `pattern` and `edit-flags`
2876bca
[ci] created cases for all arches for bin tests in `tests/heap.py`
de824b8
fixed `heap` tests for good
1a4c061
- added ci test for `glibcarena`
c475fd0
- added tests for deprecated API
22630c2
started `gef` test module
6a020da
- added tests for `syscall-args` and `is-syscall`
969a737
- fixed `syscall-args` to also get catchpoints + tests
757eb17
- test `show_last_exception`
013d1bd
make sure `syscall-args` test collects the ABI files from `gef-extras`
be68e87
linting
357f1d4
only enable `syscall-args` test for x86
9a528d8
`syscall-args` fixed typo in i686 test
70e8b03
Fix RISCV arch detection (#790)
Grazfather 18022a8
Update tests/api/gef_arch.py
hugsy 8e4befe
fix: make shebang lines portable (#814)
theguy147 1ee790c
Merge branch 'dev' of github.com:hugsy/gef into better_tests
c2c8d08
make `heap` tests work universally
7a69c7e
disabling capstone/keystone/unicorn for some arches for now
da028a3
- fixed tests for ppc64
1800673
- `BIN_LS` -> `_target("default")`
5eb1f62
- disable pytest `--pdb` from makefile
ec352de
Apply suggestions from code review
hugsy aa8d051
fixing ci
02e9048
[tests] use camel case for format string helper test class
ae03c99
[tests] added docstring to `GefFuncDeprecatedApi`
af339c5
[tests] `edit-flags` are only for known arches for now
d2d85ff
PR review changes
hugsy bc44c2d
PR review last batch
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,7 +1,7 @@ | ||
*.out | ||
*.pyc | ||
TAGS | ||
*/__pycache__/ | ||
__pycache__ | ||
tests/*.pyc | ||
tests/pylint.html | ||
tests/pylint.txt | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,122 @@ | ||
## Testing GEF | ||
|
||
This page describes how GEF testing is done. Any new command/functionality must receive adequate testing to be merged. Also PR failing CI (test + linting) won't be merged either. | ||
|
||
|
||
### Prerequisites | ||
|
||
All the prerequisite packages are in `requirements.txt` file at the root of the project. So running | ||
|
||
```bash | ||
python -m pip install -r requirements.txt --user -U | ||
``` | ||
|
||
is enough to get started. | ||
|
||
|
||
### Running tests | ||
|
||
|
||
#### Using `make tests` | ||
|
||
For testing GEF on the architecture on the host running the tests (most cases), simply run | ||
|
||
```bash | ||
cd /root/of/gef | ||
make test | ||
``` | ||
|
||
Otherwise, `make` can be adjusted to : | ||
- run one test/one group of tests specifically | ||
- eg. `make test_cmd_heap` will trigger all the test functions for the `heap` command | ||
- run for a different architecture | ||
- eg. on x86, you can test x86 too with `TARGET=i686 make TARGET=i686 test` | ||
|
||
Unless a specific test is mentioned, the command `make test` will run all the non-benchmark related tests. | ||
At the end, a summary of explanation will be shown, clearly indicating the tests that have failed: for instance: | ||
|
||
``` | ||
=================================== short test summary info ================================== | ||
FAILED tests/commands/heap.py::HeapCommand::test_cmd_heap_bins_large - AssertionError: 'siz... | ||
FAILED tests/commands/heap.py::HeapCommand::test_cmd_heap_bins_small - AssertionError: 'siz... | ||
FAILED tests/commands/heap.py::HeapCommand::test_cmd_heap_bins_unsorted - AssertionError: '... | ||
======================== 3 failed, 4 passed, 113 deselected in 385.77s (0:06:25)============== | ||
``` | ||
|
||
You can then use `pytest` directly to help you fix each error specifically. | ||
|
||
|
||
#### Using `pytest` | ||
|
||
GEF entirely relies on [`pytest`](https://pytest.org) for its testing. Refer to the project documentation for details. | ||
|
||
Adding a new command __requires__ for extensive testing in a new dedicated test module that should be located in `/root/of/gef/tests/commands/my_new_command.py` | ||
|
||
A skeleton of a test module would look something like: | ||
|
||
```python | ||
""" | ||
`my-command` command test module | ||
""" | ||
|
||
|
||
from tests.utils import GefUnitTestGeneric, gdb_run_cmd, gdb_start_silent_cmd | ||
|
||
|
||
class MyCommandCommand(GefUnitTestGeneric): | ||
"""`my-command` command test module""" | ||
|
||
def test_cmd_my_command(self): | ||
# `my-command` is expected to fail if the session is not active | ||
self.assertFailIfInactiveSession(gdb_run_cmd("my-command")) | ||
|
||
# `my-command` should never throw an exception in GDB when running | ||
res = gdb_start_silent_cmd("my-command") | ||
self.assertNoException(res) | ||
|
||
# it also must print out a "Hello World" message | ||
self.assertIn("Hello World", res) | ||
``` | ||
|
||
When running your test, you can summon `pytest` with the `--pdb` flag to enter the python testing environment to help you get more information about the reason of failure. | ||
|
||
### Linting GEF | ||
|
||
You can use the Makefile at the root of the project to get the proper linting settings. For most cases, the following command is enough: | ||
|
||
```bash | ||
cd /root/of/gef | ||
make lint | ||
``` | ||
|
||
|
||
### Benchmarking GEF | ||
|
||
Benchmarking relies on `pytest-benchmark` and is experimental for now. | ||
|
||
You can run all benchmark test cases as such: | ||
|
||
```bash | ||
cd /root/of/gef | ||
pytest -k benchmark | ||
``` | ||
|
||
which will return (after some time) an execution summary | ||
|
||
``` | ||
tests/perf/benchmark.py .. [100%] | ||
|
||
|
||
---------------------------------------- benchmark: 3 tests ----------------------------------- | ||
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations | ||
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ||
time_baseline 612.2325 (1.0) 630.3416 (1.01) 623.7984 (1.01) 7.2848 (1.64) 626.1485 (1.01) 9.9971 (1.81) 1;0 1.6031 (0.99) 5 1 | ||
time_cmd_context 613.8124 (1.00) 625.8964 (1.0) 620.1908 (1.0) 4.4532 (1.0) 619.8831 (1.0) 5.5109 (1.0) 2;0 1.6124 (1.0) 5 1 | ||
time_elf_parsing 616.5053 (1.01) 638.6965 (1.02) 628.1588 (1.01) 8.2465 (1.85) 629.0099 (1.01) 10.7885 (1.96) 2;0 1.5920 (0.99) 5 1 | ||
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ||
|
||
Legend: | ||
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile. | ||
OPS: Operations Per Second, computed as 1 / Mean | ||
============================================== 3 passed, 117 deselected in 14.78s ============================================= | ||
``` |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"have to be" or "have been"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good call.