Improved test_commands.py

[gsingh93]

test_commands.py is a smoke test that runs almost every command to make sure no exceptions are thrown. This test was skipped because it was slow and flaky. These PR makes a few changes to this:

@pytest.mark.parametrize("name", filtered_commands)
@pytest.mark.skipif(condition=should_skip_test, reason="flaky test")
@pytest.mark.xfail(reason="flaky test")

First, instead of running every command in a loop, we run each command in its own test. This helps because the tests can then be parallelized, and if a test fails you only see that commands output and not every commands output.

Then, instead of skipping the test unconditionally, we only skip it if we detect we're not running on CI and if the RUN_FLAKY environment variable is not set. This means it won't slow down your test workflow unless you really want to run it with RUN_FLAKY=1 ./tests.sh.

Finally, the test is marked as xfail. If any of the parameterized tests fail, their result will be XFAIL instead of FAILED. If any of those tests pass, they'll be listed as XPASS instead of PASSED. None of these results will affect the results of the CI. The reason this is done is so we can still use the test locally and on CI without this flaky test making us think we have real errors. As we improve this test in the future, we can remove this and require the CI pass this test as well.

[codecov-commenter]

Codecov Report

Merging #1350 (9d579fb) into dev (ac5a6eb) will increase coverage by 2.89%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##              dev    #1350      +/-   ##
==========================================
+ Coverage   54.89%   57.78%   +2.89%     
==========================================
  Files         153      153              
  Lines       19392    19392              
  Branches     1816     1816              
==========================================
+ Hits        10645    11206     +561     
+ Misses       8298     7644     -654     
- Partials      449      542      +93     

Impacted Files	Coverage Δ
pwndbg/commands/context.py	66.28% <0.00%> (+0.22%)	⬆️
pwndbg/gdblib/regs.py	71.89% <0.00%> (+0.65%)	⬆️
pwndbg/commands/procinfo.py	81.96% <0.00%> (+0.81%)	⬆️
pwndbg/disasm/__init__.py	60.95% <0.00%> (+1.36%)	⬆️
pwndbg/gdblib/tls.py	36.36% <0.00%> (+1.51%)	⬆️
pwndbg/commands/nearpc.py	90.32% <0.00%> (+1.61%)	⬆️
pwndbg/gdblib/memory.py	69.49% <0.00%> (+1.69%)	⬆️
pwndbg/commands/next.py	85.18% <0.00%> (+1.85%)	⬆️
pwndbg/gdblib/hooks.py	100.00% <0.00%> (+2.00%)	⬆️
pwndbg/lib/config.py	93.10% <0.00%> (+2.29%)	⬆️
... and 50 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.