Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: use pytest for qemu-system tests #1679

Merged
merged 3 commits into from
Apr 22, 2023
Merged

Conversation

theguy147
Copy link
Contributor

@theguy147 theguy147 commented Apr 21, 2023

This PR adds the feature to use pytest (including its coverage capabilities) to the qemu-system tests. I took the scripts for the userland tests as a template and modified them until things worked as I wanted to.

I also added a few, not very meaningful but exemplary, tests for the kernel commands kbase, kchecksec, kcmdline, kconfig, kversion, slab list and slab info.

I have not yet taken a look at the necessary changes to the github workflows yet so I open this PR as a Draft for now.

Screenshot from 2023-04-21 21-41-29

EDIT: now I also implemented the changes for the github workflow.

The failing test tests/test_loads.py::test_loads_binary_with_core_without_crashing on the archlinux docker is unrelated to this PR.

@codecov-commenter
Copy link

codecov-commenter commented Apr 21, 2023

Codecov Report

Merging #1679 (683d440) into dev (e6037f5) will increase coverage by 0.67%.
The diff coverage is n/a.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@            Coverage Diff             @@
##              dev    #1679      +/-   ##
==========================================
+ Coverage   59.12%   59.80%   +0.67%     
==========================================
  Files         175      175              
  Lines       21057    21057              
  Branches     1866     1866              
==========================================
+ Hits        12451    12594     +143     
+ Misses       8003     7848     -155     
- Partials      603      615      +12     

see 17 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@theguy147
Copy link
Contributor Author

theguy147 commented Apr 21, 2023

lol, the PR decreases the coverage by 56.41%? I guess this is only the coverage for the kernel commands now.

Either I messed something up, or CodeCovs promise does not hold :D

No setup required. Automatic report merging for all CI and languages into a single report.

EDIT: ah, the report was now corrected automatically...

@theguy147 theguy147 marked this pull request as ready for review April 21, 2023 20:27
@theguy147
Copy link
Contributor Author

I just noticed that it might also be useful to only execute tests for certain architectures or kernel types later on, so I passed them through to the test via env vars.

Conditional tests could be executed e.g. like this:

import os
import pytest

ARCH = os.getenv("PWNDBG_ARCH")
KERNEL_TYPE = os.getenv("PWNDBG_KERNEL_TYPE")

@pytest.skipif(ARCH not in ["x86_64"], reason="")
def test_only_on_x86_64():
    raise Exception("This test is only executed on x86_64")


def pytest_collection_modifyitems(self, items):
for item in items:
self.collected.append(item.nodeid)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or: self.collected.extend(item.nodeid for item in items)

@disconnect3d
Copy link
Member

This is awesome work. Thanks!

@disconnect3d disconnect3d merged commit 91c72a0 into pwndbg:dev Apr 22, 2023
@gsingh93 gsingh93 mentioned this pull request Apr 23, 2023
5 tasks
@theguy147 theguy147 deleted the pytest_qemu branch May 20, 2023 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants