Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] Add pytest and emulation metadata testing #71

Merged
merged 8 commits into from
Aug 31, 2023

Conversation

terrancedejesus
Copy link
Collaborator

@terrancedejesus terrancedejesus commented Aug 25, 2023

Overview

This PR adds pytest as a dependency to SWAT. It also adds test_emulation.py with several tests regarding emulation module metadata that is required. This emulation metadata is used to dynamically determine, scopes, services, ATT&CK mappings and more which affect other features.

Additional Information

  • The pyproject.toml incorrectly pointed to README.md, this has been fixed.
  • Added github workflow for pytest checks

Testing

  • pytest addition: run pytest, pytest tests/ or poetry run pytest tests/
  • emulations missing metadata: find an emulations and remove techniques, name, scopes, services or execute method and run tests

Copy link
Collaborator

@Mikaayenson Mikaayenson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple small questions; otherwise LGTM

@@ -9,6 +9,8 @@ Current (Unreleased)
--------------------

- Initial start of the changelog documentation. [`@terrancedejesus <https://github.com/terrancedejesus>`_]
- Added `pytest` to the project. [`@terrancedejesus <https://github.com/terrancedejesus>`_]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So every PR will have a change in the CHANGELOG? Might be good to unittest this.

tests/test_emulations.py Show resolved Hide resolved

emulations_list = get_all_emulation_classes.__func__()

@pytest.mark.parametrize("emulation", emulations_list, ids=lambda e: e.name)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: you could refactor all of these tests into a single test since you're using pytest and already using the parametrize decorator:

attributes_to_check = [
    ("parser", "has no parser defined"),
    ("services", "has no services defined"),
    ("scopes", "has no scopes defined"),
    ("techniques", "has no techniques defined"),
    ("execute", "has no execute method defined"),
]

@pytest.mark.parametrize("emulation", emulations_list, ids=lambda e: e.name)
@pytest.mark.parametrize("attribute,error_message", attributes_to_check)
def test_attributes_defined(emulation: Type[BaseEmulation], attribute: str, error_message: str):
    """Test if attributes/methods are defined in emulation."""
    assert hasattr(emulation, attribute), f'Emulation "{emulation.name}" {error_message}'

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++

@@ -9,6 +9,8 @@ Current (Unreleased)
--------------------

- Initial start of the changelog documentation. [`@terrancedejesus <https://github.com/terrancedejesus>`_]
- Added `pytest` to the project. [`@terrancedejesus <https://github.com/terrancedejesus>`_]
- Added `test_emulation.py` test file with several test cases for emulation metadata. [`@terrancedejesus <https://github.com/terrancedejesus>`_]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should bucket the changelog under version bumps. Which means

  • for changes requiring a version bump and targeting dev, they would bucket up
  • for changes directly to main, they would append the current version

terrancedejesus and others added 2 commits August 31, 2023 08:17
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
@terrancedejesus terrancedejesus merged commit a91f9b3 into main Aug 31, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants