-
-
Notifications
You must be signed in to change notification settings - Fork 230
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Need a to_junit method to standardize the report format of TestSuiteResult #1685
Comments
Hi, I would like to work on this issue. Could you guide me a bit? |
Hi @Kranium2002 sounds great, thanks for being open to contributing! The TestSuiteResult class needs a new method that we will name JUnit XML format would look something like this for test suite results:
Let us know if you have any questions! |
I'll do this by this weekend, could you maybe assign this issue to me? |
If I may I did a quick script that can improve compatibility with from giskard import demo, Model, Dataset, testing, Suite
import pytest
model, df = demo.titanic()
wrapped_dataset = Dataset(
df=df,
target="Survived",
cat_columns=["Pclass", "Sex", "SibSp", "Parch", "Embarked"],
)
suite = (
Suite()
.add_test(testing.test_f1(dataset=wrapped_dataset, threshold=.6))
.add_test(testing.test_accuracy(dataset=wrapped_dataset))
)
my_first_model = Model(model=model, model_type="classification")
suite_results = suite.run(model=my_first_model)
@pytest.mark.parametrize("test_result", suite_results.results, ids = lambda t: t[0])
def test_giskard(test_result):
name_, result_, data_ = test_result
assert result_.passed, result_.messages which can be called via: python -W ignore -m pytest test_ml.py and here is the output:
This is not bad but if we improve the TestResult class we can have richer From this using python -W ignore -m pytest tests.py --junitxml=junit-example.xml would create the report with "base" library (see attached file) |
looking at the code for the tests ideally refactoring the function so that they implement an |
I think that keeping test return In a meanwhile it's fairly easy to add an assert method in class GiskardTest(Artifact[TestFunctionMeta], ABC):
# Omitting existing methods
def assert(self):
result = self.execute()
if type(result) == bool:
assert result
else:
assert result.passed, '\n'.join([repr(message) for message in result.messages]) Then you'll have this in your tests: def test_model_accuracy(model, dataset):
from giskard.testing import test_accuracy
test_accuracy(model, dataset, threshold=0.75).assert() |
this a great suggestion!! Let me try this locally. I think this should easily address the design compatibility with |
Started working on this in #1703 |
Discussed in https://github.com/orgs/Giskard-AI/discussions/1681
To standardize the reporting from the test suite, we could add a
to_junit
method to ourTestSuiteResult
Originally posted by AdriMarteau December 15, 2023
I am looking at integrating giskard in a CI process and I was wondering if it was compatible with pytest.
This would help with leveraging CI tools reporting capabilities.
My idea was to use a command like:
The text was updated successfully, but these errors were encountered: