Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a to_junit method to standardize the report format of TestSuiteResult #1703

Conversation

Kranium2002
Copy link
Contributor

Description

This PR adds to_Junit function to TestSuitResult Class

Related Issue

This PR is related to issue number #1685

Type of Change

  • 📚 Examples / docs / tutorials / dependencies update
  • 🔧 Bug fix (non-breaking change which fixes an issue)
  • 🥂 Improvement (non-breaking change which improves an existing feature)
  • 🚀 New feature (non-breaking change which adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to change)
  • 🔐 Security fix

Checklist

  • I've read the CODE_OF_CONDUCT.md document.
  • I've read the CONTRIBUTING.md guide.
  • I've updated the code style using make codestyle.
  • I've written tests for all new methods and classes that I created.
  • I've written the docstring in Google format for all the methods and classes that I used.

@Kranium2002 Kranium2002 changed the title chore(add):to_junit function Add a to_junit method to standardize the report format of TestSuiteResult Dec 24, 2023
@Kranium2002
Copy link
Contributor Author

@luca-martial Would you please review my PR ?

Copy link
Member

@kevinmessiaen kevinmessiaen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR,

my concern is that we lose too much information about the tests when exporting and reading the report don't allow to understand the results.

In my opinion we should generate report like this:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites time="8.605871" tests="3">
    <testsuite name="Test accuracy (model=..., dataset=..., metric=0.5)" time="10.605871">
        <testcase name="metric_name" time="6.605871" />
    </testsuite>
    <testsuite name="Test F1 (model=..., dataset=..., metric=0.5)" time="2">
        <testcase name="metric_name" time="2">
            <failure message="Test failed with metric of 0.3" type="TestFailed">
                <!-- Full Test result information here (metric, output_ds, messages, ...) -->
            </failure>
        </testcase>
    </testsuite>
    <testsuite name="Test with error (model=..., dataset=..., metric=0.5)" time="2">
        <testcase name="metric_name" time="2">
            <failure message="Insert error message here" type="Error">
                <!-- stacktrace given in result.message) -->
            </failure>
        </testcase>
    </testsuite>
</testsuites>

giskard/core/suite.py Outdated Show resolved Hide resolved
@Kranium2002
Copy link
Contributor Author

@kevinmessiaen I still need help in implementing the runtime calculation logic. Would you please help me with that?

@kevinmessiaen
Copy link
Member

@kevinmessiaen I still need help in implementing the runtime calculation logic. Would you please help me with that?

Sure, let me look at it

@kevinmessiaen
Copy link
Member

Let's merge without the execution time of each tests at first. I created an issue to tackle the problem of adding execution time: #1759

@kevinmessiaen kevinmessiaen merged commit 4418e5b into Giskard-AI:main Jan 23, 2024
13 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants