Add test coverage validation step to CI workflow(AST-22222)#1424
Open
cx-amol-mane wants to merge 9 commits intomainfrom
Open
Add test coverage validation step to CI workflow(AST-22222)#1424cx-amol-mane wants to merge 9 commits intomainfrom
cx-amol-mane wants to merge 9 commits intomainfrom
Conversation
This commit introduces a new job in the CI workflow to validate that all integration tests are covered by the defined CI matrix patterns. The validation checks for uncovered tests and provides feedback on the total number of tests found and those covered by patterns. This enhancement aims to ensure comprehensive test coverage and improve the reliability of the CI process.
cx-amol-mane
requested review from
cx-anjali-deore,
cx-anurag-dalke and
cx-umesh-waghode
as code owners
cx-amol-mane
changed the title
This commit modifies the CI workflow configuration by correcting the pattern for scan operations to include "ScanWorkFlow" and updating the SCM tests pattern to account for "BitBucket". These changes ensure that the CI process accurately reflects the intended test coverage and improves the clarity of the test categorization.
|
Great job! No new security vulnerabilities introduced in this pull request |
This commit introduces a new pre-test cleanup step to delete stale test projects before running integration tests. Additionally, it improves the test rerun logic by capturing exit codes and handling various failure patterns, ensuring that tests are retried correctly if they fail or panic. These changes aim to enhance the reliability and cleanliness of the CI testing process.
…eouts This commit improves the CI workflow by adding checks for infrastructure-related failures that should not trigger test reruns. It also updates the logic to provide clearer messages regarding test timeouts, ensuring better diagnostics during the CI process. The changes aim to enhance the reliability and clarity of test execution outcomes.
This commit improves the CI workflow by adding a validation step to identify uncovered integration tests and dynamically generate a catch-all group for those tests. It updates the test execution logic to conditionally run uncovered tests and provides warnings when tests are not covered by existing CI patterns. These changes aim to enhance test coverage management and improve the clarity of test execution outcomes.
This commit introduces a new test function, AmolTestAuthValidate, to the integration tests for authentication validation. The function executes the "auth validate" command and asserts successful authentication, enhancing the coverage of authentication-related tests.
…cy in integration tests
This commit significantly improves the CI workflow by implementing a robust retry mechanism for failed tests. It introduces multiple methods to identify failed tests, including standard failure patterns, panic detection, and API/auth errors. The logic now deduplicates and formats failed test names for reruns, ensuring a more reliable and informative testing process. These enhancements aim to increase the resilience of the CI pipeline and provide clearer diagnostics for test failures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit introduces a new job in the CI workflow to validate that all integration tests are covered by the defined CI matrix patterns. The validation checks for uncovered tests and provides feedback on the total number of tests found and those covered by patterns. This enhancement aims to ensure comprehensive test coverage and improve the reliability of the CI process.
By submitting this pull request, you agree to the terms within the Checkmarx Code of Conduct. Please review the contributing guidelines for guidance on creating high-quality pull requests.
Description
Please provide a summary of the changes and the related issue. Include relevant motivation and context.
Type of Change
Related Issues
Link any related issues or tickets.
Checklist
Screenshots (if applicable)
Add screenshots to help explain your changes.
Additional Notes
Add any other relevant information.