Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail the test Pipeline if <N test results are reported #92

Closed
kmahone opened this issue Dec 14, 2018 · 0 comments · Fixed by #1032
Closed

Fail the test Pipeline if <N test results are reported #92

kmahone opened this issue Dec 14, 2018 · 0 comments · Fixed by #1032
Assignees
Labels
area-TestInfrastructure Issue in the test infrastructure (e.g. in Helix scripts) team-Controls Issue for the Controls team

Comments

@kmahone
Copy link
Member

kmahone commented Dec 14, 2018

We report test results from Helix into Azure Pipelines. If a test fails it should show up in the build test report and the build will show as failed.

However, if something goes wrong and something prevents the results from being reported, we might not notice that some results are missing from the build.

It would be nice if we have some extra step in the Pipeline that did something like "If less than 700 test results were reported then fail the build with an error". This would convert "absence of evidence" to "evidence of absence".

I don't think there is any existing feature that we can use for this, but we might be able to create something via the REST api: https://docs.microsoft.com/en-us/rest/api/azure/devops/test/results/list?view=azure-devops-rest-5.0

@jevansaks jevansaks added the area-TestInfrastructure Issue in the test infrastructure (e.g. in Helix scripts) label Dec 17, 2018
@jevansaks jevansaks added this to Approved in Controls Triage Feb 11, 2019
@kmahone kmahone self-assigned this Jul 10, 2019
Controls Triage automation moved this from Backlog to Closed Jul 11, 2019
@jevansaks jevansaks added the team-Controls Issue for the Controls team label Nov 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-TestInfrastructure Issue in the test infrastructure (e.g. in Helix scripts) team-Controls Issue for the Controls team
Projects
No open projects
2 participants