Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI/CD jobs can succeed despite test failures. #2707

Closed
rectified95 opened this issue Jul 26, 2023 · 2 comments · Fixed by #2924
Closed

CI/CD jobs can succeed despite test failures. #2707

rectified95 opened this issue Jul 26, 2023 · 2 comments · Fixed by #2924
Assignees
Labels
bug Something isn't working ci/cd Issue is specific to CI/CD triaged Discussed in a triage meeting
Milestone

Comments

@rectified95
Copy link
Collaborator

rectified95 commented Jul 26, 2023

Describe the bug

  1. There are a few PRs for which the pipeline shows a passing run, even though their logs contain test failures.
    Ex. https://github.com/microsoft/ebpf-for-windows/actions/runs/5672733794/job/15373312646
bindmonitor_tailcall_native_test
-------------------------------------------------------------------------------
D:\a\ebpf-for-windows\ebpf-for-windows\tests\api_test\api_test.cpp([64](https://github.com/microsoft/ebpf-for-windows/actions/runs/5672733794/job/15373312646#step:19:65)4)
...............................................................................

D:\a\ebpf-for-windows\ebpf-for-windows\tests\api_test\api_test.cpp(624): FAILED:
  REQUIRE( perform_bind(&sockets[0], 30000) == 0 )
with expansion:
  -1 == 0

[12:54:44] :: api_test.exe Passed

I've also seen an example of this in other PRs.
Have we added any indirection test invocation and result gathering?

  1. Crash dumps are not always picked up by the respective job.
    From the same run as above:
[12:52:40] :: Executing api_test.exe
[12:52:40] :: C:\Program Files\OpenCppCoverage\OpenCppCoverage.exe -q --modules=C:\eBPF --export_type binary:api_test.exe.cov -- api_test.exe
[12:54:31] :: Generation of crash dump succeeded
Error: [12:54:41] :: [error] 

----------------------------------------------------
*** Your application has thrown an unhandled exception. Code: 32[21](https://github.com/microsoft/ebpf-for-windows/actions/runs/5672733794/job/15373312646#step:19:22)[22](https://github.com/microsoft/ebpf-for-windows/actions/runs/5672733794/job/15373312646#step:19:23)5477: EXCEPTION_ACCESS_VIOLATION
----------------------------------------------------
See https://github.com/OpenCppCoverage/OpenCppCoverage/wiki/FAQ#your-application-has-thrown-an-unhandled-exception-code-3221225477-exception_access_violation for additional information.

Error: [12:54:44] :: [error] Your program stop with error code: -2147483645

Then during the step 'Check for crash dumps':
These files don't exist: c:/dumps/x64/Debug/*.dmp

Steps taken to reproduce bug

run the CI

Expected behavior

CI runs never show up as successful when their components fail.

Actual outcome

CI runs sometimes show up as successful when their components fail.

@rectified95 rectified95 added the bug Something isn't working label Jul 26, 2023
@rectified95 rectified95 self-assigned this Jul 27, 2023
@dthaler dthaler added the ci/cd Issue is specific to CI/CD label Jul 31, 2023
@dahavey dahavey added this to the 2308 milestone Jul 31, 2023
@dthaler dthaler added the triaged Discussed in a triage meeting label Jul 31, 2023
@shankarseal shankarseal modified the milestones: 2308, 2309 Aug 31, 2023
@rectified95
Copy link
Collaborator Author

rectified95 commented Sep 13, 2023

We missed another regression because of this, will start looking today/tomorrow.
fyi @shankarseal
Failure introduced in https://github.com/microsoft/ebpf-for-windows/actions/runs/6152010592/job/16693947569?pr=2823

@rectified95
Copy link
Collaborator Author

Looks like the test EXE doesn't the exit code properly:
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working ci/cd Issue is specific to CI/CD triaged Discussed in a triage meeting
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants