New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Intermittent test failures involving TestStartosisAddServiceTestSuite #1538
Comments
One more here: https://app.circleci.com/pipelines/github/kurtosis-tech/kurtosis/7874/workflows/63979657-73a6-4919-b27b-b6fc9d0814b2/jobs/107163 This time not add service, but
also looks random, and possibly something being torn down prematurely |
looks like a plain bug: our code being wrong, nothing to do with CI. I really, really, really think we should prioritize digging out what the cause is and fix these... flaky tests erode the trust in the CI across the entire team, and with merge queues flaky tests are only going exacerbate the problem. Our CI test suite didn't used to be flaky, and in the past ~6-9 months, it's gotten increasingly so. |
Cool found the issue in the tests themselves (excuse my go learning adventure):
The error at
Then after that adding a That will only show us the real errors, and it only affects 1 of the 3 cases above, from what I can see. There's still something weird in the test infra / CI (long running tests, maybe things timing out, etc) #1559 contains the fix for this one, plus similar tests |
One more random failure (occurred locally with
|
In relation to the panic error https://app.circleci.com/pipelines/github/kurtosis-tech/kurtosis/7874/workflows/63979657-73a6-4919-b27b-b6fc9d0814b2/jobs/107163 , I've just noticed you can see in the
which happens exactly when the test itself fails:
So I'm pretty certain the true cause is something funky in the test infra / timing / stuff closing or timing out / etc |
## Description: When an unexpected error happens in `enclave_context.RunStarlarkPackageBlocking`, it will return `(nil, error)`. Some tests don't account for this, so they don't detect or report said error, then panic when trying to reference the `nil` result. This adds `require.NoError(t, err)` to all the tests I could find that hit this codepath. ## Is this change user facing? NO ## References (if applicable): #1538 --------- Co-authored-by: Omar <omar@omar>
@omar711 Are you still investigating/working on that? |
Hi @laurentluce , not actively. I dig in when I see something I can debug, then leave it be. e.g. #1559 fixed a bug that might hide underlying failures. I don't think I've got near to the root of it though. |
What's your CLI version?
0.84.3
Description & steps to reproduce
I've seen the
TestStartosisAddServiceTestSuite
test fail twice in CI checks with seemingly unrelated PRs. The failures can be fixed by re-running, where they usually then pass.Examples: https://app.circleci.com/pipelines/github/kurtosis-tech/kurtosis/7789/workflows/26e1b319-a461-455a-b790-a27fe7bcd6ae/jobs/105839 and https://app.circleci.com/pipelines/github/kurtosis-tech/kurtosis/7775/workflows/b40031d3-47fd-43b9-8265-94fe5d59ea5e/jobs/105604
Interesting notes:
Which suggests there's something awkward in this test that makes it a little brittle.
Desired behavior
Tests + CI run reliably
What is the severity of this bug?
Papercut; this bug is frustrating, but I have a workaround.
What area of the product does this pertain to?
Other: anything not covered by the above
The text was updated successfully, but these errors were encountered: