fix(tests): flaky integration tests in release pipeline #5485
Conversation
If multiple similar pipelines (patch pipelines, test pipelines) are running the same CodeBuild test, they interfere with each other. In particular, the 'test' pipeline (which tests modifications to the pipeline before they are applied to the real pipeline) is triggered off of the same branch and takes about as long as the real pipeline, so tests are likely to execute at the same time and touch the same stacks, thereby causing one of the tests to fail (and our release pipeline to be flaky). They two pipelines will have different CodeBuild projects though, so name the stacks that are being touched after the CodeBuild project, thereby ensuring isolation.
rix0rrr
requested review from
RomainMuller,
skinny85,
MrArnoldPalmer and
nija-at
| CODEBUILD_PROJECT=$(echo ${CODEBUILD_BUILD_ID:-} | cut -d: -f 1) | ||
|
|
||
| if [[ "${CODEBUILD_PROJECT:-}" != "" ]]; then | ||
| export STACK_NAME_PREFIX="${CODEBUILD_PROJECT}" |
There was a problem hiding this comment.
Could be worth actually having the actual execution ID in there, too, so that even two concurrent executions of the same project don't interfere with each other? I'm not willing to die on that hill though!
There was a problem hiding this comment.
If we wanted that we could just generate a UUID4 at the start of the run. The downside of that is that we might accumulate garbage, if deploying succeeds but then something fails to clean up the stacks. After a while, our tests will fail because our account has filled up with stale resources, and will need maintenance.
If we do this, we get automatic overwriting of old stacks while still ensuring enough isolation where it matters.
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
|
Thank you for contributing! Your pull request is now being automatically merged. |
|
Thank you for contributing! Your pull request is now being automatically merged. |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
If multiple similar pipelines (patch pipelines, test pipelines) are
running the same CodeBuild test, they interfere with each other.
In particular, the 'test' pipeline (which tests modifications to the
pipeline before they are applied to the real pipeline) is triggered
off of the same branch and takes about as long as the real pipeline,
so tests are likely to execute at the same time and touch the same
stacks, thereby causing one of the tests to fail (and our release
pipeline to be flaky).
They two pipelines will have different CodeBuild projects though,
so name the stacks that are being touched after the CodeBuild project,
thereby ensuring isolation.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license