Speed Up Publish OSS Artifacts for Cloud Workflow #19696
Conversation
when using inputs for concurrency expression, needs to be on job level
docker buildx will actually build the images
git-phu
changed the title
|
@davinchia looks when switching from |
| @@ -144,3 +157,29 @@ jobs: | |||
| - name: Cleanup Docker buildx | |||
| run: docker buildx rm oss-buildx | |||
| shell: bash | |||
|
|
|||
| stop-runner: | |||
There was a problem hiding this comment.
oh did we previously not stop runners?
There was a problem hiding this comment.
yup, seems like a bug that went unnoticed
| @@ -10,6 +12,9 @@ on: | |||
| required: false | |||
| jobs: | |||
| find_valid_pat: | |||
| concurrency: | |||
There was a problem hiding this comment.
for my understanding: what is this for?
There was a problem hiding this comment.
this is so only one instance of this workflow can be actively running for a specific oss sha (so we don't attempt to push artifacts for the same sha from multiple concurrent workflows)
There was a problem hiding this comment.
ah ok! can we also add a comment here?
- for both publish jobs we only want to allow one running workflow per SHA. - for stopping the runner, use the find a PAT step (job was removed)
| group: ${{ github.workflow }}-${{ inputs.oss_ref || github.sha }} | ||
|
|
||
| env: | ||
| S3_BUILD_CACHE_ACCESS_KEY_ID: ${{ secrets.SELF_RUNNER_AWS_ACCESS_KEY_ID }} |
There was a problem hiding this comment.
Maybe a comment is nice here that says something along the lines of
//these environment variables are necessary for gradle cache task
|
forgot to merge this in after the code freeze but I'll do so now One more verification that it's still working: https://github.com/airbytehq/airbyte/actions/runs/3699244652 |
This reverts commit 863d8da.
This reverts commit 863d8da.
What
Attempt to speed up Publish OSS Artifacts for Cloud Workflow
How
gradle testinstead ofgradle buildto skip unnecessary steps before publishing jarsbuildDockerImagetasks (since this will cause the workflow to build the same images twice).docker buildx, just runcopyGeneratedTarand only havebuildxactually build images.Others:
Tests
Successful run: https://github.com/airbytehq/airbyte/actions/runs/3520087241
gw testbuild scan: https://scans.gradle.com/s/vfs6gdoaq3ue4/timelineActual Speedup:
Not very definitive since this workflow seems to have high variance in run time (can take ~18 minutes or ~28 minutes, not sure what affects this) but it seems like going from
gradle build->gradle testsaves us about ~3 minutes.Also copying this from a comment below:
We could also just try merging this in and seeing what run times look like after this change