Enhance /publish to allow for multiple connectors and parallel execution

[Phlair]

What

This is a semi-overhaul to the publish command so that you can now provide multiple comma separated connectors rather than only publish one at a time. It is 100% backwards compatible and passing in a single connector will work just as before.

It uses Github Actions strategy matrix to run the publish job in parallel for each of the passed in connectors.

This is meant as a short-term solution to improve the experience of publishing multiple connectors. It could certainly be better but we'll be addressing publishing more holistically soon as part of this.

Example of new output

don't worry... just dummy success/fails in this screenshot:

Example of action flow

Notes

The reason this doesn't spin up a dynamic pool of agents (it spins up a static 5 when parallel is switched on) is due to limitations with github strategy matrix and outputs (context here and here)

TODO

• docs!!!
• Make the output more readable
  • the usual 'print back to comment' gets messy with multiple connectors in parallel. Even looks like some comments get missed (probably a timing race condition)
• Make a change here so that we can add a custom group label to the runners. This will allow us to point at our pool of runners for parallel execution.
  • spoke to Topher, we decided to make a new fork of the repo under airbytehq org for us to use
• make the bash monstrosity more readable (jq?, separate script?)
  • it's not the perfect solution but I added comments explaining what it's doing
• account for any funkiness from running publish for a 2nd/3rd/etc. connector on the same agent that already published a connector.
  • the sentry cli being already installed was an easy enough fix
  • made sure to attempt an rm of the python venv so we're starting fresh

Nice to have todos

• detail the exact flow of what's going on (probably somewhere else, maybe a loom)
• make it so you don't have to write connectors/*
• make it so you don't even need to specify the connectors and it figures them out by the dockerfiles with diffs in the PR

[Phlair]

testing out below by manually triggering from actions tab and pointing at my manually created comments. (I needed to publish these java connectors anyway)

[Phlair]

mega publish test 1

🕑 connectors/destination-azure-blob-storage, connectors/destination-bigquery, connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2512057019
❌ Failed to publish connectors/destination-azure-blob-storage, connectors/destination-bigquery, connectors/destination-bigquery-denormalized
❌ Couldn't auto-bump version for connectors/destination-azure-blob-storage, connectors/destination-bigquery, connectors/destination-bigquery-denormalized

[Phlair]

mega publish test 2

🕑 connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/2512091768
🚀 Successfully published connectors/destination-azure-blob-storage
🚀 Auto-bumped version for connectors/destination-azure-blob-storage
✅ connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/2512091768
🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2512091768
❌ Failed to publish connectors/destination-bigquery
❌ Couldn't auto-bump version for connectors/destination-bigquery

[Phlair]

final test (expecting S3 & github to fail as I'm not actually bumping them)

🕑 Publishing the following connectors:

Running tests before publishing: true
https://github.com/airbytehq/airbyte/actions/runs/2535769188

Connector	Published	Definitions generated
connectors/source-s3	❌	❌
connectors/source-github	❌	❌
connectors/source-openweather	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@e8a5c7f). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #13864   +/-   ##
=========================================
  Coverage          ?   93.67%           
=========================================
  Files             ?        4           
  Lines             ?       79           
  Branches          ?        0           
=========================================
  Hits              ?       74           
  Misses            ?        5           
  Partials          ?        0           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e8a5c7f...cf25915. Read the comment docs.

[Phlair]

failing test is just for Sonar Scan / Tests for connectors/destination-mongodb (pull_request) which is safe to ignore here

[sherifnada]

nice!

[sherifnada]

is this used within EC2 somewhere? what is the functional significance for adding this label property?

[Phlair]

It adds the label to the runner as seen by github (UI here).

Labels we had before:

• [self-hosted, Linux, X64] on every runner
• a random unique label to id a runner

This adds a new label that is unique to the github action run, meaning if we spin up 5 runners in the same action run, we now have a label that is shared across them all, giving us a pool of runners that we can use in the runs-on: section of a job (if we want, but it's not mandated, you can still use specific unique id)