Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

connectors-ci: deprecate slash publish #25865

Merged
merged 21 commits into from
May 22, 2023

Conversation

alafanechere
Copy link
Contributor

@alafanechere alafanechere commented May 6, 2023

What

Closes #25432

As we move forward with "publish connectors on merge" we should deprecate the /publish command.

How

  • Make the publish-connectors.yml worfklow release non pre-prerelease images
  • Rename publish-command.yml to legacy-publish-command.yml so that /publish does not run the usual worfklow. /legacy-publish can be used using the old /publish is needed (e.g: normalization publication)
  • Make publish-command.yml post a deprecationg message so that /publish use leads to deprecation message comment.
  • Update docs to remove references to /publishj

User impact

⚠️ Merging this PR will actually enable the production use of "publish connectors on merge".

@octavia-squidington-iii octavia-squidington-iii added the area/documentation Improvements or additions to documentation label May 6, 2023
@alafanechere alafanechere marked this pull request as ready for review May 6, 2023 17:20
@alafanechere alafanechere requested a review from a team as a code owner May 6, 2023 17:20
@alafanechere alafanechere requested review from a team May 6, 2023 17:21
Copy link
Contributor

@bnchrch bnchrch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few small things! Otherwise looks good

@@ -31,7 +31,7 @@ Our SSH connector support is designed to be easy to plug into any existing conne
Replace port_key and host_key as necessary. Look at `transform_postgres()` to see an example.
2. To make sure your changes are present in Normalization when running tests on the connector locally, you'll need to change [this version tag](https://github.com/airbytehq/airbyte/blob/6d9ba022646441c7f298ca4dcaa3df59b9a19fbb/airbyte-workers/src/main/java/io/airbyte/workers/normalization/DefaultNormalizationRunner.java#L50) to `dev` so that the new locally built docker image for Normalization is used. Don't push this change with the PR though.
3. If your `host_key="host"` and `port_key="port"` then this step is not necessary. However if the key names differ for your connector, you will also need to add some logic into `sshtunneling.sh` (within airbyte-workers) to handle this, as currently it assumes that the keys are exactly `host` and `port`.
4. When making your PR, make sure that you've version bumped Normalization (in `airbyte-workers/src/main/java/io/airbyte/workers/normalization/DefaultNormalizationRunner.java` and `airbyte-integrations/bases/base-normalization/Dockerfile`). You'll need to /test & /publish Normalization _first_ so that when you /test the connector, it can use the new version.
4. When making your PR, make sure that you've version bumped Normalization (in `airbyte-workers/src/main/java/io/airbyte/workers/normalization/DefaultNormalizationRunner.java` and `airbyte-integrations/bases/base-normalization/Dockerfile`). You'll need to /test & /legacy-publish Normalization _first_ so that when you /test the connector, it can use the new version.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alafanechere why do we still tell them to use legacy?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Normalization publish is not yet implemented in our CI pipelines ...

* `/publish connector=connectors/source-sendgrid` - Publish the docker image if it doesn't exist for a single connector on the latest PR commit.

### 5. Automatically Run From `master`
### 4. Automatically Run From `master`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things.

  1. Should we not still mention /test
  2. Should me mention in the merge to master section that the connector will be published?

Copy link
Contributor

@sherifnada sherifnada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a "PR blurb" going out to devs to tell them what the new workflow is? would love to see that too

@@ -27,6 +27,7 @@ jobs:
build-connector
publish-connector
publish
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so is the /publish command still available after this PR? which workflow file does it use/how does it know to use it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This worfklow is responsible for writing the deprecation message: .github/workflows/publish-command.yml
Commands are mapped to workflow via the -command suffix in their filename.


Connector icons should be square SVGs and be located in [this directory](https://github.com/airbytehq/airbyte/tree/master/airbyte-config-oss/init-oss/src/main/resources/icons).

Connector documentation and changelogs are markdown files which live either [here for sources](https://github.com/airbytehq/airbyte/tree/master/docs/integrations/sources), or [here for destinations](https://github.com/airbytehq/airbyte/tree/master/docs/integrations/destinations).

### The /publish command
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since normalization people are still using this, could we keep this reference around somewhere? maybe just in the normalization README?

@alafanechere
Copy link
Contributor Author

alafanechere commented May 12, 2023

Is there a "PR blurb" going out to devs to tell them what the new workflow is? would love to see that too

Here

⚠️ The publish slash command is now deprecated.

The connector publication happens on merge to the master branch.

Please join the #publish-on-merge-updates slack channel to track ongoing publish pipelines.

Please reach out to the @dev-connector-ops team if you need support in publishing a connectors.

@alafanechere alafanechere reopened this May 22, 2023
@alafanechere alafanechere merged commit 80032f7 into master May 22, 2023
@alafanechere alafanechere deleted the augustin/connectors-ci/deprecate-slash-publish branch May 22, 2023 08:10
marcosmarxm pushed a commit to natalia-miinto/airbyte that referenced this pull request Jun 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/documentation Improvements or additions to documentation normalization
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Deprecate /publish command on PRs
4 participants