Skip to content

Commit

Permalink
airbyte-ci: Switch to prod pypi (#34606)
Browse files Browse the repository at this point in the history
  • Loading branch information
Joe Reuter authored Jan 30, 2024
1 parent b37bde8 commit d1334be
Show file tree
Hide file tree
Showing 12 changed files with 95 additions and 40 deletions.
5 changes: 5 additions & 0 deletions .github/actions/run-airbyte-ci/action.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,10 @@ inputs:
python_registry_token:
description: "Python registry API token to publish python package"
required: false
python_registry_url:
description: "Python registry URL to publish python package"
default: "https://upload.pypi.org/legacy/"
required: false

runs:
using: "composite"
Expand Down Expand Up @@ -135,6 +139,7 @@ runs:
PRODUCTION: ${{ inputs.production }}
PULL_REQUEST_NUMBER: ${{ github.event.pull_request.number }}
PYTHON_REGISTRY_TOKEN: ${{ inputs.python_registry_token }}
PYTHON_REGISTRY_URL: ${{ inputs.python_registry_url }}
S3_BUILD_CACHE_ACCESS_KEY_ID: ${{ inputs.s3_build_cache_access_key_id }}
S3_BUILD_CACHE_SECRET_KEY: ${{ inputs.s3_build_cache_secret_key }}
SENTRY_DSN: ${{ inputs.sentry_dsn }}
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/publish-airbyte-lib-command-manually.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ jobs:
uses: actions/checkout@v3
- name: Publish
id: publish-airbyte-lib
uses: ./.github/actions/run-dagger-pipeline
uses: ./.github/actions/run-airbyte-ci
with:
context: "manual"
dagger_cloud_token: ${{ secrets.DAGGER_CLOUD_TOKEN }}
Expand All @@ -53,5 +53,5 @@ jobs:
s3_build_cache_access_key_id: ${{ secrets.SELF_RUNNER_AWS_ACCESS_KEY_ID }}
s3_build_cache_secret_key: ${{ secrets.SELF_RUNNER_AWS_SECRET_ACCESS_KEY }}
tailscale_auth_key: ${{ secrets.TAILSCALE_AUTH_KEY }}
subcommand: 'poetry --package-path=airbyte-lib publish --registry-url="https://test.pypi.org/legacy/"'
subcommand: "poetry --package-path=airbyte-lib publish"
python_registry_token: ${{ secrets.PYPI_TOKEN }}
47 changes: 30 additions & 17 deletions airbyte-ci/connectors/pipelines/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -385,16 +385,18 @@ Publish all connectors modified in the head commit: `airbyte-ci connectors --mod

### Options

| Option | Required | Default | Mapped environment variable | Description |
| ------------------------------------ | -------- | --------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--pre-release/--main-release` | False | `--pre-release` | | Whether to publish the pre-release or the main release version of a connector. Defaults to pre-release. For main release you have to set the credentials to interact with the GCS bucket. |
| `--spec-cache-gcs-credentials` | False | | `SPEC_CACHE_GCS_CREDENTIALS` | The service account key to upload files to the GCS bucket hosting spec cache. |
| `--spec-cache-bucket-name` | False | | `SPEC_CACHE_BUCKET_NAME` | The name of the GCS bucket where specs will be cached. |
| `--metadata-service-gcs-credentials` | False | | `METADATA_SERVICE_GCS_CREDENTIALS` | The service account key to upload files to the GCS bucket hosting the metadata files. |
| `--metadata-service-bucket-name` | False | | `METADATA_SERVICE_BUCKET_NAME` | The name of the GCS bucket where metadata files will be uploaded. |
| `--slack-webhook` | False | | `SLACK_WEBHOOK` | The Slack webhook URL to send notifications to. |
| `--slack-channel` | False | | `SLACK_CHANNEL` | The Slack channel name to send notifications to. |
| `--ci-requirements` | False | | | Output the CI requirements as a JSON payload. It is used to determine the CI runner to use. |
| Option | Required | Default | Mapped environment variable | Description |
| ------------------------------------ | -------- | ------------------------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--pre-release/--main-release` | False | `--pre-release` | | Whether to publish the pre-release or the main release version of a connector. Defaults to pre-release. For main release you have to set the credentials to interact with the GCS bucket. |
| `--spec-cache-gcs-credentials` | False | | `SPEC_CACHE_GCS_CREDENTIALS` | The service account key to upload files to the GCS bucket hosting spec cache. |
| `--spec-cache-bucket-name` | False | | `SPEC_CACHE_BUCKET_NAME` | The name of the GCS bucket where specs will be cached. |
| `--metadata-service-gcs-credentials` | False | | `METADATA_SERVICE_GCS_CREDENTIALS` | The service account key to upload files to the GCS bucket hosting the metadata files. |
| `--metadata-service-bucket-name` | False | | `METADATA_SERVICE_BUCKET_NAME` | The name of the GCS bucket where metadata files will be uploaded. |
| `--slack-webhook` | False | | `SLACK_WEBHOOK` | The Slack webhook URL to send notifications to. |
| `--slack-channel` | False | | `SLACK_CHANNEL` | The Slack channel name to send notifications to. |
| `--ci-requirements` | False | | | Output the CI requirements as a JSON payload. It is used to determine the CI runner to use. |
| `--python-registry-token` | False | | `PYTHON_REGISTRY_TOKEN` | The API token to authenticate with the registry. For pypi, the `pypi-` prefix needs to be specified |
| `--python-registry-url` | False | https://upload.pypi.org/legacy/ | `PYTHON_REGISTRY_URL` | The python registry to publish to. Defaults to main pypi |


I've added an empty "Default" column, and you can fill in the default values as needed.
Expand All @@ -406,14 +408,24 @@ flowchart TD
validate[Validate the metadata file]
check[Check if the connector image already exists]
build[Build the connector image for all platform variants]
publish_to_python_registry[Push the connector image to the python registry if enabled]
upload_spec[Upload connector spec to the spec cache bucket]
push[Push the connector image from DockerHub, with platform variants]
pull[Pull the connector image from DockerHub to check SPEC can be run and the image layers are healthy]
upload_metadata[Upload its metadata file to the metadata service bucket]
validate-->check-->build-->upload_spec-->push-->pull-->upload_metadata
validate-->check-->build-->upload_spec-->publish_to_python_registry-->push-->pull-->upload_metadata
```

#### Python registry publishing

If `remoteRegistries.pypi.enabled` in the connector metadata is set to `true`, the connector will be published to the python registry.
To do so, the `--python-registry-token` and `--python-registry-url` options are used to authenticate with the registry and publish the connector.
If the current version of the connector is already published to the registry, the publish will be skipped.

On a pre-release, the connector will be published as a `.dev<N>` version.


### <a id="connectors-bump_version"></a>`connectors bump_version` command

Bump the version of the selected connectors.
Expand Down Expand Up @@ -534,12 +546,12 @@ For poetry packages, the package name and version can be taken from the `pyproje

#### Options

| Option | Required | Default | Mapped environment variable | Description |
| ------------------------- | -------- | ----------------------- | --------------------------- | -------------------------------------------------------------------------------------------------------- |
| `--publish-name` | False | | | The name of the package. Not required for poetry packages that define it in the `pyproject.toml` file |
| `--publish-version` | False | | | The version of the package. Not required for poetry packages that define it in the `pyproject.toml` file |
| `--python-registry-token` | True | | PYTHON_REGISTRY_TOKEN | The API token to authenticate with the registry. For pypi, the `pypi-` prefix needs to be specified |
| `--registry-url` | False | https://pypi.org/simple | | The python registry to publish to. Defaults to main pypi |
| Option | Required | Default | Mapped environment variable | Description |
| ------------------------- | -------- | ------------------------------- | --------------------------- | -------------------------------------------------------------------------------------------------------- |
| `--publish-name` | False | | | The name of the package. Not required for poetry packages that define it in the `pyproject.toml` file |
| `--publish-version` | False | | | The version of the package. Not required for poetry packages that define it in the `pyproject.toml` file |
| `--python-registry-token` | True | | PYTHON_REGISTRY_TOKEN | The API token to authenticate with the registry. For pypi, the `pypi-` prefix needs to be specified |
| `--python-registry-url` | False | https://upload.pypi.org/legacy/ | PYTHON_REGISTRY_URL | The python registry to publish to. Defaults to main pypi |

### <a id="metadata-validate-command-subgroup"></a>`metadata` command subgroup

Expand Down Expand Up @@ -597,6 +609,7 @@ E.G.: running `pytest` on a specific test folder:

| Version | PR | Description |
| ------- | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| 3.9.0 | [#34606](https://github.com/airbytehq/airbyte/pull/34606) | Allow configuration of python registry URL via environment variable. |
| 3.8.1 | [#34607](https://github.com/airbytehq/airbyte/pull/34607) | Improve gradle dependency cache volume protection. |
| 3.8.0 | [#34316](https://github.com/airbytehq/airbyte/pull/34316) | Expose Dagger engine image name in `--ci-requirements` and add `--ci-requirements` to the `airbyte-ci` root command group. |
| 3.7.3 | [#34560](https://github.com/airbytehq/airbyte/pull/34560) | Simplify Gradle task execution framework by removing local maven repo support. |
Expand Down
19 changes: 18 additions & 1 deletion airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/publish/commands.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
from pipelines.cli.click_decorators import click_ci_requirements_option
from pipelines.cli.confirm_prompt import confirm
from pipelines.cli.dagger_pipeline_command import DaggerPipelineCommand
from pipelines.consts import ContextState
from pipelines.consts import DEFAULT_PYTHON_PACKAGE_REGISTRY_URL, ContextState
from pipelines.helpers.utils import fail_if_missing_docker_hub_creds


Expand Down Expand Up @@ -59,6 +59,19 @@
envvar="SLACK_CHANNEL",
default="#connector-publish-updates",
)
@click.option(
"--python-registry-token",
help="Access token for python registry",
type=click.STRING,
envvar="PYTHON_REGISTRY_TOKEN",
)
@click.option(
"--python-registry-url",
help="Which python registry registry to publish to. If not set, the default pypi is used. For test pypi, use https://test.pypi.org/legacy/",
type=click.STRING,
default=DEFAULT_PYTHON_PACKAGE_REGISTRY_URL,
envvar="PYTHON_REGISTRY_URL",
)
@click.pass_context
async def publish(
ctx: click.Context,
Expand All @@ -69,6 +82,8 @@ async def publish(
metadata_service_gcs_credentials: str,
slack_webhook: str,
slack_channel: str,
python_registry_token: str,
python_registry_url: str,
) -> bool:
ctx.obj["spec_cache_gcs_credentials"] = spec_cache_gcs_credentials
ctx.obj["spec_cache_bucket_name"] = spec_cache_bucket_name
Expand Down Expand Up @@ -109,6 +124,8 @@ async def publish(
s3_build_cache_access_key_id=ctx.obj.get("s3_build_cache_access_key_id"),
s3_build_cache_secret_key=ctx.obj.get("s3_build_cache_secret_key"),
use_local_cdk=ctx.obj.get("use_local_cdk"),
python_registry_token=python_registry_token,
python_registry_url=python_registry_url,
)
for connector in ctx.obj["selected_connectors_with_modified_files"]
]
Expand Down
4 changes: 4 additions & 0 deletions airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/publish/context.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,16 @@ def __init__(
s3_build_cache_access_key_id: Optional[str] = None,
s3_build_cache_secret_key: Optional[str] = None,
use_local_cdk: bool = False,
python_registry_token: Optional[str] = None,
python_registry_url: Optional[str] = None,
) -> None:
self.pre_release = pre_release
self.spec_cache_bucket_name = spec_cache_bucket_name
self.metadata_bucket_name = metadata_bucket_name
self.spec_cache_gcs_credentials = sanitize_gcs_credentials(spec_cache_gcs_credentials)
self.metadata_service_gcs_credentials = sanitize_gcs_credentials(metadata_service_gcs_credentials)
self.python_registry_token = python_registry_token
self.python_registry_url = python_registry_url
pipeline_name = f"Publish {connector.technical_name}"
pipeline_name = pipeline_name + " (pre-release)" if pre_release else pipeline_name

Expand Down
10 changes: 10 additions & 0 deletions airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/publish/pipeline.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -352,6 +352,16 @@ async def _run_python_registry_publish_pipeline(context: PublishConnectorContext
if not python_registry_context:
return results, False

if not context.python_registry_token or not context.python_registry_url:
# If the python registry token or url are not set, we can't publish to the python registry - stop the pipeline.
return [
StepResult(
PublishToPythonRegistry(python_registry_context),
status=StepStatus.FAILURE,
stderr="Pypi publishing is enabled, but python registry token or url are not set.",
)
], True

check_python_registry_package_exists_results = await CheckPythonRegistryPackageDoesNotExist(python_registry_context).run()
results.append(check_python_registry_package_exists_results)
if check_python_registry_package_exists_results.status is StepStatus.SKIPPED:
Expand Down
7 changes: 4 additions & 3 deletions airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/poetry/publish/commands.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,10 +47,11 @@ def _validate_python_version(_ctx: dict, _param: dict, value: Optional[str]) ->
envvar="PYTHON_REGISTRY_TOKEN",
)
@click.option(
"--registry-url",
"--python-registry-url",
help="Which registry to publish to. If not set, the default pypi is used. For test pypi, use https://test.pypi.org/legacy/",
type=click.STRING,
default=DEFAULT_PYTHON_PACKAGE_REGISTRY_URL,
envvar="PYTHON_REGISTRY_URL",
)
@click.option(
"--publish-name",
Expand All @@ -69,7 +70,7 @@ async def publish(
ctx: click.Context,
click_pipeline_context: ClickPipelineContext,
python_registry_token: str,
registry_url: str,
python_registry_url: str,
publish_name: Optional[str],
publish_version: Optional[str],
) -> bool:
Expand All @@ -85,7 +86,7 @@ async def publish(
ci_context=ctx.obj.get("ci_context"),
ci_gcs_credentials=ctx.obj["ci_gcs_credentials"],
python_registry_token=python_registry_token,
registry=registry_url,
registry=python_registry_url,
package_path=ctx.obj["package_path"],
package_name=publish_name,
version=publish_version,
Expand Down
Loading

0 comments on commit d1334be

Please sign in to comment.