Skip to content

Commit

Permalink
airbyte-ci: add connectors list command (#26496)
Browse files Browse the repository at this point in the history
  • Loading branch information
alafanechere committed May 24, 2023
1 parent 360e2f9 commit 7d7996f
Show file tree
Hide file tree
Showing 2 changed files with 61 additions and 1 deletion.
25 changes: 25 additions & 0 deletions tools/ci_connector_ops/ci_connector_ops/pipelines/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ N.B: This project will eventually be moved to `airbyte-ci` root directory.
* [Options](#options)
- [`connectors` command subgroup](#connectors-command-subgroup)
* [Options](#options-1)
- [`connectors list` command](#connectors-list-command)
- [`connectors test` command](#connectors-test-command)
* [Examples](#examples-)
* [What it runs](#what-it-runs-)
Expand Down Expand Up @@ -88,6 +89,30 @@ Available commands:
| `--modified` | False | False | Run the pipeline on only the modified connectors on the branch or previous commit (depends on the pipeline implementation). |
| `--concurrency` | False | 5 | Control the number of connector pipelines that can run in parallel. Useful to speed up pipelines or control their resource usage. |

### <a id="connectors-list-command"></a>`connectors list` command
Retrieve the list of connectors satisfying the provided filters.

#### Examples
List all connectors:

`airbyte-ci connectors list`

List generally available connectors:

`airbyte-ci connectors --release-stage=generally_available list`

List connectors changed on the current branch:

`airbyte-ci connectors --modified list`

List connectors with a specific language:

`airbyte-ci connectors --language=python list`

List connectors with multiple filters:

`airbyte-ci connectors --language=low-code --release-stage=generally_available list`

### <a id="connectors-test-command"></a>`connectors test` command
Run a test pipeline for one or multiple connectors.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,10 @@
from ci_connector_ops.pipelines.publish import run_connector_publish_pipeline
from ci_connector_ops.pipelines.tests import run_connector_test_pipeline
from ci_connector_ops.pipelines.utils import DaggerPipelineCommand, get_modified_connectors, get_modified_metadata_files
from ci_connector_ops.utils import ConnectorLanguage, get_all_released_connectors
from ci_connector_ops.utils import ConnectorLanguage, console, get_all_released_connectors
from rich.logging import RichHandler
from rich.table import Table
from rich.text import Text

# CONSTANTS

Expand Down Expand Up @@ -374,3 +376,36 @@ def validate_publish_options(pre_release: bool, context_object: Dict[str, Any]):
for k in ["spec_cache_bucket_name", "spec_cache_gcs_credentials", "metadata_service_bucket_name", "metadata_service_gcs_credentials"]:
if not pre_release and context_object.get(k) is None:
click.Abort(f'The --{k.replace("_", "-")} option is required when running a main release publish pipeline.')


@connectors.command(cls=DaggerPipelineCommand, help="List all selected connectors.")
@click.pass_context
def list(
ctx: click.Context,
):
selected_connectors = [
(connector, bool(modified_files))
for connector, modified_files in ctx.obj["selected_connectors_and_files"].items()
if connector.metadata
]

selected_connectors = sorted(selected_connectors, key=lambda x: x[0].technical_name)
table = Table(title=f"{len(selected_connectors)} selected connectors")
table.add_column("Modified")
table.add_column("Connector")
table.add_column("Language")
table.add_column("Release stage")
table.add_column("Version")
table.add_column("Folder")

for connector, modified in selected_connectors:
modified = "X" if modified else ""
connector_name = Text(connector.technical_name)
language = Text(connector.language.value) if connector.language else "N/A"
release_stage = Text(connector.release_stage)
version = Text(connector.version)
folder = Text(str(connector.code_directory))
table.add_row(modified, connector_name, language, release_stage, version, folder)

console.print(table)
return True

0 comments on commit 7d7996f

Please sign in to comment.