connectors insights

.github/workflows/airbyte-ci-tests.yml

@@ -35,6 +35,7 @@ jobs:
             - airbyte-ci/connectors/pipelines/**
             - airbyte-ci/connectors/base_images/**
             - airbyte-ci/connectors/common_utils/**
+            - airbyte-ci/connectors/connectors_insights/**
             - airbyte-ci/connectors/connector_ops/**
             - airbyte-ci/connectors/connectors_qa/**
             - airbyte-ci/connectors/ci_credentials/**

.github/workflows/connectors_insights.yml

@@ -0,0 +1,34 @@
+name: Connectors Tests
+
+on:
+  schedule:
+    # 0AM UTC is 2AM CEST, 3AM EEST, 5PM PDT.
+    - cron: "0 0 * * *"
+  workflow_dispatch:
+
+jobs:
+  connectors_insights:
+    name: Connectors Insights generation
+    runs-on: connector-tooling-large
+    timeout-minutes: 1440 # 24 hours
+    env:
+      GOOGLE_APPLICATION_CREDENTIALS: ${{ secrets.METADATA_SERVICE_PROD_GCS_CREDENTIALS }}
+      DAGGER_CLOUD_TOKEN: ${{ secrets.DAGGER_CLOUD_TOKEN_2 }}
+    steps:
+      - name: Get Dagger Engine Image
+        uses: ./.github/actions/get-dagger-engine-image
+        with:
+          dagger_engine_image: "registry.dagger.io/engine:v0.9.6"
+      - name: Checkout Airbyte
+        uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.10"
+      - name: Install and configure Poetry
+        uses: snok/install-poetry@v1
+      - name: Install connector insights
+        shell: bash
+        run: |
+          poetry -C airbyte-ci/connectors/connectors_insights install
+          poetry -C airbyte-ci/connectors/connectors_insights run connector-insights generate --gcs-uri=gs://prod-airbyte-cloud-connector-metadata-service/connector_insights --connector-directory airbyte-integrations/connectors/ --concurrency 10

airbyte-ci/connectors/connector_ops/connector_ops/utils.py

@@ -613,6 +613,31 @@ def cloud_usage(self) -> Optional[str]:
 
         return get(connector_entry, "generated.metrics.cloud.usage")
 
+    @property
+    def image_address(self) -> str:
+        return f'{self.metadata["dockerRepository"]}:{self.metadata["dockerImageTag"]}'
+
+    @property
+    def cdk_name(self) -> str | None:
+        try:
+            return [tag.split(":")[-1] for tag in self.metadata["tags"] if tag.startswith("cdk:")][0]
+        except IndexError:
+            return None
+
+    @property
+    def base_image_address(self) -> str | None:
+        return self.metadata.get("connectorBuildOptions", {}).get("baseImage")
+
+    @property
+    def uses_base_image(self) -> bool:
+        return self.base_image_address is not None
+
+    @property
+    def base_image_version(self) -> str | None:
+        if not self.uses_base_image:
+            return None
+        return self.base_image_address.split(":")[1].split("@")[0]
+
     def get_secret_manager(self, gsm_credentials: str):
         return SecretsManager(connector_name=self.technical_name, gsm_credentials=gsm_credentials)
 

airbyte-ci/connectors/connectors_insights/README.md

@@ -0,0 +1,60 @@
+# Connectors Insights
+
+Connectors Insights is a Python project designed to generate various insights from analysis of our connectors code. This project utilizes Poetry for dependency management and packaging.
+
+## Artifacts Produced
+The project generates the following artifacts:
+
+- `insights.json`: Contains general insights and metadata about the connectors.
+- `sbom.json`: Contains the Software Bill Of Material. Produced by [Syft](https://github.com/anchore/syft).
+
+## Installation
+To install the project and its dependencies, ensure you have Poetry installed, then run:
+```sh
+poetry install
+```
+
+## Usage
+The Connectors Insights project provides a command-line interface (CLI) to generate the artifacts. Below is the command to run the CLI:
+
+```sh
+# From airbyte root directory
+connectors-insights generate --output-directory <path-to-local-output-dir> --gcs-uri=gs://<bucket>/<key-prefix> --connector-directory airbyte-integrations/connectors/ --concurrency 2 --rewrite
+```
+
+### CLI Options
+
+- `generate`: The command to generate the artifacts.
+
+- `-o, --output-dir`: Specifies the local directory where the generated artifacts will be saved. In this example, artifacts are saved to `/Users/augustin/Desktop/insights`.
+
+- `-g, --gcs-uri`: The Google Cloud Storage (GCS) URI prefix where the artifacts will be uploaded. In the form: `gs://<bucket>/<key-prefix>`.
+
+- `-d, --connector-directory`: The directory containing the connectors. This option points to the location of the connectors to be analyzed, here it is `airbyte-integrations/connectors/`.
+
+- `-c, --concurrency`: Sets the level of concurrency for the generation process. In this example, it is set to `2`.
+
+- `--rewrite`: If provided, this flag indicates that existing artifacts should be rewritten if they already exist.
+
+## Example
+To generate the artifacts and save them both locally and to GCS, you can use the following command:
+
+```sh
+connectors-insights generate --output-directory <path-to-local-output-dir> --gcs-uri=gs://<bucket>/<key-prefix> --connector-directory airbyte-integrations/connectors/ --concurrency 2 --rewrite
+```
+
+This command will generate `insights.json` and `sbom.json` files, saving them to the specified local directory and uploading them to the specified GCS URI if `--gcs-uri` is passed.
+
+### Examples of generated artifacts
+* [`insights.json`](https://storage.googleapis.com/prod-airbyte-cloud-connector-metadata-service/connector_insights/source-faker/latest/insights.json)
+* [`sbom.json`](https://storage.googleapis.com/prod-airbyte-cloud-connector-metadata-service/connector_insights/source-faker/latest/sbom.json)
+
+
+## Orchestration
+
+This CLI is currently running nightly in GitHub Actions. The workflow can be found in `.github/workflows/connector_insights.yml`.
+
+## Changelog
+
+### 0.1.0
+- Initial release