Airbyte-ci: Add skippable connector test steps

airbyte-ci/connectors/pipelines/README.md

@@ -52,8 +52,8 @@ If you face any installation problems feel free to reach out the Airbyte Connect
 ### Setting up connector secrets access
 
 If you plan to use Airbyte CI to run CAT (Connector Acceptance Tests), we recommend setting up GSM
-access so that Airbyte CI can pull remote secrets from GSM. For setup instructions, see the 
-CI Credentials package (which Airbyte CI uses under the hood) README's 
+access so that Airbyte CI can pull remote secrets from GSM. For setup instructions, see the
+CI Credentials package (which Airbyte CI uses under the hood) README's
 [Get GSM Access](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/ci_credentials/README.md#get-gsm-access)
 instructions.
 
@@ -155,7 +155,7 @@ Available commands:
 | `--enable-dependency-scanning / --disable-dependency-scanning` | False    | ` --disable-dependency-scanning` |                             | When enabled the dependency scanning will be performed to detect the connectors to select according to a dependency change.                                                                                                                                                                                                                                                                                                                                                               |
 | `--docker-hub-username`                                        |          |                                  | DOCKER_HUB_USERNAME         | Your username to connect to DockerHub. Required for the publish subcommand.                                                                                                                                                                                                                                                                                                                                                                                                               |
 | `--docker-hub-password`                                        |          |                                  | DOCKER_HUB_PASSWORD         | Your password to connect to DockerHub. Required for the publish subcommand.                                                                                                                                                                                                                                                                                                                                                                                                               |
-                                                                                                                                           
+
 
 ### <a id="connectors-list-command"></a>`connectors list` command
 Retrieve the list of connectors satisfying the provided filters.
@@ -236,8 +236,8 @@ flowchart TD
 
 | Option              | Multiple | Default value | Description                                                                                                                                                                                              |
 | ------------------- | -------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `--skip-step/-x`    | True     |               | Skip steps by id e.g. `-x unit -x acceptance`                                                                                                         |
 | `--fail-fast`       | False    | False         | Abort after any tests fail, rather than continuing to run additional tests. Use this setting to confirm a known bug is fixed (or not), or when you only require a pass/fail result.                      |
-| `--fast-tests-only` | True     | False         | Run unit tests only, skipping integration tests or any tests explicitly tagged as slow. Use this for more frequent checks, when it is not feasible to run the entire test suite.                         |
 | `--code-tests-only` | True     | False         | Skip any tests not directly related to code updates. For instance, metadata checks, version bump checks, changelog verification, etc. Use this setting to help focus on code quality during development. |
 | `--concurrent-cat`  | False    | False         | Make CAT tests run concurrently using pytest-xdist. Be careful about source or destination API rate limits.                                                                                              |
 
@@ -408,6 +408,7 @@ This command runs the Python tests for a airbyte-ci poetry package.
 ## Changelog
 | Version | PR                                                         | Description                                                                                               |
 | ------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
+| 2.6.0   | [#31974](https://github.com/airbytehq/airbyte/pull/?????)  | Add -x option to connector test to allow for skipping steps                                               |
 | 2.5.3   | [#31974](https://github.com/airbytehq/airbyte/pull/31974)  | Fix latest CDK install and pip cache mount on connector install.                            |
 | 2.5.2   | [#31871](https://github.com/airbytehq/airbyte/pull/31871)  | Deactivate PR comments, add HTML report links to the PR status when its ready.                            |
 | 2.5.1   | [#31774](https://github.com/airbytehq/airbyte/pull/31774)  | Add a docker configuration check on `airbyte-ci` startup.                                                 |

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/context.py

@@ -6,9 +6,11 @@
 
 from datetime import datetime
 from types import TracebackType
-from typing import Optional
+from typing import List, Optional
 
 import yaml
+import functools
+
 from anyio import Path
 from asyncer import asyncify
 from dagger import Directory, Secret
@@ -18,6 +20,7 @@
 from pipelines.helpers.connectors.modifed import ConnectorWithModifiedFiles
 from pipelines.helpers.github import update_commit_status_check
 from pipelines.helpers.slack import send_message_to_webhook
+from pipelines.helpers.run_steps import RunStepOptions
 from pipelines.helpers.utils import METADATA_FILE_NAME
 from pipelines.models.contexts import PipelineContext
 
@@ -49,8 +52,6 @@ def __init__(
         reporting_slack_channel: Optional[str] = None,
         pull_request: PullRequest = None,
         should_save_report: bool = True,
-        fail_fast: bool = False,
-        fast_tests_only: bool = False,
         code_tests_only: bool = False,
         use_local_cdk: bool = False,
         use_host_gradle_dist_tar: bool = False,
@@ -60,6 +61,7 @@ def __init__(
         s3_build_cache_access_key_id: Optional[str] = None,
         s3_build_cache_secret_key: Optional[str] = None,
         concurrent_cat: Optional[bool] = False,
+        run_step_options: RunStepOptions = RunStepOptions(),
     ):
         """Initialize a connector context.
 
@@ -78,8 +80,6 @@ def __init__(
             slack_webhook (Optional[str], optional): The slack webhook to send messages to. Defaults to None.
             reporting_slack_channel (Optional[str], optional): The slack channel to send messages to. Defaults to None.
             pull_request (PullRequest, optional): The pull request object if the pipeline was triggered by a pull request. Defaults to None.
-            fail_fast (bool, optional): Whether to fail fast. Defaults to False.
-            fast_tests_only (bool, optional): Whether to run only fast tests. Defaults to False.
             code_tests_only (bool, optional): Whether to ignore non-code tests like QA and metadata checks. Defaults to False.
             use_host_gradle_dist_tar (bool, optional): Used when developing java connectors with gradle. Defaults to False.
             open_report_in_browser (bool, optional): Open HTML report in browser window. Defaults to True.
@@ -99,8 +99,6 @@ def __init__(
         self._updated_secrets_dir = None
         self.cdk_version = None
         self.should_save_report = should_save_report
-        self.fail_fast = fail_fast
-        self.fast_tests_only = fast_tests_only
         self.code_tests_only = code_tests_only
         self.use_local_cdk = use_local_cdk
         self.use_host_gradle_dist_tar = use_host_gradle_dist_tar
@@ -110,6 +108,7 @@ def __init__(
         self.s3_build_cache_access_key_id = s3_build_cache_access_key_id
         self.s3_build_cache_secret_key = s3_build_cache_secret_key
         self.concurrent_cat = concurrent_cat
+        self._connector_secrets = None
 
         super().__init__(
             pipeline_name=pipeline_name,
@@ -128,6 +127,7 @@ def __init__(
             ci_git_user=ci_git_user,
             ci_github_access_token=ci_github_access_token,
             open_report_in_browser=open_report_in_browser,
+            run_step_options=run_step_options,
         )
 
     @property
@@ -206,6 +206,11 @@ def docker_hub_password_secret(self) -> Optional[Secret]:
             return None
         return self.dagger_client.set_secret("docker_hub_password", self.docker_hub_password)
 
+    async def connector_secrets(self):
+        if self._connector_secrets is None:
+            self._connector_secrets = await secrets.get_connector_secrets(self)
+        return self._connector_secrets
+
     async def get_connector_dir(self, exclude=None, include=None) -> Directory:
         """Get the connector under test source code directory.
 

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/test/commands.py

@@ -12,6 +12,7 @@
 from pipelines.cli.dagger_pipeline_command import DaggerPipelineCommand
 from pipelines.consts import ContextState
 from pipelines.helpers.github import update_global_commit_status_check_for_tests
+from pipelines.helpers.run_steps import RunStepOptions
 from pipelines.helpers.utils import fail_if_missing_docker_hub_creds
 
 
@@ -30,27 +31,27 @@
     type=bool,
     is_flag=True,
 )
-@click.option(
-    "--fast-tests-only",
-    help="When enabled, slow tests are skipped.",
-    default=False,
-    type=bool,
-    is_flag=True,
-)
 @click.option(
     "--concurrent-cat",
     help="When enabled, the CAT tests will run concurrently. Be careful about rate limits",
     default=False,
     type=bool,
     is_flag=True,
 )
+@click.option(
+    '--skip-step',
+    '-x',
+    multiple=True,
+    type=str,
+    help="Skip a step by name. Can be used multiple times to skip multiple steps.",
+)
 @click.pass_context
 async def test(
     ctx: click.Context,
     code_tests_only: bool,
     fail_fast: bool,
-    fast_tests_only: bool,
     concurrent_cat: bool,
+    skip_step: str,
 ) -> bool:
     """Runs a test pipeline for the selected connectors.
 
@@ -70,6 +71,11 @@ async def test(
         update_global_commit_status_check_for_tests(ctx.obj, "success")
         return True
 
+    run_step_options = RunStepOptions(
+        fail_fast=fail_fast,
+        skip_steps=skip_step,
+    )
+
     connectors_tests_contexts = [
         ConnectorContext(
             pipeline_name=f"Testing connector {connector.technical_name}",
@@ -86,18 +92,18 @@ async def test(
             ci_context=ctx.obj.get("ci_context"),
             pull_request=ctx.obj.get("pull_request"),
             ci_gcs_credentials=ctx.obj["ci_gcs_credentials"],
-            fail_fast=fail_fast,
-            fast_tests_only=fast_tests_only,
             code_tests_only=code_tests_only,
             use_local_cdk=ctx.obj.get("use_local_cdk"),
             s3_build_cache_access_key_id=ctx.obj.get("s3_build_cache_access_key_id"),
             s3_build_cache_secret_key=ctx.obj.get("s3_build_cache_secret_key"),
             docker_hub_username=ctx.obj.get("docker_hub_username"),
             docker_hub_password=ctx.obj.get("docker_hub_password"),
             concurrent_cat=concurrent_cat,
+            run_step_options=run_step_options,
         )
         for connector in ctx.obj["selected_connectors_with_modified_files"]
     ]
+
     try:
         await run_connectors_pipelines(
             [connector_context for connector_context in connectors_tests_contexts],

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/test/pipeline.py

@@ -3,125 +3,67 @@
 #
 """This module groups factory like functions to dispatch tests steps according to the connector under test language."""
 
-import itertools
 from typing import List
 
 import anyio
-import asyncer
 from connector_ops.utils import METADATA_FILE_NAME, ConnectorLanguage
 from pipelines.airbyte_ci.connectors.context import ConnectorContext
 from pipelines.airbyte_ci.connectors.reports import ConnectorReport
 from pipelines.airbyte_ci.connectors.test.steps import java_connectors, python_connectors
 from pipelines.airbyte_ci.connectors.test.steps.common import QaChecks, VersionFollowsSemverCheck, VersionIncrementCheck
 from pipelines.airbyte_ci.metadata.pipeline import MetadataValidation
-from pipelines.models.steps import StepResult
+from pipelines.helpers.run_steps import StepToRun, run_steps
 
 LANGUAGE_MAPPING = {
-    "run_all_tests": {
-        ConnectorLanguage.PYTHON: python_connectors.run_all_tests,
-        ConnectorLanguage.LOW_CODE: python_connectors.run_all_tests,
-        ConnectorLanguage.JAVA: java_connectors.run_all_tests,
-    },
-    "run_code_format_checks": {
-        ConnectorLanguage.PYTHON: python_connectors.run_code_format_checks,
-        ConnectorLanguage.LOW_CODE: python_connectors.run_code_format_checks,
-        # ConnectorLanguage.JAVA: java_connectors.run_code_format_checks
+    "get_test_steps": {
+        ConnectorLanguage.PYTHON: python_connectors.get_test_steps,
+        ConnectorLanguage.LOW_CODE: python_connectors.get_test_steps,
+        ConnectorLanguage.JAVA: java_connectors.get_test_steps,
     },
 }
 
 
-async def run_metadata_validation(context: ConnectorContext) -> List[StepResult]:
-    """Run the metadata validation on a connector.
-    Args:
-        context (ConnectorContext): The current connector context.
-
-    Returns:
-        List[StepResult]: The results of the metadata validation steps.
-    """
-    return [await MetadataValidation(context).run()]
-
-
-async def run_version_checks(context: ConnectorContext) -> List[StepResult]:
-    """Run the version checks on a connector.
-
-    Args:
-        context (ConnectorContext): The current connector context.
-
-    Returns:
-        List[StepResult]: The results of the version checks steps.
-    """
-    return [await VersionFollowsSemverCheck(context).run(), await VersionIncrementCheck(context).run()]
-
-
-async def run_qa_checks(context: ConnectorContext) -> List[StepResult]:
-    """Run the QA checks on a connector.
-
-    Args:
-        context (ConnectorContext): The current connector context.
-
-    Returns:
-        List[StepResult]: The results of the QA checks steps.
-    """
-    return [await QaChecks(context).run()]
-
-
-async def run_code_format_checks(context: ConnectorContext) -> List[StepResult]:
-    """Run the code format checks according to the connector language.
-
-    Args:
-        context (ConnectorContext): The current connector context.
-
-    Returns:
-        List[StepResult]: The results of the code format checks steps.
-    """
-    if _run_code_format_checks := LANGUAGE_MAPPING["run_code_format_checks"].get(context.connector.language):
-        return await _run_code_format_checks(context)
-    else:
-        context.logger.warning(f"No code format checks defined for connector language {context.connector.language}!")
-        return []
-
-
-async def run_all_tests(context: ConnectorContext) -> List[StepResult]:
-    """Run all the tests steps according to the connector language.
+def get_test_steps(context: ConnectorContext) -> List[StepToRun]:
+    """Get all the tests steps according to the connector language.
 
     Args:
         context (ConnectorContext): The current connector context.
 
     Returns:
-        List[StepResult]: The results of the tests steps.
+        List[StepResult]: The list of tests steps.
     """
-    if _run_all_tests := LANGUAGE_MAPPING["run_all_tests"].get(context.connector.language):
-        return await _run_all_tests(context)
+    if _get_test_steps := LANGUAGE_MAPPING["get_test_steps"].get(context.connector.language):
+        return _get_test_steps(context)
     else:
         context.logger.warning(f"No tests defined for connector language {context.connector.language}!")
         return []
 
 
-async def run_connector_test_pipeline(context: ConnectorContext, semaphore: anyio.Semaphore) -> ConnectorReport:
-    """Run a test pipeline for a single connector.
+async def run_connector_test_pipeline(context: ConnectorContext, semaphore: anyio.Semaphore):
+    """
+    Compute the steps to run for a connector test pipeline.
+    """
 
-    A visual DAG can be found on the README.md file of the pipelines modules.
+    steps_to_run = get_test_steps(context)
 
-    Args:
-        context (ConnectorContext): The initialized connector context.
+    if not context.code_tests_only:
+        steps_to_run += [
+            [
+                StepToRun(id="metadata_validation", step=MetadataValidation(context)),
+                StepToRun(id="version_follow_check", step=VersionFollowsSemverCheck(context)),
+                StepToRun(id="version_inc_check", step=VersionIncrementCheck(context)),
+                StepToRun(id="qa_checks", step=QaChecks(context)),
+            ]
+        ]
 
-    Returns:
-        ConnectorReport: The test reports holding tests results.
-    """
     async with semaphore:
         async with context:
-            async with asyncer.create_task_group() as task_group:
-                tasks = [
-                    task_group.soonify(run_all_tests)(context),
-                    task_group.soonify(run_code_format_checks)(context),
-                ]
-                if not context.code_tests_only:
-                    tasks += [
-                        task_group.soonify(run_metadata_validation)(context),
-                        task_group.soonify(run_version_checks)(context),
-                        task_group.soonify(run_qa_checks)(context),
-                    ]
-            results = list(itertools.chain(*(task.value for task in tasks)))
+            result_dict = await run_steps(
+                runnables=steps_to_run,
+                options=context.run_step_options,
+            )
+
+            results = list(result_dict.values())
             context.report = ConnectorReport(context, steps_results=results, name="TEST RESULTS")
 
         return context.report