Airbyte-ci: Add skippable connector test steps #32188

bnchrch · 2023-11-03T22:50:10Z

Problem

As a connector developer I would like to be able to skip individual steps in the test pipeline

Proposed Solution

airbyte-ci connectors --name=source-s3 test -x unit -x acceptance

Implementation

Defining the Steps to be run as a list

def get_test_steps(context: ConnectorContext) -> List[StepToRun]:
    """
    Get all the tests steps for a Python connector.
    """
    return [
        [StepToRun(id="build", step=BuildConnectorImages(context, LOCAL_BUILD_PLATFORM))],
        [StepToRun(
            id="unit",
            step=UnitTests(context),
            args=lambda results: {"connector_under_test": results["build"].output_artifact[LOCAL_BUILD_PLATFORM]},
            depends_on=["build"],
        )],
        [
            StepToRun(
                id="integration",
                step=IntegrationTests(context),
                args=lambda results: {"connector_under_test": results["build"].output_artifact[LOCAL_BUILD_PLATFORM]},
                depends_on=["build"],
            ),
            StepToRun(
                id="acceptance",
                step=AcceptanceTests(context, context.concurrent_cat),
                args=lambda results: {"connector_under_test_container": results["build"].output_artifact[LOCAL_BUILD_PLATFORM]},
                depends_on=["build"],
            ),
            StepToRun(id="check_base_image", step=CheckBaseImageIsUsed(context), depends_on=["build"]),
        ],
    ]

Other minor changes

connector_secrets is now explicitly part of the ConnectorContext

Why a list?

This will give us a type and structure we can manupilate prior to running, enabling features like

Skipping steps
Passing extra args
Eagerly failing tests on failing dependencies
Generating a test report prior to the tests completing

Why a new class?

The new class StepToRun was added for two reasons

Its difficult to add this functionality to all steps without a much larger rewrite
The need for dynamic args when calling run still would require an addtional class / datamodel
This lets us keep all coordination logic inside the run_steps.py file

…options-for-steps

vercel · 2023-11-03T22:50:17Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Dec 15, 2023 8:16pm

alafanechere

I LOVE the direction where this is going.

alafanechere · 2023-11-07T13:38:02Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/test/commands.py

+@click.option(
+    '--skip-step',
+    '-x',
+    multiple=True,
+    type=str,
+    help="Skip a step by name. Can be used multiple times to skip multiple steps.",
+)


How big of a lift would it be to have an --only option to select a specific test suite?

Not much at all! I can do that in another PR

Issue here: https://github.com/airbytehq/airbyte/issues/33309

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/test/commands.py

airbyte-ci/connectors/pipelines/pipelines/helpers/run_steps.py

alafanechere · 2023-11-07T14:06:09Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/context.py

+    async def connector_secrets(self):
+        if self._connector_secrets is None:
+            self._connector_secrets = await secrets.get_connector_secrets(self)
+        return self._connector_secrets
+


Is this related to your changes?

It is! We were mounting connector secrets in between steps. So I hoisted that logic into the context so when you request connector_secrets it mounts and gets them if it hasnt already

Could we rename the function to get_connector_secrets then? Or would it make sense to make secrets.get_connector_secrets a method of the context class?

Im torn on this

On one hand it belongs in context given how coupled it is to a ConnectorContext

On the other hand connector secrets is a foundational concept that deserves a separate file to hold onto the logic.

I think we gain more from it being separate.

WDYT?

I suggest renaming it to get_connector_secrets as it:

Downloads the connector secrets

Assign the secrets to self._connector_secrets.

And we rename self._connector_secrets to self.connector_secrets to make it public.

I'd be confused to have a method named connector_secrets acting as a getter on self._connector_secrets

Sounds good! Though, lets leave the internal field with the underscore. We want devs to always reference the getter and feel bad otherwise

alafanechere · 2023-11-07T14:38:21Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/test/steps/python_connectors.py

+            args=lambda results: {"connector_under_test": results["build"].output_artifact[LOCAL_BUILD_PLATFORM]},
+            depends_on=["build"],


If I'm not mistaken the dependency is always implicitly defined by the fact that a Step args are from a dependency output_artifact. Could we infer the dependency tree dynamically from the args then?
I know explicit is better than implicit 😄 , so feel free to reject this suggestion

Thats fair! but there sometimes output that may not be an artifact, thus theres no implicit relationship either.

e.g. we have a step called PushToDockerHub and a step called VerifyOnDockerHub

alafanechere · 2023-11-07T14:44:10Z

airbyte-ci/connectors/pipelines/tests/test_helpers/test_run_steps.py

Could you add tests that check that argument passing between steps works?

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/test/steps/java_connectors.py

alafanechere · 2023-11-07T14:49:55Z

airbyte-ci/connectors/pipelines/pipelines/helpers/run_steps.py

+    id: str
+    step: Step
+    args: ARGS_TYPE = field(default_factory=dict)
+    depends_on: List[str] = field(default_factory=list)


I would find it more correct and robust if the depends_on type was a list of StepToRun instances

Im not full against that but why is that?

A child could know its parents if depends_on was a StepToRun

And it limits id typos as we'd pass around a variable

So with the need of having --help list the name of available steps to skip We've added the pattern where steps id's are defined as constants and then reused.

That should fix the typo problem you pointed out here.

Which I think is a good approach since we dont want to pass around fully instantiated StepToRun objects as that would lead us into a couple of painful areas

Passing the BuildStepInstance everywhere

Preventing us from having a polymorphic build step. e.g. BuildStepInstance or BuildWithNormalizationStepInstance

WDYT?

alafanechere · 2023-11-07T14:56:13Z

airbyte-ci/connectors/pipelines/pipelines/helpers/run_steps.py

+
+
+@dataclass(frozen=True)
+class StepToRun:


Just opening the discussion here.
Could we rename StepToRun to Step
And Step to Operation?

A Operation.__init__ would take a Step instance and would access parameters from Step.args.

Step instances would be "singleton"
Operation instance would not be singleton: an operation can be re-used in multiple step.
And I think Operation might be worth being functions as they'll be stateless.

Im all for finding a way to cut down the awkwardness (and layers) of these abstraction.

Hearing works like "singleton", "would be" and "I think" is telling me that this might be better tackled on its own so we can just wrestle with restructuring the taxonomy and not restructuring the taxonomy AND adding new functionality.

WDYT?

bnchrch · 2023-12-11T19:05:43Z

@alafanechere This is ready for review again!

Since the last review

Updated to latest master
Added support for --help listing skippable steps for -x
Made StepToRun ids less vulnerable to typos
Added checks for args being one of callable, awaitable or dict
Updated tests to have descriptions
Added test cases for skip steps, fast failures, concurrency, invalid args and more
Added configurable concurrency support

I also left a few discussions open around depends_on and get_connector_secrets

What im worried about is that I may have clobbered a change we've made in the last month. I havent found any regression but would love your eagle eyes on that!

airbyte-ci/connectors/pipelines/pipelines/consts.py

alafanechere

I think there's a bug in the input validation:
airbyte-ci connectors --name=source-faker test -x=acceptance:

┃ Error: Invalid value for '--skip-step' / '-x': '=acceptance' is not one of <CONNECTOR_TEST_STEP_ID.ACCEPTANCE: 'acceptance'>, <CONNECTOR_TEST_STEP_ID.BUILD_NORMALIZATION: 'build_normalization'>, <CONNECTOR_TEST_STEP_ID.BUILD_TAR: 'build_tar'>, <CONNECTOR_TEST_STEP_ID.BUILD: 'build'>, <CONNECTOR_TEST_S
┃ TEP_ID.CHECK_BASE_IMAGE: 'check_base_image'>, <CONNECTOR_TEST_STEP_ID.INTEGRATION: 'integration'>, <CONNECTOR_TEST_STEP_ID.METADATA_VALIDATION: 'metadata_validation'>, <CONNECTOR_TEST_STEP_ID.QA_CHECKS: 'qa_checks'>, <CONNECTOR_TEST_STEP_ID.UNIT: 'unit'>, <CONNECTOR_TEST_STEP_ID.VERSION_FOLLOW_CHECK:
┃ 'version_follow_check'>, <CONNECTOR_TEST_STEP_ID.VERSION_INC_CHECK: 'version_inc_check'>, <CONNECTOR_TEST_STEP_ID.TEST_ORCHESTRATOR: 'test_orchestrator'>, <CONNECTOR_TEST_STEP_ID.DEPLOY_ORCHESTRATOR: 'deploy_orchestrator'>.

…options-for-steps

bnchrch · 2023-12-14T23:35:31Z

I think thats just a quirk of click.choice

We have the same issue with the -a flag in build

bnchrch · 2023-12-15T00:06:12Z

@alafanechere Ready for your review again :)

alafanechere · 2023-12-15T14:22:04Z

@bnchrch I figured that = can be used with long options -- but with short one a space is expected... Might be a convention I was not aware of.
I changed the Choice a bit to have a cleaner output in case the passed -x value is invalid

alafanechere

Ship it 🎉
I think I fixed the test suite + conflicts.
I manually tested connectors test , everything is working as expected so far!

Co-authored-by: Augustin <augustin@airbyte.io> Co-authored-by: bnchrch <bnchrch@users.noreply.github.com> Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>

bnchrch added 10 commits October 23, 2023 09:29

Remove fail fast

f1732af

Remove fast test only

866bc56

Add new pipeline demo file

5a1c1cc

Remove code format checks

27267a6

Wrapped step

3c0b584

Merge remote-tracking branch 'origin/master' into bnchrch/airbyte-ci/…

bab78ab

…options-for-steps

Add get test step skeleton

2de8d43

Get runnable working

f90f002

Add skip support

dd4d275

Add get_test_steps to java connector

ab950d3

Remove old code

258b66d

bnchrch force-pushed the bnchrch/airbyte-ci/options-for-steps branch from 2cb24a3 to 258b66d Compare November 3, 2023 22:50

bnchrch added 3 commits November 3, 2023 16:51

Remove outdatted methods

3a518ca

Add tests

be10290

Rename to run_step

3505170

bnchrch force-pushed the bnchrch/airbyte-ci/options-for-steps branch from aa29862 to 3505170 Compare November 3, 2023 23:32

bnchrch added 2 commits November 6, 2023 07:57

Add paralell tests

e59dc3c

Add concurrency tests

3e54bd4

bnchrch requested review from alafanechere and erohmensing November 6, 2023 17:00

bnchrch added 3 commits November 6, 2023 10:02

Remove commented out code

ef6f26a

Fix secrets

5a61d50

Get async args working

bba7057

bnchrch marked this pull request as ready for review November 6, 2023 22:17

bnchrch requested a review from a team November 6, 2023 22:17

bnchrch added 3 commits November 6, 2023 15:32

Add new type for results

0abd1eb

Add new tests

45a7e23

Add step tree logging

fbcf237

alafanechere reviewed Nov 7, 2023

View reviewed changes

bnchrch added 5 commits December 11, 2023 09:47

Ensure --help works

31d98c9

Add tests for skip steps

8a5bbb9

Parameterize invalid args

5ac61e3

Add concurrency support

f6ed2a5

Fix mocker tests

8f57adb

bnchrch requested a review from alafanechere December 11, 2023 19:05

bnchrch mentioned this pull request Oct 11, 2023

Add the -k ability to the airbyte-ci test command #24061

Closed

alafanechere reviewed Dec 13, 2023

View reviewed changes

airbyte-ci/connectors/pipelines/pipelines/consts.py Outdated Show resolved Hide resolved

alafanechere requested changes Dec 13, 2023

View reviewed changes

Merge remote-tracking branch 'origin/master' into bnchrch/airbyte-ci/…

fa233f9

…options-for-steps

bnchrch added 2 commits December 14, 2023 15:44

Address PR comments

cddd975

Fix secrets

4ffa61d

bnchrch requested a review from alafanechere December 15, 2023 00:05

alafanechere added 3 commits December 15, 2023 14:56

improve Choice and skip step validation for better UX

13349dd

fix tests

b99e39c

Merge branch 'master' into bnchrch/airbyte-ci/options-for-steps

ffae7e2

alafanechere approved these changes Dec 15, 2023

View reviewed changes

vercel bot deployed to Preview December 15, 2023 14:28 View deployment

Fix test

485d0c4

bnchrch enabled auto-merge (squash) December 15, 2023 20:16

bnchrch merged commit e444d75 into master Dec 15, 2023
21 checks passed

bnchrch deleted the bnchrch/airbyte-ci/options-for-steps branch December 15, 2023 20:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Airbyte-ci: Add skippable connector test steps #32188

Airbyte-ci: Add skippable connector test steps #32188

bnchrch commented Nov 3, 2023 •

edited

vercel bot commented Nov 3, 2023 •

edited

alafanechere left a comment

alafanechere Nov 7, 2023

bnchrch Nov 7, 2023 •

edited

alafanechere Nov 7, 2023

bnchrch Nov 7, 2023

alafanechere Nov 7, 2023

bnchrch Dec 8, 2023

alafanechere Dec 13, 2023

bnchrch Dec 14, 2023

alafanechere Nov 7, 2023

bnchrch Dec 9, 2023

alafanechere Nov 7, 2023

alafanechere Nov 7, 2023

bnchrch Nov 7, 2023

alafanechere Nov 7, 2023

alafanechere Nov 7, 2023 •

edited

bnchrch Dec 11, 2023

alafanechere Nov 7, 2023

bnchrch Nov 7, 2023

alafanechere Nov 7, 2023

bnchrch commented Dec 11, 2023 •

edited

alafanechere left a comment

bnchrch commented Dec 14, 2023

bnchrch commented Dec 15, 2023

alafanechere commented Dec 15, 2023

alafanechere left a comment

		args=lambda results: {"connector_under_test": results["build"].output_artifact[LOCAL_BUILD_PLATFORM]},
		depends_on=["build"],

Airbyte-ci: Add skippable connector test steps #32188

Airbyte-ci: Add skippable connector test steps #32188

Conversation

bnchrch commented Nov 3, 2023 • edited

Problem

Proposed Solution

Implementation

Other minor changes

Why a list?

Why a new class?

vercel bot commented Nov 3, 2023 • edited

alafanechere left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bnchrch Nov 7, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alafanechere Nov 7, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bnchrch commented Dec 11, 2023 • edited

alafanechere left a comment

Choose a reason for hiding this comment

bnchrch commented Dec 14, 2023

bnchrch commented Dec 15, 2023

alafanechere commented Dec 15, 2023

alafanechere left a comment

Choose a reason for hiding this comment

bnchrch commented Nov 3, 2023 •

edited

vercel bot commented Nov 3, 2023 •

edited

bnchrch Nov 7, 2023 •

edited

alafanechere Nov 7, 2023 •

edited

bnchrch commented Dec 11, 2023 •

edited