Airbyte-ci: Add ClickPipelineContext and update test to use #31628

bnchrch · 2023-10-19T21:36:06Z

Overview

Adds the ClickPipelineContext to give developers a primitive to merge click commands, dagger clients and pipeline contexts into one
Updates airbyte-ci test to use ClickPipelineContext
Adds new click decorators for automatically attaching click arguments to the click context object so that child commands may use them

vercel · 2023-10-19T21:36:11Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Nov 7, 2023 6:14pm

bnchrch · 2023-10-19T21:52:35Z

airbyte-ci/connectors/pipelines/pipelines/cli/airbyte_ci.py

-@track_command
+@click_track_command
+@click_merge_args_into_context_obj
+@click_append_to_context_object("is_ci", lambda ctx: not ctx.obj["is_local"])


@alafanechere @erohmensing Added a new decorator to standardize appending to the context, and make it closer to argument/option decorators.

basically replaces

ctx.obj["is_local"] = value

Not super attached but felt useful

❓ In the case that setting the parameter is more complicated than just setting a value, would you suggest adding a util method and referencing it here? or doing it in the method itself?

I would potentially be looking into making this accept a coroutine in addition to a callable or value, if it fixes my problems with trying to append something. But that doesn't have to be done here :)

Yup! Would definately recommend a named function instead of a lambda.

Also would welcome a coroutine version! Ill look into updating it to allow that

bnchrch · 2023-10-19T21:53:50Z

airbyte-ci/connectors/pipelines/pipelines/cli/airbyte_ci.py

-@click.pass_context
-@track_command
+@click_track_command
+@click_merge_args_into_context_obj


@alafanechere @erohmensing This is what merges all click arguments/options into the context object by default.

basically another name would be @pass_all_click_args_to_child

I saw this in use before I got down to my comment. This is sexy 🔥

bnchrch · 2023-10-19T21:54:51Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/playground/commands.py

+from pipelines.cli.click_decorators import LazyPassDecorator, click_ignore_unused_kwargs, click_merge_args_into_context_obj
+from pipelines.models.contexts.click_pipeline_context import ClickPipelineContext
+
+pass_pipeline_context: LazyPassDecorator = LazyPassDecorator(ClickPipelineContext)


@alafanechere @erohmensing This is the new kid on the block for instantiating a pipeline/dagger client

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/playground/commands.py

airbyte-ci/connectors/pipelines/pipelines/models/singleton.py

airbyte-ci/connectors/pipelines/pipelines/models/contexts/click_pipeline_context.py

airbyte-ci/connectors/pipelines/pipelines/cli/airbyte_ci.py

…line-context

alafanechere

I love the direction! You're becoming a decorator wizard!
My main concerns are the following:

Some command options end up in the command function signature, some others don't. I'd love to have a strict command signature policy. I'd be fine if all the options ended up being merged in the context obj.
You wrote Adds the ClickPipelineContext to give developers a primitive to merge click commands, dagger clients and pipeline contexts into one . I don't see how click context gets merged with the current PipelieContext class. So far the benefits I've grasped are:

less boilerplate code to load option values to the context.
A singleton dagger client

Let me know if I missed something!

alafanechere · 2023-10-25T20:41:35Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/commands.py

+@click_merge_args_into_context_obj
+@click_ignore_unused_kwargs


I find it awkward that some options end up being part of the function signature and others don't.
Can we "simply" remove all function args except ctx and is ctx access in the body of the function?

Yup! can do!

alafanechere · 2023-10-25T20:45:24Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/test/commands.py

+from pipelines.helpers.utils import sh_dash_c
+from pipelines.models.contexts.click_pipeline_context import ClickPipelineContext
+
+pass_pipeline_context: LazyPassDecorator = LazyPassDecorator(ClickPipelineContext)


Could we pass_pipeline_context click_pipeline_context module?

Im not sure I follow?

Sorry, missings words.
Can we from pipelines.models.contexts.click_pipeline_context import pass_pipeline_context

+1, since pretty much everywhere will be using this

alafanechere · 2023-10-25T20:48:08Z

airbyte-ci/connectors/pipelines/pipelines/cli/airbyte_ci.py

+    main_logger.info(f"Modified Files: {ctx.obj['modified_files']}")
+
+
+def _get_gha_workflow_run_id(ctx: click.Context) -> Optional[str]:


Do we really need such a getter? Isn't ctx.obj.get("gha_workflow_run_id") sufficient?

alafanechere · 2023-10-25T20:48:44Z

airbyte-ci/connectors/pipelines/pipelines/cli/airbyte_ci.py

+    return ctx.obj["gha_workflow_run_id"]
+
+
+def _get_pull_request(ctx: click.Context):


A type hint on the return would be nice

alafanechere · 2023-10-25T20:49:42Z

airbyte-ci/connectors/pipelines/pipelines/cli/airbyte_ci.py

-@track_command
+@click_track_command
+@click_merge_args_into_context_obj
+@click_append_to_context_object("is_ci", lambda ctx: not ctx.obj["is_local"])


alafanechere · 2023-10-25T21:02:57Z

airbyte-ci/connectors/pipelines/pipelines/models/contexts/click_pipeline_context.py

+    class Config:
+        arbitrary_types_allowed = True


Why are we declaring a nested class here?

This class was lifted from Aircmd with minor modifications. Didnt want to mess with an implementation that was working

airbyte-ci/connectors/pipelines/pipelines/models/contexts/click_pipeline_context.py

airbyte-ci/connectors/pipelines/pipelines/stolen/__init__.py

airbyte-ci/connectors/pipelines/pipelines/models/contexts/click_pipeline_context.py

alafanechere · 2023-11-02T15:19:17Z

Some command options end up in the command function signature, some others don't. I'd love to have a strict command signature policy. I'd be fine if all the options ended up being merged in the context obj.

@alafanechere I dont think this is achievable without extending / wrapping @click.command Which I think is a bit too big of a lift to restrict use.

Even if we don't restrict use, can we make sure that we follow this policy? In other words, in this PR can you avoid passing the options to the signature and just pull their values from the context? Or is it something click forces you to do?

You wrote Adds the ClickPipelineContext to give developers a primitive to merge click commands, dagger clients and pipeline contexts into one . I don't see how click context gets merged with the current PipelineContext class.

Ah, test may not be the best example but if you look here

airbyte/airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/test/commands.py

Line 20 in 24297df

pipeline_context: ClickPipelineContext,
@click.command()
@click.argument("poetry_package_path")
@click.option("--test-directory", default="tests", help="The directory containing the tests to run.")
@pass_pipeline_context
@click_ignore_unused_kwargs
async def test(
    pipeline_context: ClickPipelineContext,
    poetry_package_path: str,
    test_directory: str,
):
pipeline_context is the only context object necessary.

You can get the instantiated dagger client via pipeline_context.get_dagger_client()

You can get the click context via pipeline_context._click_context

You can access click params and object via pipeline_context.params

@bnchrch yep it's nice.
My question was rather "how easy is it now to avoid instantiating ConnectorContext with such a bulky approach?"

…line-context

alafanechere

Nothing blocking! Thank you for the effort!
Please run a pre-release + connector test manually before merging 🙏

alafanechere · 2023-11-07T13:23:08Z

airbyte-ci/connectors/pipelines/pipelines/models/contexts/click_pipeline_context.py

+
+    _dagger_client_lock: anyio.Lock = PrivateAttr(default_factory=anyio.Lock)
+
+    async def get_dagger_client(self, client: Optional[Client] = None, pipeline_name: Optional[str] = None) -> Client:


I'd love to have a public PipelineContext.dagger_client attribute / property.
It will be more convenient than running await pipeline_context.get_dagger_client()
Could we expose an async init class method to initialize PipelineContext where the _dagger_client attribute would be set?

pipeline_context = await PipelineContext.init(data) pipeline_context.dagger_client

The benefit of get_dagger_client is that you can pass custom pipeline name though.
Let me know your opinion about this.

Closing the loop:

I left a reply in slack on this but TLDR I dont think the gains of avoiding the async call here are worth it.

The Reply:

So If we cant await in the constructor

Then we would have to explicitly call
pipeline_context = await PipelineContext.init(data)
somewhere in the command hierarchy

which has a few downsides (as far as im understanding it)

extra boilerplate

an extra thing to remember to run

that extra thing may be called well before the area of the code your working on (i.e. in the group and your working on a subcommand) leaving you unaware of its existence or necessity

and an extra piece to consider when chaining together subcommands

The upside is we dont use await

Im not sure thats worth the speed trade off

Though Im not sure the speed price were paying

alafanechere · 2023-11-07T13:24:21Z

airbyte-ci/connectors/pipelines/pipelines/models/contexts/click_pipeline_context.py

+
+    _dagger_client_lock: anyio.Lock = PrivateAttr(default_factory=anyio.Lock)
+
+    async def get_dagger_client(self, client: Optional[Client] = None, pipeline_name: Optional[str] = None) -> Client:


Suggested change

async def get_dagger_client(self, client: Optional[Client] = None, pipeline_name: Optional[str] = None) -> Client:

async def get_dagger_client(self, pipeline_name: Optional[str] = None) -> Client:

I don't think this client parameter is used in your function.

…line-context

bnchrch · 2023-11-07T21:11:41Z

Tests are passing
Connector Prerelease was successful
Connector test was successful

bnchrch added 8 commits October 19, 2023 14:24

Add lazy decorator

a4994f4

Add playground subcommands

ccddd26

steal

96643f1

Show demo of context coming through

efa0786

Add pass_click_context_and_args_to_children

76458dc

Add click decorator to reduce boilerplate

58845d3

Add click_append_to_context_object decorator

435f44d

Move files

55baf8f

bnchrch requested review from alafanechere and erohmensing October 19, 2023 21:36

Automated Commit - Formatting Changes

8bfe2aa

bnchrch commented Oct 19, 2023

View reviewed changes

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/playground/commands.py Outdated Show resolved Hide resolved

c-p-b reviewed Oct 20, 2023

View reviewed changes

airbyte-ci/connectors/pipelines/pipelines/models/singleton.py Show resolved Hide resolved

c-p-b reviewed Oct 20, 2023

View reviewed changes

airbyte-ci/connectors/pipelines/pipelines/models/contexts/click_pipeline_context.py Show resolved Hide resolved

c-p-b reviewed Oct 20, 2023

View reviewed changes

airbyte-ci/connectors/pipelines/pipelines/models/contexts/click_pipeline_context.py Outdated Show resolved Hide resolved

alafanechere reviewed Oct 24, 2023

View reviewed changes

airbyte-ci/connectors/pipelines/pipelines/cli/airbyte_ci.py Show resolved Hide resolved

bnchrch added 4 commits October 24, 2023 11:38

Merge remote-tracking branch 'origin/master' into bnchrch/simple-pipe…

a9b7365

…line-context

Bring up to date with async click

27043ae

Replace test with playground impl

c53310a

Add singleton test

b07da00

bnchrch requested review from c-p-b and alafanechere October 24, 2023 19:29

bnchrch marked this pull request as ready for review October 24, 2023 19:29

bnchrch requested a review from a team October 24, 2023 19:29

bnchrch changed the title ~~Draft: ClickPipelineContext~~ Airbyte-ci: Add ClickPipelineContext and update test to use Oct 24, 2023

alafanechere requested changes Oct 25, 2023

View reviewed changes

bnchrch and others added 2 commits November 2, 2023 15:31

Automated Commit - Formatting Changes

7ca02fc

Run license check on repo (#32154)

bcee91f

octavia-approvington requested a review from a team as a code owner November 6, 2023 16:13

Merge remote-tracking branch 'origin/master' into bnchrch/simple-pipe…

77858b7

…line-context

octavia-squidington-iv requested a review from a team November 7, 2023 01:50

vercel bot deployed to Preview November 7, 2023 01:53 View deployment

bnchrch added 3 commits November 6, 2023 18:53

Use ctx params instead of args

fd5e49c

Merge remote-tracking branch 'origin/master' into bnchrch/simple-pipe…

38521a1

…line-context

Revert additional formating

c290dd8

octavia-squidington-iii removed area/connectors Connector related issues CDK Connector Development Kit labels Nov 7, 2023

bnchrch and others added 3 commits November 7, 2023 02:08

Automated Commit - Formatting Changes

3dc9853

Fix tests

1aeb577

Remove comment

cf50807

bnchrch requested a review from alafanechere November 7, 2023 02:43

alafanechere approved these changes Nov 7, 2023

View reviewed changes

bnchrch added 2 commits November 7, 2023 11:08

Merge remote-tracking branch 'origin/master' into bnchrch/simple-pipe…

755d02a

…line-context

Remove unused command

7ff56fb

bnchrch merged commit 22fc00a into master Nov 7, 2023
21 checks passed

bnchrch deleted the bnchrch/simple-pipeline-context branch November 7, 2023 21:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Airbyte-ci: Add ClickPipelineContext and update test to use #31628

Airbyte-ci: Add ClickPipelineContext and update test to use #31628

bnchrch commented Oct 19, 2023 •

edited

Loading

vercel bot commented Oct 19, 2023 •

edited

Loading

bnchrch Oct 19, 2023

alafanechere Oct 25, 2023

erohmensing Nov 1, 2023

bnchrch Nov 1, 2023

bnchrch Oct 19, 2023

erohmensing Oct 24, 2023

bnchrch Oct 19, 2023

alafanechere left a comment

alafanechere Oct 25, 2023

bnchrch Oct 31, 2023

alafanechere Oct 25, 2023

bnchrch Oct 31, 2023

alafanechere Oct 31, 2023

erohmensing Oct 31, 2023

alafanechere Oct 25, 2023

alafanechere Oct 25, 2023

alafanechere Oct 25, 2023

alafanechere Oct 25, 2023

bnchrch Oct 31, 2023

alafanechere commented Nov 2, 2023

alafanechere left a comment

alafanechere Nov 7, 2023

bnchrch Nov 7, 2023

alafanechere Nov 7, 2023

bnchrch commented Nov 7, 2023 •

edited

Loading

		@click_merge_args_into_context_obj
		@click_ignore_unused_kwargs

		main_logger.info(f"Modified Files: {ctx.obj['modified_files']}")


		def _get_gha_workflow_run_id(ctx: click.Context) -> Optional[str]:

		return ctx.obj["gha_workflow_run_id"]


		def _get_pull_request(ctx: click.Context):


		_dagger_client_lock: anyio.Lock = PrivateAttr(default_factory=anyio.Lock)

		async def get_dagger_client(self, client: Optional[Client] = None, pipeline_name: Optional[str] = None) -> Client:

	async def get_dagger_client(self, client: Optional[Client] = None, pipeline_name: Optional[str] = None) -> Client:
	async def get_dagger_client(self, pipeline_name: Optional[str] = None) -> Client:

Airbyte-ci: Add ClickPipelineContext and update test to use #31628

Airbyte-ci: Add ClickPipelineContext and update test to use #31628

Conversation

bnchrch commented Oct 19, 2023 • edited Loading

Overview

vercel bot commented Oct 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alafanechere left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alafanechere commented Nov 2, 2023

alafanechere left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bnchrch commented Nov 7, 2023 • edited Loading

bnchrch commented Oct 19, 2023 •

edited

Loading

vercel bot commented Oct 19, 2023 •

edited

Loading

bnchrch commented Nov 7, 2023 •

edited

Loading