airbyte-ci: ClickPipelineContext handles dagger logging #33419
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
|
Current dependencies on/for this PR: This stack of pull requests is managed by Graphite. |
alafanechere
force-pushed
the
augustin/12-13-airbyte-ci_ClickPipelineContext_handles_dagger_logging
branch
2 times, most recently
from
6ddfdf5
to
68e0adc
Compare
| if self.params.get("show_dagger_logs", False): | ||
| log_output = sys.stdout | ||
| else: | ||
| log_output, self._click_context().obj["dagger_logs_path"] = self._create_dagger_client_log_file() |
There was a problem hiding this comment.
Nice change!
One house keeping thing
1. Move this logic to its own method
def get_log_output(self):
if self.params.get("show_dagger_logs", False):
log_output = sys.stdout
else:
log_output, self._click_context().obj["dagger_logs_path"] = self._create_dagger_client_log_file()
return log_output
| @@ -97,6 +103,14 @@ async def get_dagger_client(self, pipeline_name: Optional[str] = None, log_outpu | |||
| assert self._dagger_client, "Error initializing Dagger client" | |||
| return self._dagger_client.pipeline(pipeline_name) if pipeline_name else self._dagger_client | |||
|
|
|||
| def _create_dagger_client_log_file(self) -> Tuple[io.FileIO, str]: | |||
There was a problem hiding this comment.
❓ Instead of adding this logic here, could we not hoist it into airbyte_ci.py or dagger_run.py and make it shared between all pipelines?
There was a problem hiding this comment.
It's something I considered but as the log file is tightly coupled to the client I prefer to keep it where the client is instantiated.
I'm also conscious that we currently have inconsistencies in the way we handle dagger client logging in our commands.
I created this issue to consolidate the approach.
TLDR; all our commands should get their dagger client from ClickPipelineContext
There was a problem hiding this comment.
That sounds fine then!
At that point in time we should be in a good place to introduce the GlobalSettings pattern from aircmd,
alafanechere
force-pushed
the
augustin/12-13-airbyte-ci_ClickPipelineContext_handles_dagger_logging
branch
2 times, most recently
from
34beb12
to
3e1ad9e
Compare
| @@ -97,6 +103,14 @@ async def get_dagger_client(self, pipeline_name: Optional[str] = None, log_outpu | |||
| assert self._dagger_client, "Error initializing Dagger client" | |||
| return self._dagger_client.pipeline(pipeline_name) if pipeline_name else self._dagger_client | |||
|
|
|||
| def _create_dagger_client_log_file(self) -> Tuple[io.FileIO, str]: | |||
There was a problem hiding this comment.
That sounds fine then!
At that point in time we should be in a good place to introduce the GlobalSettings pattern from aircmd,
alafanechere
force-pushed
the
augustin/12-13-airbyte-ci_ClickPipelineContext_handles_dagger_logging
branch
from
3e1ad9e
to
fe819b4
Compare
alafanechere
deleted the
augustin/12-13-airbyte-ci_ClickPipelineContext_handles_dagger_logging
branch
Suspect IssuesThis pull request was deployed and Sentry observed the following issues:
Did you find this useful? React with a 👍 or 👎 |
What
ClickPipelineContextusers rely on theget_dagger_clientmethod to get the dagger client as a singleton.We originally exposed a
log_outputparameter to this method, but as it returns a singleton dagger client, the passedlog_outputis not taken into account if the singleton already exists.How
The
get_dagger_clientdefines the log output path according to theshow_dagger_logsparams.If
show_dagger_logsis true: show the logs in the stdout, otherwise log to a file.Next:
show_dagger_logsparam/options and always log to a filedagger runwrapped commandClickPipelineContextto have a consistent logging management implementation