-
Notifications
You must be signed in to change notification settings - Fork 13.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
make docker operators always use DockerHook
for API calls
#28363
Conversation
DockerHook
for API callsDockerHook
for API calls
BTW, may be someone know why docker-provider has min airflow version 2.4? |
b3408be
to
5d1401f
Compare
Because of ExternalPythonOperator #25780 |
I can't find any information why it should affect docker provider but maybe I blind Also we have this lines, which make sense if we support 2.3+ and look redundant if 2.4+ airflow/airflow/providers/docker/decorators/docker.py Lines 32 to 37 in 4e545c6
|
5d1401f
to
bb77b02
Compare
It looks like it was a mistake. We should get it down to 2.3.0 in the next release: We already have a code added in the same PR that handles pre-2.4 case - https://github.com/apache/airflow/pull/25780/files#diff-1042a900e2f4c7b044453f7229033b8787d35abe66b14d0e9392fee8b1a714ffR30 So >=2.3.0 should be fine. |
I think it was just failing with 2.3 and I first bumped it to 2.4 and then added the if and forgot to move it down to 2.3.0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very nice refactor.
However I have one request - could we also handle tls parameter In Hook possible to be passed as string for backwards compatibility? Seems that it should be very easy to implement and not that unreasonable even (Hook will be then a bit more flexible on what cold be passed to tls. And we will avoid having to bump major version of the provider? Seems like all other changes are backwards compatible.
@potiuk, I will have a look again (today last day of my holidays) however based on docker-py code pass Internally we do not check type of this argument (if not count mypy), so if user pass string and this somehow work it will work. |
Ah. Then it makes perfect sense - it's really a bugfix not a backwards-compatibility probiem then. |
Those tests are running various tests with Providers Manager and could catch errors introduced by provider-only changes so they should be added to "Always" tests to be "always" run in our CI. Skipping this test was the main reason why apache#28700 change was needed after the apache#28363 was merged.
maybe somewhere in the past tls_config could be a string (I didn't check) but the main issue that Hook previously do not use in DockerOperator and others and even when it use then tls_config constructed from multiple separate parameters. |
Right now
DockerHook
only partially uses inDockerOperator
andDockerSwarmOperator
.This PR refactor some parts
__init__
, e.g. obtain connection.docker_conn_id
now optional. If user not required login to docker Registry this field might be set to None.Previously this functional sits in
DockerOperator
DockerOperator
toDockerHook
This kind of integration will allow in the future move all complex logic from Operators to Hook and make them reusable.
Also this PR pre-requirements for #26162