DockerOperator SSL Verification

[krutoileshii]

Apache Airflow Provider(s)

docker

Versions of Apache Airflow Providers

Trying to authenticate with the DockerOpeartor using airflow connection to an internal registry and getting the TLS certificate verification failure. I am able to login just fine into the registry from command line with no issues.

Insecure registry is set up as well already. Not sure how to bypass this. I do not have access to the certificate for the server. Any help would be appreciated.

[2025-08-08, 13:03:52] INFO - Connection Retrieved 'docker-connection': source="airflow.hooks.base"
[2025-08-08, 13:03:52] INFO - Login into Docker Registry: https://allo-asp03:6278/: source="airflow.task.hooks.airflow.providers.docker.hooks.docker.DockerHook"
[2025-08-08, 13:03:52] ERROR - Login failed: source="airflow.task.hooks.airflow.providers.docker.hooks.docker.DockerHook"
[2025-08-08, 13:03:52] ERROR - Task failed with exception: source="task"
APIError: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("login attempt to https://allo-asp03:6278 failed with status: authenticating creds for "allo-asp03:6278": pinging container registry allo-asp03:6278: Get "https://allo-asp03:6278/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority")
File "/virt_envs/.airflow_venv/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 877 in run

File "/virt_envs/.airflow_venv/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1164 in _execute_task

File "/virt_envs/.airflow_venv/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 397 in wrapper

File "/virt_envs/.airflow_venv/lib/python3.12/site-packages/airflow/providers/docker/operators/docker.py", line 482 in execute

File "/virt_envs/.airflow_venv/lib/python3.12/site-packages/airflow/providers/docker/operators/docker.py", line 356 in cli

File "/usr/local/lib/python3.12/functools.py", line 998 in __get__

File "/virt_envs/.airflow_venv/lib/python3.12/site-packages/airflow/providers/docker/hooks/docker.py", line 157 in api_client

File "/virt_envs/.airflow_venv/lib/python3.12/site-packages/airflow/providers/docker/hooks/docker.py", line 198 in __login

File "/virt_envs/.airflow_venv/lib/python3.12/site-packages/docker/api/daemon.py", line 152 in login

File "/virt_envs/.airflow_venv/lib/python3.12/site-packages/docker/api/client.py", line 281 in _result

File "/virt_envs/.airflow_venv/lib/python3.12/site-packages/docker/api/client.py", line 277 in _raise_for_status

File "/virt_envs/.airflow_venv/lib/python3.12/site-packages/docker/errors.py", line 39 in create_api_error_from_http_exception

HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/auth
File "/virt_envs/.airflow_venv/lib/python3.12/site-packages/docker/api/client.py", line 275 in _raise_for_status

File "/virt_envs/.airflow_venv/lib/python3.12/site-packages/requests/models.py", line 1026 in raise_for_status

Apache Airflow version

3.0.3

Operating System

Ubuntu WSL2

Deployment

Virtualenv installation

Deployment details

podman version 4.3.1

What happened

I would expect airflow to honor the insecure registries.

What you think should happen instead

Need an ability to bypass this verification.

How to reproduce

Have private registry with insecure certs
Add docker_connection
Add proper insecure registry configs
Have a dag that pulls from that registry.
Receive error

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

I converted it into a discussion. It looks like you have an issue with self-signed certificates for your infrastructure. That's what usually "failed to verify certificate: x509: certificate signed by unknown authority" unless you have some misconfiguration of your infrastructure and for example remove all "built-in" certificates that are coming either from system of yours or Python.

It's not really an "airflow" issue - it's more how you are configure SSL certificates used by Python packages to connect to the servers - you seem to have self-signed certificate or certificate signed by authority that is not in the "regular" certificates that Python uses. There are quite a few issues here and you can google it as well on how to make sure that Python certificates contain your self-signed certificates (look for self-signed here in the issues - you will likely find a few things you can try in your deployment - also if you ask for configuring self-signed certificates to be used by Python in Google.

Depending on Python version and operting system - there might be different solutions and different ways as far as I know.

Generally also better solution is to pay for a good certificate that is widely recognized by defaullt certification checks built-in. Paying for certificates or even using let's encrypt quickly rotating certificates on your server is a good approach. Especially if you consider thaat you are now trying to offload that to volunteers working in their free time mostly - developing software that you can use for free. It would seem resonable that your company can at least have some good certificates, or maybe instead find a way how to make their infrastructure works with such self-signed certificates for Python and all other tools.

[gopidesupavan]

Yes indeed it looks ceritificate issue.

@krutoileshii did you read through the documentation?
https://airflow.apache.org/docs/apache-airflow-providers-docker/stable/_api/airflow/providers/docker/hooks/docker/index.html
http://airflow.apache.org/docs/apache-airflow-providers-docker/stable/decorators/docker.html#parameters

This clearly gives option to provide your certificates.

[krutoileshii]

yes I've read through those, but don't have access to certs at the moment. hence was looking for a way to bypass. should still be possible since one might be using someone else's registry that has a self signed cert.

[potiuk]

Then no. Certificates are security. Your production setting is following your environment security - so if self-signed certificates are used, you need to make sure the environment is configured to handle it. It's Deployment Problem really that you should solve - in many cases bypassing security is a "securty issue" on it's own.

And "bypassing security" is pretty much never a good solution. There are some ways that makes sense (for example installing your certificate in your system). Which is a kind of bypass - and I recommended you to look for those options in general deployment docs.

And yes - you might implement this is if you find a way - but I think asking maintainers on how to bypass security is a bad idea. They will generally answer you what I did - if you want to do it, do it on your own and find a way - but we will not advise you to do it, because we don't want to take responsibiity for bad choices of yours.