Skip to content

Do not forward an empty ca_certs to the OpenSearch log client - #72552

Open
FrankYang0529 wants to merge 1 commit into
apache:mainfrom
FrankYang0529:airflow-opensearch-skip-empty-ca-certs
Open

Do not forward an empty ca_certs to the OpenSearch log client#72552
FrankYang0529 wants to merge 1 commit into
apache:mainfrom
FrankYang0529:airflow-opensearch-skip-empty-ca-certs

Conversation

@FrankYang0529

@FrankYang0529 FrankYang0529 commented Sep 5, 2026

Copy link
Copy Markdown
Member

Why

  • [opensearch_configs] ca_certs defaults to an empty string in provider.yaml, and get_os_kwargs_from_config() forwards the whole section to OpenSearch(...).
  • opensearch-py only falls back to certifi when ca_certs is absent. With use_ssl and verify_certs enabled, an empty string makes it raise ImproperlyConfigured("Root certificates are missing ...").

How

  • get_os_kwargs_from_config() drops ca_certs when it is empty, so opensearch-py falls back to certifi. It still forwards a configured path unchanged.

Verification

  • Unit test: uv run --project opensearch pytest providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py
  • Integration test:
  1. Setup
export AIRFLOW_HOME=/tmp/airflow-test-opensearch
mkdir -p $AIRFLOW_HOME
  1. Start opensearch
docker run -d --name os-b0047 -p 9200:9200 -e discovery.type=single-node -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=Str0ng-Pass" -e "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" opensearchproject/opensearch:2
ngrok http https://localhost:9200
  1. Add airflow.cfg
cat > $AIRFLOW_HOME/airflow.cfg <<'EOF'
[core]
load_examples = False

[logging]
remote_logging = True
delete_local_logs = False

[opensearch]
host = https://<host>
port = 443
username = admin
password = B0047-Str0ng!Pass
write_to_os = True
json_format = True
write_stdout = False

[opensearch_configs]
http_compress = False
use_ssl = True
verify_certs = True
ssl_assert_hostname = False
ssl_show_warn = False
ca_certs =

[api_auth]
jwt_secret = demo-test
EOF
  1. Run airflow version

On main branch, it shows

ImportError: Unable to load logging config from airflow.config_templates.airflow_local_settings.DEFAULT_LOGGING_CONFIG due to: ImproperlyConfigured:Root certificates are missing for certificate validation. Either pass them in using the ca_certs parameter or install certifi to use it automatically.

In this branch, it can show the version without error.


Was generative AI tooling used to co-author this PR?
  • Yes - Claude Code

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Signed-off-by: PoAn Yang <payang@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant