Skip to content

Add OpensearchRemoteLogIO.from_config and register opensearch scheme#70295

Open
FrankYang0529 wants to merge 1 commit into
apache:mainfrom
FrankYang0529:airflow-70272
Open

Add OpensearchRemoteLogIO.from_config and register opensearch scheme#70295
FrankYang0529 wants to merge 1 commit into
apache:mainfrom
FrankYang0529:airflow-70272

Conversation

@FrankYang0529

@FrankYang0529 FrankYang0529 commented Jul 23, 2026

Copy link
Copy Markdown
Member

Why

#67056 decoupled remote logging from the hardcoded branches in airflow_local_settings.py: core and the Task SDK now resolve the handler via ProvidersManager dispatch on the [logging] remote_base_log_folder URL scheme. This migrates the opensearch scheme.

How

  • Add OpensearchRemoteLogIO.from_config() reading the [opensearch] section — a mirror of the legacy branch, so the user-facing config surface is unchanged.
  • Register the opensearch scheme in provider.yaml.

Verify

  • uv run --project providers/opensearch pytest providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py
  • end to end test
# 1. Create config
mkdir -p files/airflow-breeze-config
cat > files/airflow-breeze-config/environment_variables.env <<'EOF'
export AIRFLOW__LOGGING__REMOTE_LOGGING=True
export AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER=opensearch://
export AIRFLOW__OPENSEARCH__HOST=http://opensearch:9200
export AIRFLOW__OPENSEARCH__PORT=9200
export AIRFLOW__OPENSEARCH__WRITE_TO_OS=True
export AIRFLOW__OPENSEARCH__JSON_FORMAT=True
export AIRFLOW__OPENSEARCH__WRITE_STDOUT=False
export AIRFLOW__OPENSEARCH__TARGET_INDEX=airflow-logs
EOF

# 2. Start Airflow
breeze start-airflow --integration opensearch --backend postgres --load-example-dags

# 3. Run Dag
airflow dags unpause example_bash_operator
airflow dags trigger example_bash_operator
airflow dags list-runs example_bash_operator

# 4. Check log
curl -s localhost:9200/airflow-logs/_count ; echo
curl -s 'localhost:9200/airflow-logs/_search?q=log_id:example_bash_operator&size=3'

Was generative AI tooling used to co-author this PR?
  • Yes - Claude Code

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Signed-off-by: PoAn Yang <payang@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add OpensearchRemoteLogIO.from_config and register opensearch scheme

1 participant