Skip to content

Serve logs from scheduler if any executor is LocalExecutor - #71283

Open
topherinternational wants to merge 1 commit into
apache:mainfrom
topherinternational:scheduler-command-serve-logs-local
Open

Serve logs from scheduler if any executor is LocalExecutor#71283
topherinternational wants to merge 1 commit into
apache:mainfrom
topherinternational:scheduler-command-serve-logs-local

Conversation

@topherinternational

@topherinternational topherinternational commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

This change causes the scheduler to start the local log server if the LocalExecutor is anywhere in the list of configured executors; previously, scheduler would start the log server only if LocalExecutor was the default (first) executor.

In my org, we set up multiple executors on Airflow 2 with the core.executor config set to KubernetesExecutor,LocalExecutor. We put the KubernetesExecutor first as a fail-safe - in the event a task runs without specifying an executor, we want it to run in an isolated pod so it doesn't threaten resources on the scheduler or its locally-running tasks.

After we upgraded to Airflow 3, we noticed:

  • Task logs would not display in the UI for running local tasks (logs would display as expected for finished local tasks, and running and finished k8s tasks)
  • Logs in the api-server would show a corresponding Connection refused error trying to connect to the scheduler on port 8793
  • The scheduler was not actually listening on 8793, i.e. the local log server had not been started

I examined the code and found that scheduler_command launches the log server only if LocalExecutor is the default executor (i.e., the first executor listed in the configuration).

This PR solves the issue by examining all configured executor types instead of only the default when deciding whether to launch the log server.

I've added two test cases to the existing parameterized TestSchedulerCommand.test_serve_logs_on_scheduler() method:

  • "LocalExecutor,KubernetesExecutor" passes before and after this fix, showing that the log server is launched when the local executor is the default in a multiple-executor setup
  • "KubernetesExecutor,LocalExecutor" fails before this fix and passes after it, showing that the log server is launched when the local executor is non-default in a multiple-executor setup

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

@eladkal eladkal added this to the Airflow 3.3.2 milestone Aug 8, 2026
@eladkal eladkal added type:bug-fix Changelog: Bug Fixes backport-to-v3-3-test Backport to v3-3-test labels Aug 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:CLI backport-to-v3-3-test Backport to v3-3-test type:bug-fix Changelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants