Skip to content

Apache Airflow 3.3.0

Latest

Choose a tag to compare

@vatsrahul1001 vatsrahul1001 released this 06 Jul 15:16

📦 PyPI: https://pypi.org/project/apache-airflow/3.3.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.3.0/
🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.3.0/release_notes.html
🐳 Docker Image: "docker pull apache/airflow:3.3.0"
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.3.0

Significant Changes

Asset Partitioning (#64571, #65447, #66030, #66848, #67184, #67475, #67716, #68978)

Building on the asset partitioning introduced in 3.2.0, Airflow 3.3.0 substantially expands how a
single upstream asset event fans out to partitioned downstream Dag runs. New partition mappers —
RollupMapper (many-to-one), FanOutMapper (one-to-many), and FixedKeyMapper +
SegmentWindow (categorical rollup) — compose with time windows (day/week/month/quarter/year) and a
wait_policy (WaitForAll or MinimumCount(n)) to control when partitioned runs fire. Windows
can fan out forward or backward in time, and total fan-out per upstream event is bounded by the new
[scheduler] partition_mapper_max_downstream_keys config (configurable per mapper). Airflow 3.3.0
also adds the PartitionedAtRuntime timetable, which lets a Dag declare that its partition key(s)
are assigned when the run starts rather than mapped from an upstream event.

For detailed usage instructions, see :doc:/authoring-and-scheduling/assets.

Task and Asset State Store (#65759, #66073, #66160, #66463, #66586, #66859, #67041, #67292, #67319)

Airflow 3.3.0 introduces a first-class state store for tasks and assets (AIP-103). Tasks can persist
arbitrary key-value state that survives across retries and runs via a new task_state_store accessor, and
assets can carry their own state via asset_state_store — both available from the Task SDK. State is kept
in the metadata database by default, or in a custom worker-side backend ([workers] state_store_backend), supports per-key retention with periodic garbage collection and an optional
clear_on_success, and is fully manageable through the Core API and Execution API.

For detailed usage instructions, see :doc:/core-concepts/task-and-asset-state-store.

Pluggable Retry Policies (#65474)

Task retry behaviour is now pluggable (AIP-105). In addition to a fixed retries count, you can
attach a custom retry policy that decides whether and when a task is retried, enabling strategies such
as retrying only on specific exceptions or backing off based on custom logic.

For detailed usage instructions, see :ref:concepts:retry-policies.

Language Task SDK (Java and Go) (#65958, #67161, #67635, #67699)

Airflow 3.3.0 adds a Coordinator layer (AIP-108) that lets individual task implementations be written
in non-Python languages while the Dag and its scheduling stay in Python. A task is declared in the Dag
with @task.stub(queue=...); the worker routes it to a configured coordinator
(JavaCoordinator for JVM languages, ExecutableCoordinator for self-contained native binaries
such as Go) that runs the task in a language runtime and proxies Variables, Connections, and XComs back
through the Execution API.

.. warning::

The Coordinator layer and the Java/Go SDKs are experimental in 3.3.0 and may change in future
versions based on user feedback.

For detailed usage instructions, see :doc:/authoring-and-scheduling/language-sdks/index.

Dag bundle version on clear, rerun, and backfill (#63884)

The new rerun_with_latest_version setting controls whether a cleared, rerun, or backfilled Dag run
uses the latest bundle version or the original version from the initial run. The default is resolved
by precedence: an explicit request parameter/CLI flag, then the Dag-level rerun_with_latest_version,
then [core] rerun_with_latest_version, and finally False for clear/rerun and True for
backfills (preserving historical behaviour). Airflow 2.x always reran with the latest code; 3.x
introduced bundle versioning defaulting to the original version, and this setting gives users control.

See :doc:/administration-and-deployment/dag-bundles for full details.

Provider example Dags as dedicated bundles (#66161)

Example Dags shipped by provider distributions are now discovered via ProvidersManager and
registered as their own Dag bundles — one per provider, named
apache-airflow-providers-<distribution>-example-dags (or <distribution>-example-dags for
third-party providers). The [core] load_examples option still gates whether they are registered.
REST API clients that filtered bundle_name by "dags-folder" for provider-shipped example Dags
must update to the new per-provider bundle names; Dag identifiers are unchanged.

Remote logging resolution decoupled from airflow.logging_config (#67056)

Remote task log handler resolution is now owned by the shared airflow_shared.logging.factory
module and applies a single, well-defined precedence:

  1. a user-defined [logging] logging_config_class exporting REMOTE_TASK_LOG /
    DEFAULT_REMOTE_CONN_ID (existing custom configs keep working);
  2. ProvidersManager scheme dispatch — the scheme of [logging] remote_base_log_folder selects a
    provider RemoteLogIO class, instantiated via a no-argument from_config() classmethod;
  3. a transitional legacy fallback reading airflow_local_settings.py (to be removed in Airflow 4.0).

airflow.logging_config.load_logging_config is deprecated (it now emits DeprecationWarning and
delegates to new private helpers), and configure_logging no longer eagerly resolves the remote
handler — resolution is lazy on first use. Providers that registered a remote-logging: block but do
not implement from_config are skipped with a warning and fall through to the legacy path.

Migration: replace direct calls to airflow.logging_config.load_logging_config() with the new
helpers, and have provider remote-log handler classes implement a no-argument from_config
classmethod that reads airflow.providers.common.compat.sdk.conf.

OpenTelemetry timer metrics now use Histogram (#64207)

OpenTelemetry timer and timing metrics are now recorded as Histograms instead of Gauges, preserving
count, sum, and bucket distribution across recordings.

Dag-processing "seconds ago" metric is now tagged (#62487)

dag_processing.last_run.seconds_ago.{dag_file} is now a legacy metric. The new
dag_processing.last_run.seconds_ago is emitted with file_path, bundle_name and
file_name tags (file_path + bundle_name uniquely identify the Dag file). The legacy metric
is still emitted by default and can be disabled via [metrics] legacy_names_on.

New Deadlines page under Browse (#67586)

A new Deadlines page is available under the Browse menu, accessible to any role that already has
can_read and menu_access on Dag Runs.

New Features

  • Add partition clear support to the REST API matching the CLI, with a clearPartitions endpoint and partition_key/partition_date window selectors on clearDagRuns (#68702)
  • Add [core] mp_start_method and [core] mp_forkserver_preload configuration options (which can be overridden per [scheduler]/[triggerer]/[dag_processor]) to control the multiprocessing start method (#68875)
  • Add a durable toggle to ResumableJobMixin to opt out of resumable execution (#68623)
  • Add a @result decorator to mark a TaskFlow task as the Dag's result task (#64563)
  • Add [triggerer] shared_stream_cohort_grace_period to reduce missed events on triggerer restart (#68888)
  • Propagate partition_date from producer Dag runs to consumers of partitioned assets (#67285)
  • Make the task and asset state store accessible from triggers via AssetStateStoreAccessors (#67839)
  • Add OpenTelemetry head sampling support (#68591)
  • Add async XCom accessors for async tasks (#68299)
  • Add an async aget_hook method to BaseHook for async tasks (#68506)
  • Allow custom partition Window subclasses via a plugin registry (#68717)
  • Support an extra field for the Coordinator (#68694)
  • Apply rerun_with_latest_version to TriggerDagRunOperator reruns (#67273)
  • Scope the XCom Execution API to teams in multi-team mode (#68850)
  • Enforce pool team ownership in the scheduling loop (#68649)
  • UI: Add team name to the asset graph view (#68457)
  • Populate partition_date for partitioned Dag runs whose composite asset key has a single time-based dimension (#68442)
  • UI: Add a column to the asset store table linking to the task instance that wrote it (#68395)
  • UI: Add a custom expiration datetime picker for the task store modal (#68394)
  • UI: Add additional task instance attributes to the task instance details section (#68378)
  • UI: Add a Details tab to the mapped task instance view (#68340)
  • UI: Add bulk marking of Dag runs as success or failed from multi-select (#68278)
  • Add --team-name support to the pool CLI commands (#68110)
  • UI: Add a full-screen toggle to the code viewer (#68044)
  • Add API endpoint support for consumer team asset filtering (#68034)
  • UI: Add bulk clear selection for task instances (#68029)
  • Add awaiting_input task state for Human-in-the-Loop, running off the triggerer (#68028)
  • Add bulk API to mark Dag runs as success or failed (#67948)
  • Record writer info for every asset store write for better cross-linkage (#67902)
  • Register XCom output_type classes from a worker-side Dag walk (#67875)
  • Add FixedKeyMapper and SegmentWindow for categorical asset-partition rollup (#67716)
  • Add a bulk POST /dags/{dag_id}/clearDagRuns API endpoint (#67709)
  • Return Pydantic model instances through XCom for structured output (#67644)
  • Add the ability to apply a note when clearing a Dag run or task instances (#67639)
  • Add consumer_teams to AssetAccessControl in the Task SDK (#67625)
  • Populate trigger team_name at creation time for multi-team support (#67605)
  • UI: Add bulk Clear on the Dag Runs list page (#67564)
  • Add multi-team query filtering to triggerer trigger assignment (#67517)
  • Add forward fan-out support via the forward kwarg on Window (AIP-76) (#67475)
  • Add patch task state API and expires_at support in the set API (AIP-103) (#67319)
  • Add a team_name column to the trigger table for multi-team triggerer support (#67305)
  • UI: Add asset and task store views (#67292)
  • Add a --team-name CLI argument to the triggerer for multi-team (#67254)
  • Add an allow_global option to asset access control (#67251)
  • Add mTLS and private CA support to the API client and server (#67214)
  • Add Markdown documentation support for TaskGroups (#67207)
  • Add a per-mapper max_fan_out override for partition fan-out cap (#67184)
  • Add timezone support to the SDK temporal partition mappers (#67164)
  • Add ResumableJobMixin with SparkSubmitOperator for surviving worker failures (#67118)
  • Add bulk delete for Dag runs (#67095)
  • Add a nav_top_level option for plugin nav items (#67084)
  • Add Core API endpoints for task state and asset state (AIP-103) (#67041)
  • Replace allow_producer_teams with access_control on Asset (#66954)
  • Add worker-side custom state backend support (AIP-103) (#66859)
  • Let partitioned Dag runs fire on a partial upstream window with wait_policy (#66848)
  • Consume task-emitted partition keys on asset events (AIP-76) (#66782)
  • Add per-task state key retention from operators (AIP-103) (#66699)
  • Add a callback_execution_timeout config for deadline callbacks (#66609)
  • Add a clear_on_success config to wipe task state on success (AIP-103) (#66586)
  • Add a partitions clear CLI command to reset DagRun partition fields (#66520)
  • Make CORS allow_credentials configurable (#66503)
  • Add periodic task state garbage collection and retention support (AIP-103) (#66463)
  • Add URI sanitizers and asset factories for new schemes (#66426)
  • Add a teams sync CLI command (#66418)
  • Add remote log upload support for callback subprocesses (#66379)
  • Add by-name/by-uri asset state routes and AssetUriRef support (AIP-103) (#66336)
  • UI: Add support for rendering multi-type params (#66278)
  • Filter Dags by teams when registering asset changes (#66168)
  • Wire up Task SDK communication and context access for task/asset state (AIP-103) (#66160)
  • UI: Add marking a task group as success or failed (#66146)
  • Add Execution API endpoints for task and asset states (AIP-103) (#66073)
  • Add FanOutMapper for one-to-many partition fan-out (#66030)
  • Add Variable.keys() to list variable keys by prefix in the Task SDK (#66022)
  • Add an airflow dags clear command for partition-range reprocessing (#66004)
  • Add a memray_detailed_tracing option for deeper memory profiling (#65996)
  • UI: Add support for different graph directions in the asset graph view (#65948)
  • UI: Add a Clear All Mapped Tasks button (#65813)
  • Add allow_producer_teams to the Asset SDK class (#65790)
  • UI: Show expected duration based on historical average in Dag Run details (#65722)
  • Add team name to the task context (#65617)
  • Add an on_kill() hook to BaseTrigger to handle user actions on triggers (#65590)
  • Allow accessing a Dag's members via [] (#65586)
  • Add pluggable retry policies for Airflow tasks (AIP-105) (#65474)
  • Add support for format="Duration" in params (#65469)
  • UI: Add pagination to the grid view (#65388)
  • Add partition_key to the task context (#65359)
  • Make the blocked-thread warning threshold configurable (#65009)
  • Add name fields to SDK deadline alerts (#64926)
  • Add dynamic interval resolution support via Variables for deadline alerts (#64751)
  • Add an is_backfillable property to Dag API responses (#64644)
  • Return dag-specified results in the dag run wait API (#64577)
  • Hold a Dag run until all upstream partitions arrive (AIP-76) (#64571)
  • Add a way to mark a return-value XCom as the dag result (#64522)
  • Allow accessing a TaskGroup's members via [] (#64430)
  • UI: Redo the Gantt chart (#64335)
  • UI: Add task-level filters to the Dag graph tab (#64271)
  • UI: Add bulk Clear, Mark Success/Fail, and delete for multiple task instances (#64141)
  • Check that multi-team is enabled when a team name is provided to the API (#63994)
  • Add a DagRunType for operators (#63733)
  • UI: Add search functionality to the task log viewer (#63467)
  • Add patching of task group instances in the API (#62812)
  • Add deadlines API endpoints (#62583)
  • Add async connection testing via workers for security isolation (#62343)
  • Add run_after to TriggerDagRunOperator (#62259)
  • UI: Display deadlines on the Dag Run and Overview tabs (#62195)
  • Re-enable the start_from_trigger feature with template-field rendering (#55068)
  • Backfill partitioned Dags by partition-date range (#67537)
  • Add a producer-side acknowledgement channel to shared-stream triggers (#67523)
  • UI: Add partition_date to the Dag run detail page (#68977)
  • Expose the upstream partition_key on triggering_asset_events and dag_run.consumed_asset_events (AIP-76) (#69120)
  • Allow get/set/delete/clear of AssetStateStoreAccessor to run on the triggerer (#68966)

Bug Fixes

  • Fix KubernetesExecutor scheduler crash caused by a pod_override that cannot be pickled when running in-cluster (#68831)
  • UI: Fix dashboard alert clamping and collapse controls (#68893)
  • Stabilize mapped-task XCom result ordering in the Dag run wait endpoint by ordering on task_id/map_index (#68550)
  • Only log task state cleanup when a worker state store backend is configured (#68878)
  • Fix in-process Execution API loop stopped while transport still in use (#68865)
  • Fix task state store custom expiry datetime missing timezone on save (#68823)
  • Do not leak threads from InProcessExecutionAPI (#68840)
  • Fix partitioned backfill widening a sub-day window to the whole day (#68718)
  • UI: Fix inconsistent padding between Dag Runs and Task Instances list views (#68689)
  • Skip asset-change registration for tasks with no outlets (#68687)
  • Percent-encode API client path params for keys with slashes (#68667)
  • Fix bulk create+overwrite silently resetting unset fields on pools and connections (#68645)
  • Fix triggerer crash when a trigger subclass does not call super().__init__() (#68636)
  • Fix Task SDK swallowing errors when Variable.set() or Variable.delete() fails (#68542)
  • Populate partition_date when manually triggering partitioned Dags (#68458)
  • Improve warning visibility for invalid JSON when editing variables (#68268)
  • Fix the triggerer log server port configuration key (#67785)
  • Enforce ti:self scope on /execution/task-reschedules/{ti}/start_date (#67628)
  • UI: Fix misleading Calendar "Total Runs" coloring behavior (#67595)
  • Fix Stats not being initialized in the API server lifespan (#68514)
  • Fix BackfillDagRun.partition_key type annotation (#68432)
  • Fix backward compatibility for DagRunInfo partition fields (#68342)
  • Fix airflow db clean failing on foreign-key-referenced dag_version rows (#68339)
  • Fix 500 error when listing event logs with a NULL timestamp (#68338)
  • Fix MySQL downgrade from 3.3.0 for the deadline_alert.interval JSON conversion (#68337)
  • Fix older and custom secrets backends breaking on Airflow 3.2 (#68302)
  • Fix secrets backend connection errors being silently swallowed at DEBUG level (#68301)
  • UI: Fix the instance name title shown on non-Dag pages (#68288)
  • Fix scheduler not populating partition_date for temporal asset partitions (#68266)
  • UI: Fix wrong language being auto-detected from browser preferences (#68258)
  • Honor retry_policy on non-deferrable TriggerDagRunOperator wait failures (#68254)
  • Fix scheduler crash loop when the last task instance predates Dag versioning (#68253)
  • Fix team consumer asset filtering (#68242)
  • UI: Fix sluggish multi-selection behavior in tables (#68229)
  • UI: Fix mapped task instance links for tasks without a start date (#68194)
  • Fix setup/teardown auto-inclusion when clearing or marking tasks (#68193)
  • UI: Remove redundant columns from the XCom panel on the task instance page (#68188)
  • UI: Fix Gantt tooltip showing the wrong start date on queued/scheduled segments (#68176)
  • Fix Java SDK coordinator rejecting IPv4-mapped IPv6 connections (#68169)
  • Fix Java SDK tasks being rejected by the coordinator connection-ownership check (#68147)
  • UI: Fix language key for the Dag bundle filter (#68131)
  • Fix DagFileProcessorManager silent hang on database lock contention (#68118)
  • Mask all connection extra and variable values in the API audit log (#68049)
  • Fix spurious "Failed to detach context" error on Execution API disconnects (#68039)
  • UI: Fix Dag code highlighting for triple-quoted and escaped-brace f-strings (#68026)
  • Fix cursor encoding for column-form sort parameters in the REST API (#67973)
  • Fix SimpleAuthManager not preserving the deep-link next URL on first login (#67965)
  • Guard the task stats emission to prevent errors (#67955)
  • UI: Fix task instance state badge staying stale after a Mark-as action (#67950)
  • Register nested Pydantic models for XCom deserialization (#67932)
  • Fix example_asset_store consumer crash (#67922)
  • Raise InvalidJwtError in JWTValidator.avalidated_claims() when the key ID does not match (#67909)
  • Fix Kubernetes executor pod_override being stringified without the cncf provider (#67895)
  • UI: Prevent duplicate task instance summary stream refreshes after mutations (#67892)
  • Reject negative default_retention_days in the Task SDK and core API routes (#67890)
  • Fix none_failed_min_one_success trigger rule checks (#67873)
  • Remove trigger kwargs from the REST API response (#67868)
  • UI: Fix long parameter names overflowing the Trigger Dag modal (#67859)
  • Fix misleading log message in the task runner clear-on-success block (#67836)
  • Fix scheduler crash when logging orphaned task resets (#67822)
  • UI: Fix dashboard pool summary showing incorrect deferred slot usage (#67818)
  • Fix trigger datetime deserialization (#67795)
  • UI: Fix Graph layout for TaskGroup tasks wired to external nodes (#67720)
  • Fix airflow dags clear clearing the wrong day for non-UTC partitioned timetables (#67717)
  • Fix per-index evaluation of ONE_FAILED in mapped task groups (#67684)
  • UI: Fix dialog dismissal for the Chakra upgrade (#67674)
  • UI: Hide dashboard metric percentages when a state count is capped (#67664)
  • Apply per-file authorization to the dag-source endpoint (#67662)
  • Fix airflow dags next-execution --table crash when no next run exists (#67642)
  • UI: Fix the time picker omitting seconds (#67636)
  • Filter scheduling-dependencies graph edges by readable-Dag access (#67627)
  • Mask per-key secrets-backend-kwarg overrides on the Config API (#67622)
  • Fix GET /auth/login missing a 400 response in the OpenAPI spec (#67571)
  • Fix GET /pools incorrectly documenting a 404 response in the OpenAPI spec (#67570)
  • Add a compatibility layer for import errors caused by AirflowSecretsBackendAccessDenied (#67560)
  • UI: Fix rendering of None child state (#67552)
  • Fix sort order for mapped task instances (#67551)
  • Fix import errors total-entries count with multiple Dags per file (#67550)
  • UI: Prefer active over queued state for collapsed groups (#67543)
  • Fix callback state not updating from executor events due to a UUID type mismatch (#67542)
  • Reject wildcard origin in CORS config instead of toggling credentials (#67502)
  • Guard the finally-block logger in the HTTP access log middleware (#67501)
  • Strip CR/LF from user-supplied logical date before logging (#67500)
  • Redact secret-looking query parameters in the HTTP access log (#67498)
  • UI: Fix Calendar view to respect the user-selected timezone (#67497)
  • Escape LIKE wildcards in non-search filter parameters (#67496)
  • Fix missing redaction of secret values in variable JSON (#67495)
  • Fix bulk CREATE+OVERWRITE team-context authorization bypass (#67493)
  • UI: Return 400 instead of 500 from structure_data on a malformed asset expression (#67489)
  • Fix SimpleAuthManager redirect to the next URL after login (#67483)
  • Return 400 instead of 500 from materialize_asset on invalid input (#67445)
  • UI: Restore the Monaco find widget in the Dag Code view (#67391)
  • UI: Fix an HTTPException import that turned a 400 into a 500 in the dags endpoint (#67363)
  • Restore fail_fast handling when reschedule exceeds the MySQL TIMESTAMP limit (#67353)
  • Fix the Triggered Dag button not being visible during queued/running state (#67327)
  • Fix variables import with structured falsy values (#67060)
  • Avoid logging Execution API bearer credentials (#67059)
  • Sanitize Dag processor metric file names (#67029)
  • Return a 422 when the database rejects an API payload (#66888)
  • Prevent AlreadyRunningBackfill error caused by an invalid date range request (#66874)
  • Restrict owner-link and extra-link href values to safe schemes (http, https, mailto, relative) (#66741)
  • Add a session parameter to the BaseStateBackend interface to fix custom backends (#66708)
  • Allow deadline callbacks within the same Dag module (#66702)
  • UI: Fix relative React plugin bundle URLs in dev mode (#66618)
  • Validate Dag trigger conf as a JSON object or null (#66617)
  • Require a trust sentinel for state.user injection in get_user() (#66562)
  • Use hmac.compare_digest for SimpleAuthManager password comparison (CWE-208) (#66556)
  • Set SameSite=Lax on the SimpleAuthManager all-admins login cookie (#66502)
  • Reserve /auth and /pluginsv2 from plugin URL prefixes (#66501)
  • Use a cryptographically secure RNG for SimpleAuthManager passwords (#66500)
  • Fix Triggerer runner_health_check_threshold log formatting (#66486)
  • Fix Dag processor callback cleanup for versioned bundle files (#66484)
  • Default AIRFLOW_UID to 50000 in the airflow-init chown lines (#66481)
  • Strip CR/LF from MySQL URL query values before forwarding to my.cnf (#66325)
  • Fix CronMixin not resolving cron presets before validation (#66102)
  • Fix AirflowSDKConfigParser missing the mask_secrets method (#66077)
  • Fix resolve_xcom_backend to rely on the config schema default (#65938)
  • Fix a missing import cast error in the dag_run API route (#65748)
  • Mask Dag processor connection and variable responses (#65704)
  • UI: Show import error for deactivated Dags (#65687)
  • Forward MySQL SSL params from sql_alchemy_conn to airflow db shell (#65575)
  • Disable SQLite FK checks in the 0111 migration downgrade (#65545)
  • Handle Variable values that cannot be decrypted gracefully in the stable REST API (#65452)
  • Retry TriggerDagRunOperator when the triggered DagRun fails (#65390)
  • Fix task run exceptions never being caught by Sentry (#65161)
  • Fix the bulk task instance authorization error message rendering (#64719)
  • Fix trigger template rendering failure when operator template_fields differ from trigger attributes (#64715)
  • Fix task_defer with non-JSON next_kwargs in TaskInstance (#64714)
  • Add the error as context["exception"] in InProcessTestSupervisor (#64568)
  • Fix NPM security alerts in the simple auth manager (#64309)
  • Fix Dag run trigger to surface errors instead of swallowing them (#64130)
  • Fix Task SDK Connection extras built from a URI constructor (#64120)
  • Add insert/update-on-conflict for rendered task instance fields (#63874)
  • Fix timeout_with_traceback crashes on Windows and non-main threads (#63664)
  • UI: Wrap long lines in the rendered templates view (#63492)
  • Block path traversal via ".." in dag_id and run_id (#63296)
  • Fix the scheduler health check command in docker-compose.yaml (#62280)
  • Fix unmapped task deadlock when upstream tasks are removed (#62034)
  • Forward termination signals from the supervisor to the task subprocess (#61627)
  • Check destination team permission when using bulk APIs for connections, variables, and pools (#68573)
  • Fix the execution API /health check failing on the empty-path route (#68578)
  • Fix Dag run partition key filter breaking on composite keys containing | (#68459)
  • Fix the partition clear date range for non-UTC partitioned timetables (#68460)
  • Validate that partition keys are non-empty and within the column length (#68443)
  • Fix the scheduler serving stale Dag code after an in-place serialized Dag version update (#68558)
  • Determine the latest Dag version by version number to avoid collisions when timestamps tie (#68389)
  • Fix new runs and reruns executing an outdated bundle version when the Dag serialization is unchanged (#68336)
  • Fix remote logging from the task supervisor (#68370)
  • Upload task logs even when the final state update fails (#67935)
  • Escape URLs in the Task SDK client when looking up Dag operations (#68129)
  • Fix task scheduling when multi-team is enabled (#68634)
  • Fix secret values not being masked in rendered templates when keys use dot or dash separators (#68624)
  • Fix jwt_audience for the public API being read from two different config sections (#67494)
  • Fix duplicate deadline-miss callbacks firing from multiple HA scheduler replicas (#64737)
  • Fix scheduler crash on non-ASCII Dag names when OpenTelemetry metrics are enabled (#68023)
  • Fix Dag processor crash on non-ASCII names in OpenTelemetry gauge and timer metrics (#68284)
  • Report duplicate plugin names as import errors instead of silently ignoring them (#66649)
  • Require edit permission for async connection tests that update an existing connection (#68127)
  • Restore the deprecated [core] execution_api_server_url mapping to [workers] execution_api_server_url (#63949)
  • Fix dag.test() not re-syncing sibling Dags across repeated calls (#66205)
  • UI: Invalidate per-attempt task instance caches after actions so logs and details are not stale (#67212)
  • Fix task runner failure on a duplicate task instance success-state update (#63355)
  • Fix a race condition on the order_by parameter when listing Dag runs via the REST API (#68948)
  • Exclude non-successful Dag runs from the DeadlineReference.AVERAGE_RUNTIME deadline calculation so failed runs no longer skew the computed deadline (#68949)
  • Allow InProcessExecutionAPI to start without api_auth.jwt_secret configured (#68982)
  • Make airflow dags test wait for Human-in-the-loop input instead of looping indefinitely on parked HITL tasks (#69104)
  • Fix the Java coordinator rejecting macOS dual-stack loopback connections (#68973)
  • Fix an asset-event ingestion crash for Dags using FixedKeyMapper (#69326)
  • UI: Fix the details panel header overlapping the tabs (#69318)
  • Fix new Dag versions being created when a task's retry_policy was serialized (#69315)
  • Fix retry-policy overrides not being persisted to task-instance history (#69241)
  • Fix deadline callback data not being persisted (#69259)

Miscellaneous

  • Propagate the resolved task log level and [logging] namespace_levels to language SDK runtimes (#68712)
  • Forward run-identity attributes (dag_id, run_id, run_type) to the trace sampler so a custom head sampler can differentiate by run kind (#68592)
  • Remove all_map_indices from task_state_store.clear() in the task context (#68880)
  • Optimize the dag processor by caching bundle-to-team name lookups (#68730)
  • Rename the misleading last_automated_run param to reference_run (#68714)
  • Add a team_name tag to the remaining multi-team metrics (#68601)
  • Add a team_name tag to dag processor metrics for multi-team deployments (#68599)
  • Add a team_name tag to asset metrics for multi-team deployments (#68367)
  • UI: Persist dashboard alert collapse state and clamp long alerts (#68329)
  • Optimize bulk variable deletion to avoid N+1 queries (#68508)
  • UI: Unify the Dag Code tab toolbar styling with the Logs toolbar (#68449)
  • Optimize bulk Dag run authorization to avoid N+1 team-name queries (#68286)
  • Improve airflow dags command to use bulk clear (#68280)
  • Add the task_state_store table to the airflow db clean mechanism (#68218)
  • Add metrics and traces to ResumableJobMixin for crash recovery (#68213)
  • Improve ResumableJobMixin crash-recovery observability with better logging (#68206)
  • Pass DagRun to task_instance_mutation_hook for run-aware task mutation (#68198)
  • Add team_name to multi-team metrics (#68108)
  • Reduce redundant Dag team lookups in authorization checks (#68020)
  • Enhance ResumableJobMixin.get_job_status with context for better job status tracking (#68009)
  • Propagate OpenTelemetry trace headers from the client to Execution API server-side spans (#67904)
  • Widen the type hint for the DagRun.get_task_instances / fetch_task_instances state parameter (#67880)
  • UI: Use the bulk clear Dag runs endpoint for bulk Dag run clear (#67846)
  • Add a default parameter to the task and asset state get() method (#67842)
  • Make core API routes for task and asset states interact only with the database (#67835)
  • Optimize Dag processor file-queue deduplication from O(N^2) to O(N) (#67750)
  • Add allow_consumer_teams and allow_global_consumers columns to TaskOutletAssetReference (#67730)
  • Speed up the Dags list and dashboard queries on large DagRun tables (#67721)
  • Make partition_key provenance-only and inherit it onto asset events (#67718)
  • Speed up Dag serialization by skipping a redundant asset roundtrip (#67702)
  • Cache BaseOperator.__init__ signature in operator serialization (#67701)
  • Optimize TaskGroup.topological_sort for reverse-declared Dags (#67688)
  • Update serialization for producer-side asset access control (#67658)
  • Allow outlets to be added and accessed in AssetStateAccessor (#67619)
  • Unify task/asset state storage between the Core API and Execution API (#67547)
  • Decorate custom state references with an envelope for UI clarity (#67530)
  • Simplify authoring of task and asset states by allowing JSON types (#67418)
  • Replace Sphinx Redoc with Swagger for the API docs (#67390)
  • Emit OpenTelemetry spans around listener hook calls (#67347)
  • UI: Update verbiage for lower-priority backfill runs (#67338)
  • Fix N+1 query in the bulk task instance delete endpoint (#67304)
  • Speed up TaskGroup.topological_sort with an int-indexed projected sweep (#67288)
  • UI: Use react-query native error state for bulk action hooks (#67284)
  • Wrap executor.heartbeat() in a timer to localize scheduler loop slowdowns (#66808)
  • Emit dagrun.first_task_start_delay separately from scheduling delay (#66807)
  • Share one poll loop across sibling event triggers (#66584)
  • UI: Upgrade icons, spacing, and default component themes (#66569)
  • Warn when SimpleAuthManager runs in a production-shaped deployment (#66563)
  • Migrate Stackdriver logging config to the RemoteLogIO pattern (#66513)
  • Add a BundleVersion dataclass and version_data persistence to DagVersion (#66491)
  • Avoid lazy-loading timetable fields for latest DagRuns (#66488)
  • Move allow_producer_teams to DagScheduleAssetReference (#66487)
  • Pass user teams to the create_asset_event endpoint (#66367)
  • Load USFederalHolidayCalendar lazily to reduce memory usage when loading examples (#66303)
  • Propagate task OpenTelemetry trace context through IPC into Execution API requests (#66151)
  • Surface worker Dag parse duration in the task log (#66138)
  • Skip deserializing trigger_kwargs when loading serialized Dags (#66002)
  • Honor AUTH_ROLE_PUBLIC in the FastAPI API server (#65685)
  • Add extended sysinfo for the Edge worker (#65472)
  • Clarify logs when a Dag is being processed in the Dag processor (#65196)
  • Add indexes on task_instance.dag_version_id and dag_run.created_dag_version_id (#64818)
  • Improve creation of RuntimeTaskInstance in TriggerRunner for start_for_trigger functionality (#64298)
  • Mark the Triggerer supervisor as a server context so it can read metastore connections (#64022)
  • Load hook metadata from YAML without importing the hook class (#63826)
  • Add detailed task spans (#63568)
  • Downgrade logging on query JSON parsing and add a JSON load condition (#62044)
  • UI: Add a Deadlines section with a time-range selector to the Dashboard page (#68038)
  • UI: Add a modal for editing notes with Markdown support (#68362)
  • UI: Improve the Human-In-The-Loop form UX (#68397)
  • Add a team_name tag to executor metrics for multi-team deployments (#68593)
  • Make task and asset state store row size limits configurable (#68133)
  • UI: Add notification UX for Human-In-The-Loop actions (#68346)
  • Allow synchronous deadline callbacks (SyncCallback) to access Connections and Variables (#65269)
  • Add a team_name tag to deadline metrics for multi-team deployments (#68589)
  • Add a team_name tag to scheduler metrics for multi-team deployments (#68594)
  • Defer the Cadwyn import so FastAPI/Starlette stay off the Task SDK worker path, reducing per-worker memory (#69029)

Doc Only Changes

  • Complete the Taiwanese Mandarin (zh-TW) translation (#68870)
  • Add missing Korean (ko) translations (#68600)
  • Close German (de) translation gaps (#68356)
  • Add a segment fan-out example to the asset partition example Dag (#68722)
  • Fix runtime-partition example Dags using unreachable schedules (#68719)
  • Add an example Dag for the task state store with mapped tasks (#68670)
  • Fix the gap in the Taiwanese Mandarin (zh-TW) translation (#68668)
  • Add wait-policy examples to the asset partition example Dag (#68658)
  • Add a contributing guide for language SDKs (#68330)
  • Add sdk.TIRunContext documentation for the Go SDK (#68319)
  • Add a Go Task SDK authoring guide to the docs (#68223)
  • Update supported-versions doc to mark 2.11.2 as EOL (#68212)
  • Add documentation for ResumableJobMixin and resumable tasks (#68136)
  • Add CLI examples for team-scoped pools (#68111)
  • Add docs for multi-team triggerer support (#67608)
  • Clarify trigger rule behavior for the removed upstream state (#67452)
  • Fix outdated image links in dags.rst (#67357)
  • Add an example and docs for runtime asset partitioning (AIP-76) (#67307)
  • Add documentation for the Task and Asset Store (AIP-103) (#67299)
  • Add a dynamic task mapping no-op example (#67022)
  • Add documentation about adding access_control to the Asset object (#66949)
  • Add a how-to for Dag-level retry via on_failure_callback (#66277)
  • Fix documentation after PR 62645 (#65843)
  • Add documentation for team-based asset event filtering (#65690)
  • Document on_kill()/cleanup() for triggers (#65671)
  • Explain xcom_pull behaviour without task_ids in the docs (#65406)
  • Improve standalone authentication documentation for Airflow 3.x (#65330)
  • Clarify manual Dag run data interval semantics in Airflow 3 (#64740)
  • Document and test xcom_pull run_id usage for triggered Dag runs (#63030)
  • Update params in the backfill documentation (#61821)
  • Document the apache-airflow-mypy package in the core docs (#68561)
  • Fix typos and formatting in the Fundamentals documentation (#68524)
  • Complete the Hindi (hi) UI translation (#68574)
  • Fill the Taiwanese Mandarin (zh-TW) UI translation gap (#68563)
  • Document that Dag bundle kwargs should reference a Connection rather than inline credentials (#69105)
  • Add example plugins and expand the asset-partitions documentation (#69017)
  • Java SDK docs: JUL setup, pinning java_executable, and a config-reload note (#69020)
  • Correct the example config for the coordinators (#68940)