Skip to content

fix serialize_template_field handling callable value in dict#63871

Open
wjddn279 wants to merge 4 commits intoapache:mainfrom
wjddn279:fix-serialize-template-field-handling-callable-in-dict
Open

fix serialize_template_field handling callable value in dict#63871
wjddn279 wants to merge 4 commits intoapache:mainfrom
wjddn279:fix-serialize-template-field-handling-callable-in-dict

Conversation

@wjddn279
Copy link
Copy Markdown
Contributor

closed: #63334

Cause

The serialization result of the DAG is as follows.

{
  "__var": {
    "op_args": [],
    "task_id": "consume",
    "ui_color": "#ffefeb",
    "op_kwargs": "{'values': [3, 1, 2], 'sort_key': <function unstable_dag.<locals>.<lambda> at 0xf2dc11567e20>}",
    "task_type": "_PythonDecoratedOperator",
    "retry_delay": 300.0,
    "_task_module": "airflow.providers.standard.decorators.python",
    "_operator_name": "@task",
    "template_fields": [
      "templates_dict",
      "op_args",
      "op_kwargs"
    ],
    "_needs_expansion": false,
    "python_callable_name": "consume",
    "template_fields_renderers": {
      "op_args": "py",
      "op_kwargs": "py",
      "templates_dict": "json"
    }
  },

When serializing a dict containing lambda function definitions, the object's id value is reflected as-is, causing version changes on every serialization. The root cause is that when a dict contains callable objects, is_jsonable returns False, and since the dict itself is not callable, it falls through to that code path.

Solution

I added logic to filter out this case and convert callable objects consistently within the existing dict handling (sorting) logic. I have confirmed locally that this resolves the issue.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@wjddn279 wjddn279 force-pushed the fix-serialize-template-field-handling-callable-in-dict branch from 4a10da9 to 6f6156f Compare March 18, 2026 12:30
@eladkal eladkal added this to the Airflow 3.1.9 milestone Mar 24, 2026
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Apr 1, 2026
@kaxil kaxil requested a review from Copilot April 2, 2026 00:45
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes non-deterministic DAG serialization when templated dict fields contain callable values (e.g., lambdas), preventing serialized output from changing on every run due to function object addresses.

Changes:

  • Sanitizes callable values during recursive dict sorting in serialize_template_field.
  • Refactors callable stringification into a helper for consistent formatting.
  • Adds unit tests intended to validate stable serialization when dicts contain callables.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File Description
airflow-core/src/airflow/serialization/helpers.py Makes callable values in dicts serialize to a stable string during recursive sorting.
airflow-core/tests/unit/serialization/test_helpers.py Adds regression tests around dicts containing callables to prevent unstable serialization.

@eladkal eladkal added backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch and removed backport-to-v3-1-test labels Apr 6, 2026
@eladkal eladkal modified the milestones: Airflow 3.1.9, Airflow 3.2.1 Apr 6, 2026
@wjddn279 wjddn279 force-pushed the fix-serialize-template-field-handling-callable-in-dict branch 5 times, most recently from 2e7b627 to 7c4b179 Compare April 10, 2026 04:25
@wjddn279 wjddn279 force-pushed the fix-serialize-template-field-handling-callable-in-dict branch from 7c4b179 to 8eb05f8 Compare April 10, 2026 05:46
@wjddn279 wjddn279 requested a review from XD-DENG as a code owner April 10, 2026 05:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:DAG-processing backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch ready for maintainer review Set after triaging when all criteria pass. type:bug-fix Changelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DAG version churn when using unstable parse-time callable serialization

4 participants