Skip to content

Stop variables export/import from silently corrupting values - #70944

Open
Eason09053360 wants to merge 1 commit into
apache:mainfrom
Eason09053360:fix-variable-export-import-roundtrip
Open

Stop variables export/import from silently corrupting values#70944
Eason09053360 wants to merge 1 commit into
apache:mainfrom
Eason09053360:fix-variable-export-import-roundtrip

Conversation

@Eason09053360

Copy link
Copy Markdown
Contributor

airflow variables export followed by airflow variables import can silently rewrite a variable's value and invent a description for it.

Export writes a bare value when a variable has no description, while import treats any dict carrying a "value" key as a {"value": ..., "description": ...} envelope — nothing distinguishes the envelope from a value that happens to look like one. Export also decodes the stored JSON, losing the difference between raw text and a JSON-encoded string.

Round-tripping through a file on main :

DB value before exported as after re-import
{"value": "a", "description": "b"} {"value": "a", "description": "b"} value → a, description → b
{"value": 1, "other": 2} {"value": 1, "other": 2} value → 1, other dropped
"hello" hello hello — no longer valid JSON, so deserialize_json=True starts raising

Import's reconstruction is deterministic — strings are stored verbatim, everything else is JSON-encoded, and one envelope layer is unwrapped — so export alone can be made lossless, leaving hand-written import files behaving exactly as before. Export now emits the stored form when the decoded value is a string, and wraps values that are themselves envelope-shaped so import's unwrap consumes our envelope rather than the user's data.

Exported files are unchanged for values that were never ambiguous. One visible change worth flagging: a variable stored as a JSON-encoded string (Variable.set(k, "text", serialize_json=True)) now exports as "\"text\"" rather than "text". The old output could not be re-imported without corruption, so there is no lossless way to keep it.

Left for follow-up PRs: import still truncates a hand-written {"value": 1, "other": 2} to 1, and the UI import path does not unwrap the envelope at all.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

`airflow variables export` writes a bare value for any variable without a
description, while `airflow variables import` treats every dict carrying a
"value" key as a {"value": ..., "description": ...} envelope. A variable whose
own value happens to have that shape is therefore unwrapped on the way back in:
its value is replaced by the inner "value" and a description is invented from
the inner "description". Nothing warns the operator, so a round-trip through a
file - the documented way to migrate variables between environments - quietly
rewrites their data.

Export also decodes the stored JSON before writing it out, which erases the
difference between a variable holding raw text and one holding a JSON-encoded
string. Re-importing flattens the latter, and any Dag reading it with
deserialize_json=True starts failing on a value that is no longer valid JSON.

Import's reconstruction is deterministic - strings are stored verbatim,
everything else is JSON-encoded, and one envelope layer is unwrapped - so export
alone can be made lossless. Fixing it there leaves import untouched and keeps
hand-written import files working exactly as before.
@Eason09053360
Eason09053360 marked this pull request as ready for review August 2, 2026 16:59
Comment thread airflow-core/newsfragments/70944.bugfix.rst Outdated
@Eason09053360
Eason09053360 force-pushed the fix-variable-export-import-roundtrip branch from 92d0c1b to 0aa93cb Compare August 2, 2026 17:20

@henry3260 henry3260 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The export change looks like it regresses the UI import path for the same two cases this PR fixes for the CLI, since the CLI and UI use different unwrap rules — as far as I can tell there's no export-only change that's correct for both. The PR notes the UI limitation as follow-up, but given export now depends on the unwrap rules, I'd lean towards fixing the UI side in this PR rather than deferring it.

I'm not familiar with the UI part though, so I'd like to hear your thoughts. cc @bbovenzi

Comment on lines +195 to +197
else:
if isinstance(val, str):
val = var.val

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
else:
if isinstance(val, str):
val = var.val
if isinstance(val, str):
val = var.val

Comment on lines +189 to +190
# Emit a form variables_import turns back into var.val: it stores strings verbatim,
# JSON-encodes everything else, and unwraps any dict carrying a "value" key.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Emit a form variables_import turns back into var.val: it stores strings verbatim,
# JSON-encodes everything else, and unwraps any dict carrying a "value" key.
# Mirror variables_import's reconstruction so export/import round-trips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants