[Bug] continue_as_new() loses memo upserts when memo is omitted

[Jackmeson1]

Memo behavior on continue as new is inconsistent.

Some quick AI research suggests that the behavior of a pending memo upsert varies depending on language:

SDK	Memo omitted on continue-as-new	Effect of memo upserts	Explicit empty memo
Python	Inherits the memo the current run started with	Changes are lost; deleted keys return	Treated as omitted
TypeScript	Same Core fallback to start-of-run memo	Same stale-memo behavior	Core treats empty map as omitted
.NET	Same Core fallback to start-of-run memo	Same stale-memo behavior	Core treats empty map as omitted
Ruby	Same Core fallback to start-of-run memo	Same stale-memo behavior	Empty hash is treated as omitted
Rust	Same Core fallback to start-of-run memo	Same stale-memo behavior	None and Some(empty) both become an empty command map; Core treats it as omitted
Go	Sends the current, updated memo	Changes and deletions carry forward, including removal of all keys	No separate memo override in the normal continue-as-new API
Java	Leaves memo unset; inspected server path does not inherit it	Memo is not automatically carried forward, including unchanged entries	Sends an explicit empty memo

There is no indication as yet that anyone has actually been impacted by this behavior, and it could easily be worked around by passing the memos to the continue as new.

Original Report in Python

What are you really trying to do?

Keep the memo of a workflow, including changes made with workflow.upsert_memo(), across workflow.continue_as_new(). This follows the maintainer's invitation to discuss the behavior when temporalio/sdk-python#2058 was declined (comment).

Describe the bug

workflow.continue_as_new() documents memo as "Memo for the workflow. Defaults to the current workflow's memo." (temporalio/workflow/_workflow_ops.py:881). When memo is omitted, the next run starts with the memo that the previous run started with: a changed value reverts, an added key is missing and a removed key comes back. Search attributes upserted with workflow.upsert_search_attributes() are carried over, and an explicitly supplied nonempty memo is preserved.

continue_as_new	run 1 started with	workflow.memo() in run 1 after the upsert	workflow.memo() at the start of run 2
memo omitted	{'changed': 'old value', 'removed': 'removed', 'kept': 'kept'}	{'added': 'added', 'changed': 'new value', 'kept': 'kept'}	{'changed': 'old value', 'kept': 'kept', 'removed': 'removed'}
memo=dict(workflow.memo())	same	same	{'added': 'added', 'changed': 'new value', 'kept': 'kept'}
memo omitted, every key removed	{'a': '1', 'b': '2'}	{}	{'a': '1', 'b': '2'}
memo={}, every key removed	{'a': '1', 'b': '2'}	{}	{'a': '1', 'b': '2'}

In all four runs the keyword search attribute upserted in run 1 (initial at start, upserted in run 1) is upserted in run 2, and describe() of run 2 shows the same memo as workflow.memo(). The released temporalio 1.35.0 and 1.30.0 give the same memo and search-attribute values with the same script (dictionary order and generated BuildIds differ).

Where it comes from: when memo is omitted the SDK builds no memo for the command (temporalio/worker/_workflow_instance.py:3912-3920 and :3942-3944; the payloads are only built if self._input.memo, so {} counts as omitted). sdk-core fills an empty memo of the command from the start info of the run, whereas search attributes use the current values (augment_continue_as_new_with_current_values, workflow_machines.rs:1712-1735, sdk-core f49305f1). The proto comment of the command says "If unset, re-uses the current workflow's memo" (temporalio/bridge/proto/workflow_commands/workflow_commands_pb2.pyi:999).

PR temporalio/sdk-python#2058 proposed sending the current memo when none was supplied (its description notes that this cannot cover removing every key, because the current memo is then empty and sdk-core treats an empty memo as unset). It was declined: "This doesn't sound like a change we want to make. If you want to summarize in an issue, we can consider how we want to address it." This issue follows that invitation to discuss the behavior.

Minimal Reproduction

python memocan_repro.py starts a dev server with WorkflowEnvironment.start_local() (the script passes memo=None for "omitted").

memocan_repro.py

"""upsert_memo, then continue_as_new without memo=: which memo does the next run start with?

It starts a local dev server with WorkflowEnvironment.start_local().

    python memocan_repro.py
"""

import asyncio
import sys
import uuid

import temporalio
from temporalio import workflow
from temporalio.common import SearchAttributeKey, SearchAttributePair, TypedSearchAttributes
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Worker

KEYWORD = SearchAttributeKey.for_keyword("MemoCanKeyword")


@workflow.defn
class MemoContinueAsNew:
    @workflow.run
    async def run(self, scenario: str, run_1_memo: dict | None = None) -> dict:
        if run_1_memo is None:  # first run
            if scenario.startswith("remove_all"):
                workflow.upsert_memo({key: None for key in workflow.memo()})
            else:
                workflow.upsert_memo({"changed": "new value", "added": "added", "removed": None})
            workflow.upsert_search_attributes([KEYWORD.value_set("upserted")])
            run_1_memo = dict(workflow.memo())  # the memo as this run sees it after the upsert
            # "explicit" passes the current memo; "remove_all_explicit" passes an empty one
            memo = {"explicit": dict(workflow.memo()), "remove_all_explicit": {}}.get(scenario)
            workflow.continue_as_new(args=[scenario, run_1_memo], memo=memo)
        return {
            "memo in run 1 after upsert": run_1_memo,
            "memo at the start of run 2": dict(workflow.memo()),
            "keyword search attribute in run 2": workflow.info().typed_search_attributes.get(KEYWORD),
        }


SCENARIOS = {
    # name: memo the first run is started with
    "omitted": {"changed": "old value", "removed": "removed", "kept": "kept"},
    "explicit": {"changed": "old value", "removed": "removed", "kept": "kept"},
    "remove_all": {"a": "1", "b": "2"},
    "remove_all_explicit": {"a": "1", "b": "2"},
}


async def main() -> None:
    print(f"temporalio {temporalio.__version__}, python {sys.version.split()[0]}")
    async with await WorkflowEnvironment.start_local(search_attributes=[KEYWORD]) as env:
        task_queue = str(uuid.uuid4())
        async with Worker(env.client, task_queue=task_queue, workflows=[MemoContinueAsNew]):
            for scenario, start_memo in SCENARIOS.items():
                handle = await env.client.start_workflow(
                    MemoContinueAsNew.run,
                    scenario,
                    id=f"{scenario}-{uuid.uuid4()}",
                    task_queue=task_queue,
                    memo=start_memo,
                    search_attributes=TypedSearchAttributes([SearchAttributePair(KEYWORD, "initial")]),
                )
                result = await handle.result()
                description = await env.client.get_workflow_handle(handle.id).describe()
                print(f"scenario={scenario}")
                print(f"  memo the workflow was started with:  {start_memo}")
                for label, value in result.items():
                    print(f"  {label + ':':<37}{value!r}")
                print(f"  describe() of run 2, memo:           {await description.memo()}")
                attributes = {p.key.name: p.value for p in description.typed_search_attributes}
                print(f"  describe() of run 2, search attrs:   {attributes}")


if __name__ == "__main__":
    asyncio.run(main())

Environment/Versions

• OS and processor: Linux x86-64 (Ubuntu 24.04.5)
• Temporal Version: SDK main at d488ff59 (temporalio.__version__ is 1.35.0; built from source with poe build-develop; sdk-core f49305f1), Python 3.12.3, dev server from WorkflowEnvironment.start_local() (Temporal CLI 1.10.0, server 1.32.2)
• Are you using Docker or Kubernetes or building Temporal from source? No Docker or Kubernetes; the SDK is built from source.

Additional context

Alternatives, for discussion:

1. sdk-core tracks memo upserts, as it does for search attributes, and uses them for an unset memo.
2. The docstring says that an omitted memo means the memo the run started with.

Empty memo. The memo of the command is a map and sdk-core treats an empty one as unset (attrs.memo.is_empty()); the SDK also treats memo={} as omitted. After upsert_memo removed every key (last two rows) the next run gets the start memo back. When the run started with a nonempty memo, memo={} cannot explicitly replace it with an empty memo during continue-as-new. Tracking current memo in Core could preserve the result of removing every key before continuing. Distinguishing an explicit empty override from omission would still require a separate representation or API decision.

Replay. I recorded the first run of a workflow that omits memo and replayed it with a workflow that passes memo=dict(workflow.memo()), and the other way round (script available on request). Both cross-direction replays and their two unchanged-code controls pass for this one-upsert example with the default converter and no payload codec. This checks changing the supplied memo contents, not a patched SDK/Core implementation. sdk-core's continue-as-new state machine checks event and command types without comparing memo contents (continue_as_new_workflow_state_machine.rs:68-95).

AI disclosure: this was prepared with AI assistance (Claude Opus 5.5 and Claude Sonnet 5.5).