Emit namespace migration workflow lifecycle events

[michaely520]

What changed?

Adds namespace_lifecycle start and finish events for the namespace handover, force replication, and catchup system workflows.

The events carry workflow identity and the core operation inputs. Finished events classify the result as succeeded, canceled, or failed. Force replication reports its cumulative verified workflow count and emits only one start and one finish across a continue-as-new chain.

Emission uses one shared activity, the existing system.emitNamespaceLifecycleEvents gate, disconnected cleanup for cancellation, and workflow versioning for replay compatibility. Existing shard handover events are unchanged.

Why?

These system workflows currently have no consistent operation-level event pair, which makes it difficult to correlate a namespace migration request with its final outcome.

How did you test it?

• covered by existing tests
• added new unit test(s)

go test -tags test_dep ./common/wideevents ./service/worker/migration

make fmt-imports

make lint-code reports no issues introduced by this change; the repository-wide target still reports existing findings on current main.

Potential risks

The terminal event is best effort and cannot run after server-side workflow termination or workflow run timeout because those outcomes do not execute workflow cleanup.

[michaely520]

Local real E2E verification (2026-08-20)

I ran a temporary three-cluster XDC functional harness against this branch with system.emitNamespaceLifecycleEvents=true and an in-memory OTEL LoggerProvider installed on the real Temporal servers. It used the production system worker registrations and the real EmitNamespaceMigrationWorkflowLifecycle activity (no mocked workflow/activity emission). It executed catchup, force-replication with verification, and namespace-handover through temporal-system/default-worker-tq.

The run passed twice. Focused command result:

--- PASS: TestMigrationLifecycleE2ESuite (18.52s)
    --- PASS: TestMigrationLifecycleE2ESuite/TestRealSystemWorkersEmitAllLifecycleEvents (16.17s)
PASS

Complete lifecycle-event trace from the second run (the unrelated server startup/shutdown logs are omitted):

{"service_name":"worker","event_name":"namespace_lifecycle","attributes":{"details":{"first_run_id":"01a02027-0a14-7caa-95f0-494a0cb53a9f","input":{"catchup_cluster":"catchup-passive_maaor","target_cluster":"target-passive_maaor"},"run_id":"01a02027-0a14-7caa-95f0-494a0cb53a9f","workflow_id":"migration-lifecycle-catchup-e2e","workflow_type":"catchup"},"namespace":"test-namespace-7e974859-4a2e-45eb-822d-aca3f967d194","namespace_id":"3c5e67c3-5815-4acf-879f-b90ad6cf9dec","phase":"namespace_catchup_started"}}
{"service_name":"worker","event_name":"namespace_lifecycle","attributes":{"details":{"first_run_id":"01a02027-0a14-7caa-95f0-494a0cb53a9f","input":{"catchup_cluster":"catchup-passive_maaor","target_cluster":"target-passive_maaor"},"run_id":"01a02027-0a14-7caa-95f0-494a0cb53a9f","status":"succeeded","workflow_id":"migration-lifecycle-catchup-e2e","workflow_type":"catchup"},"namespace":"test-namespace-7e974859-4a2e-45eb-822d-aca3f967d194","namespace_id":"3c5e67c3-5815-4acf-879f-b90ad6cf9dec","phase":"namespace_catchup_finished"}}
{"service_name":"worker","event_name":"namespace_lifecycle","attributes":{"details":{"first_run_id":"01a02027-0a3b-7816-897b-959befc9da3f","input":{"concurrent_activity_count":1,"overall_rps":10,"query":"","target_cluster":"target-passive_maaor","verification_enabled":true},"run_id":"01a02027-0a3b-7816-897b-959befc9da3f","workflow_id":"migration-lifecycle-force-replication-e2e","workflow_type":"force-replication"},"namespace":"test-namespace-7e974859-4a2e-45eb-822d-aca3f967d194","namespace_id":"3c5e67c3-5815-4acf-879f-b90ad6cf9dec","phase":"namespace_force_replication_started"}}
{"service_name":"worker","event_name":"namespace_lifecycle","attributes":{"details":{"first_run_id":"01a02027-0a3b-7816-897b-959befc9da3f","input":{"concurrent_activity_count":1,"overall_rps":10,"query":"","target_cluster":"target-passive_maaor","verification_enabled":true},"run_id":"01a02027-0a3b-7816-897b-959befc9da3f","status":"succeeded","verified_workflow_count":0,"workflow_id":"migration-lifecycle-force-replication-e2e","workflow_type":"force-replication"},"namespace":"test-namespace-7e974859-4a2e-45eb-822d-aca3f967d194","namespace_id":"3c5e67c3-5815-4acf-879f-b90ad6cf9dec","phase":"namespace_force_replication_finished"}}
{"service_name":"worker","event_name":"namespace_lifecycle","attributes":{"details":{"first_run_id":"01a02027-1df1-7261-86e9-bf80682d1528","input":{"allowed_lagging_seconds":10,"allowed_lagging_tasks":0,"handover_timeout_seconds":20,"remote_cluster":"target-passive_maaor"},"run_id":"01a02027-1df1-7261-86e9-bf80682d1528","workflow_id":"migration-lifecycle-handover-e2e","workflow_type":"namespace-handover"},"namespace":"test-namespace-7e974859-4a2e-45eb-822d-aca3f967d194","namespace_id":"3c5e67c3-5815-4acf-879f-b90ad6cf9dec","phase":"namespace_handover_started"}}
{"service_name":"worker","event_name":"namespace_lifecycle","attributes":{"details":{"first_run_id":"01a02027-1df1-7261-86e9-bf80682d1528","input":{"allowed_lagging_seconds":10,"allowed_lagging_tasks":0,"handover_timeout_seconds":20,"remote_cluster":"target-passive_maaor"},"run_id":"01a02027-1df1-7261-86e9-bf80682d1528","status":"succeeded","workflow_id":"migration-lifecycle-handover-e2e","workflow_type":"namespace-handover"},"namespace":"test-namespace-7e974859-4a2e-45eb-822d-aca3f967d194","namespace_id":"3c5e67c3-5815-4acf-879f-b90ad6cf9dec","phase":"namespace_handover_finished"}}

The temporary three-cluster harness was removed after verification, so the PR retains only the focused unit/workflow tests.

[michaely520]

Additional real E2E verification: force-replication continue-as-new and cancellation

I ran a temporary two-cluster XDC functional harness against this branch with system.emitNamespaceLifecycleEvents=true and a real in-memory OTEL LoggerProvider. It used the production Temporal services, production system workers, and real force-replication activities; lifecycle emission was not mocked.

Setup:

• Created three completed user workflows.
• Forced one workflow page per run (ListWorkflowsPageSize=1, PageCountPerExecution=1), which caused the force-replication workflow to continue as new.
• Enabled verification against the standby cluster.
• Started a second force-replication workflow and issued CancelWorkflow after its start lifecycle record was observed but before verification completed.

Result:

--- PASS: TestForceReplicationLifecycleE2ESuite (31.47s)
    --- PASS: TestForceReplicationLifecycleE2ESuite/TestContinueAsNewAndCancellationTraces (30.19s)
PASS
ok go.temporal.io/server/tests/xdc 32.910s

Complete filtered lifecycle trace:

{"service_name":"worker","event_name":"namespace_lifecycle","attributes":{"details":{"first_run_id":"01a02034-7d23-78b6-b899-77ac58fe761a","input":{"concurrent_activity_count":1,"overall_rps":100,"query":"","target_cluster":"standby_dndrd","verification_enabled":true},"run_id":"01a02034-7d23-78b6-b899-77ac58fe761a","workflow_id":"force-replication-continue-as-new-e2e","workflow_type":"force-replication"},"namespace":"test-namespace-b5420b5c-4363-4b75-8fe2-73a205e1c357","namespace_id":"22818d46-f62a-4f20-bbcc-ef6286bdcc8c","phase":"namespace_force_replication_started"}}
{"service_name":"worker","event_name":"namespace_lifecycle","attributes":{"details":{"first_run_id":"01a02034-7d23-78b6-b899-77ac58fe761a","input":{"concurrent_activity_count":1,"overall_rps":100,"query":"","target_cluster":"standby_dndrd","verification_enabled":true},"run_id":"f895d151-d703-4c4f-99fe-4d02f22e140b","status":"succeeded","verified_workflow_count":3,"workflow_id":"force-replication-continue-as-new-e2e","workflow_type":"force-replication"},"namespace":"test-namespace-b5420b5c-4363-4b75-8fe2-73a205e1c357","namespace_id":"22818d46-f62a-4f20-bbcc-ef6286bdcc8c","phase":"namespace_force_replication_finished"}}
{"service_name":"worker","event_name":"namespace_lifecycle","attributes":{"details":{"first_run_id":"01a02034-cb7a-7c0a-a6b0-825344933f4d","input":{"concurrent_activity_count":1,"overall_rps":100,"query":"","target_cluster":"standby_dndrd","verification_enabled":true},"run_id":"01a02034-cb7a-7c0a-a6b0-825344933f4d","workflow_id":"force-replication-canceled-e2e","workflow_type":"force-replication"},"namespace":"test-namespace-b5420b5c-4363-4b75-8fe2-73a205e1c357","namespace_id":"22818d46-f62a-4f20-bbcc-ef6286bdcc8c","phase":"namespace_force_replication_started"}}
{"service_name":"worker","event_name":"namespace_lifecycle","attributes":{"details":{"error_message":"canceled","first_run_id":"01a02034-cb7a-7c0a-a6b0-825344933f4d","input":{"concurrent_activity_count":1,"overall_rps":100,"query":"","target_cluster":"standby_dndrd","verification_enabled":true},"run_id":"01a02034-cb7a-7c0a-a6b0-825344933f4d","status":"canceled","verified_workflow_count":0,"workflow_id":"force-replication-canceled-e2e","workflow_type":"force-replication"},"namespace":"test-namespace-b5420b5c-4363-4b75-8fe2-73a205e1c357","namespace_id":"22818d46-f62a-4f20-bbcc-ef6286bdcc8c","phase":"namespace_force_replication_finished"}}

What this proves:

• Intermediate continue-as-new runs do not emit false completion records.
• The initial start and final completion are correlated by first_run_id while run_id identifies the actual emitting run.
• The verification count survives continue-as-new and finishes at 3.
• Workflow cancellation still executes the disconnected deferred lifecycle activity and emits status=canceled.
• Exactly two lifecycle records were emitted per logical workflow chain.

The temporary heavy E2E harness was removed after verification; no additional test files are included in the PR.

[qyc5937]

nit. this does not affect workflow correctness. One edge case is a force-replication chain that begins on old code and has a later continue-as-new run execute on new code; it may emit finished without ever emitting started so resulting in an incomplete lifecycle pair.