Add State Persistence: Snapshot on Stop, Restore on Start (#335 P0-b)

[thzgajendra]

Objective / Issue

P0-b of the "minikube for cloud resources" roadmap (#335), and the persistence half of #107. The standalone server was in-memory only, so cloudemu stop/start (or any process restart) lost every emulated resource — the biggest dev-experience gap versus LocalStack (whose persistence + Cloud Pods are paywalled). This adds opt-in persistence so resources survive a restart.

What we found

• No generic serializer is possible. State lives in ~473 unexported memstore.Store fields across ~77 service mocks, and the value types hold sync.Mutex, func (e.g. lambda funcData.handler), and nested stores. Go has no pickle; json/gob/msgpack all drop unexported/func fields, and reflection can't reach unexported fields without unsafe. So a "save everything" engine isn't feasible — persistence must go through clean interfaces.
• Blast radius: additive. With --persist off (the default) behavior is byte-for-byte unchanged; the in-process test-double API is untouched.

How we fixed it

• New persist package: schema-versioned Snapshot/ProviderState with Export and Restore that read/write through the existing storage / database / secrets / compute driver interfaces (the same ones seed uses). One human-readable, git-diffable JSON file spans all three providers.
• Coverage: S3/Blob/GCS · DynamoDB/Firestore/Cosmos · Secrets Manager/Key Vault/Secret Manager · EC2/VMs/GCE.
• serve: --persist restores after providers are built and snapshots after graceful shutdown (so no in-flight request races the read); --state-file; --persist-metadata-only to omit object bodies for a smaller snapshot.
• Lifecycle CLI: start manages the snapshot path under the run dir; delete removes it.
• Object bodies, secret values, and table items are saved by default. Compute instances are recreated via RunInstances, so image/type/tags are preserved but the emulator assigns fresh instance IDs/IPs.

How it works

State lives in RAM while running (fast, unchanged); on stop it's exported to one JSON file on disk, and on start it's read back in before the first request is served.

sequenceDiagram
    actor U as You
    participant CLI as cloudemu CLI
    participant S as serve (daemon)
    participant P as persist
    participant D as Drivers<br/>(S3 · DynamoDB · Secrets · EC2)
    participant F as snapshot.json<br/>(disk)

    U->>CLI: cloudemu start --persist
    CLI->>S: spawn serve --persist --state-file …
    S->>S: build providers (empty)
    alt snapshot file exists
        S->>P: ReadFile(state-file)
        P->>F: read
        F-->>P: Snapshot (schema-versioned JSON)
        P->>D: Restore — CreateBucket/PutObject,<br/>CreateTable/PutItem, CreateSecret, RunInstances
    else first run / no file
        Note over S: start empty (today's behavior)
    end
    S-->>U: endpoints up ✅

    Note over U,D: normal use — all reads/writes hit in-memory stores (RAM)
    U->>D: aws s3 cp / put-item / create-secret / run-instances

    U->>CLI: cloudemu stop
    CLI->>S: SIGTERM
    S->>S: graceful shutdown (drain in-flight, then quiescent)
    S->>P: Export current state
    P->>D: ListBuckets/GetObject, Scan, ListSecrets, DescribeInstances
    D-->>P: buckets/objects, tables/items, secrets, instances
    P->>F: WriteFile(Snapshot JSON)
    S-->>U: "state saved to snapshot.json"

Loading

Key points the diagram encodes: restore happens after providers are built but before serving (so the first request already sees prior state), snapshot happens after graceful shutdown (so no in-flight request races the read), and the whole path goes through the driver interfaces — never raw struct serialization.

Alternatives not taken

• Generic reflection/gob snapshot of all stores — impossible: unexported fields + funcs/mutexes/nested stores.
• Per-service Snapshotter for all ~77 services now — a multi-thousand-line PR that can't be TDD/E2E-certified in one pass; deferred to follow-ups.
• Event-log / command replay — higher risk (needs raw-HTTP capture, response recording, deterministic IDs/clock); snapshots first.
• Metadata-only default — initially chosen for snapshot size, but a live CLI test showed it silently restores empty object bodies; flipped so bodies persist by default with --persist-metadata-only as the opt-out.

Docs / Tests / Playground

• Docs: docs/standalone-server.md — persistence section, flag table, fidelity notes.
• Unit (TDD): 4-category export→JSON→restore round-trip; metadata-only vs full; empty/missing snapshot → start empty; on-disk WriteFile/ReadFile with schema-version rejection.

Test plan

• go test -race ./persist/... ./cmd/cloudemu/...
• Full local CI: gofmt / build / vet / go test ./... / go mod tidy / golangci-lint = clean (CodeQL: only pre-existing findings, none in changed files).
• E2E via real aws CLI + the actual cloudemu start/stop/start lifecycle: create bucket+object, DynamoDB table+item, secret, EC2 instance → stop → start → all restored.
• Edge cases: DynamoDB N/BOOL/L types round-trip correctly; --persist-metadata-only drops bodies while keeping keys; idempotent re-stop.

Risk & Rollback

• Low: feature is opt-in and off by default; disabling --persist fully reverts behavior. A missing/corrupt/old-schema snapshot fails closed (unknown schema is rejected; missing file → start empty).

Conclusion

cloudemu start --persist now keeps object storage, NoSQL tables, secrets, and compute across restarts, in one readable multi-cloud file — a free, diffable analog to LocalStack's paid persistence.

Follow-ups (tracked under #107 / #335): per-service Snapshotter for the remaining services (VPC, IAM, Lambda, SQS, SNS, …); optional versioned/shareable snapshots ("cloud pods"); event-log replay for deterministic debugging.

[NitinKumar004]

Deep review — state persistence

Strong, well-scoped work. Verified in an isolated worktree at the PR head (#331 has since merged to development, so the real diff is the 5 persist/serve/lifecycle/docs files). Gates green: build / vet / go test -race ./persist/... ./cmd/cloudemu/ (ok both) / gofmt clean.

What's right:

• Interface-based snapshot is the correct design — the justification (unexported fields + mutexes/funcs/nested stores defeat any reflection codec) holds; routing through the seed drivers gives one diffable multi-cloud file.
• Mirror rule satisfied by construction — targets wires Storage/Database/Secrets/Compute for all three providers (serve.go:186/193/200); persist is provider-agnostic; k8s correctly excluded.
• Ordering is right — restore after rebuild() before serving; snapshot after Shutdown() so no in-flight request races the read.
• Fail-fast / fail-closed — --persist without --state-file errors; unknown schema rejected; missing file → start empty. Terminated instances skipped. Metadata default correctly flipped to bodies-on.
• No new numeric-fidelity loss — the DynamoDB decoder already stores numbers as float64 (server/aws/dynamodb/types.go:52), so a JSON snapshot round-trip is lossless relative to the emulator's own representation.

Holding on one Medium plus a dead-code Low; two doc-notes below.

M1 (Medium) — non-atomic snapshot write can brick start

Snapshot.WriteFile uses os.WriteFile (truncate-in-place, not atomic). If the write is interrupted — disk-full, OOM, or stop's SIGTERM→SIGKILL escalation (stopTimeout 12s) firing while a large default-bodies snapshot is still being written after the ≤10s graceful shutdown — the file is left truncated. The next start --persist then hits persist.ReadFile → json.Unmarshal fails → restoreState returns that error (not os.ErrNotExist) → runServe aborts. So a partial snapshot fails startup hard, on the exact stop→start path the feature exists for, until the user deletes the file by hand. Inline on WriteFile. Fix: write atomically (temp in the same dir + os.Rename), and/or treat an unparseable snapshot like a missing one (warn + start empty) rather than aborting.

L2 (Low) — table restore silently drops secondary indexes

persist.Table captures only Name/PartitionKey/SortKey (persist.go:74). A table created with a GSI loses it on restore, so a Query against that index that worked before stop fails after start. Please document as a known limitation (or capture indexes).

L3 (Low) — --persist-metadata-only restores zero-byte objects

By design metadata-only omits bodies; on restore PutObject(…, nil, …) (persist.go:341) recreates keys as empty objects — GetObject then returns 0 bytes. Worth an explicit doc note so it isn't a silent surprise.

No AI attribution. Requesting changes on M1 + the L1 dead-code removal (both cheap); L2/L3 are doc follow-ups.

[NitinKumar004]

M1 (Medium). os.WriteFile truncates in place, so an interrupted write (disk-full, OOM, or stop's SIGTERM→SIGKILL escalation while a large default-bodies snapshot is mid-write after graceful shutdown) leaves a truncated file. Then start --persist → ReadFile → json.Unmarshal fails → restoreState returns the error (not os.ErrNotExist) → runServe aborts, wedging startup on the very stop→start path this feature serves, until the user deletes the file. Write atomically: marshal to a temp file in the same dir, then os.Rename onto the target (rename is atomic on the same filesystem). Optionally also treat an unparseable snapshot in restoreState like a missing one — warn and start empty — so a corrupt file never blocks boot.

[thzgajendra]

Fixed both halves. WriteFile now marshals to a temp file in the same dir and os.Renames it onto the target — atomic on the same filesystem, so an interrupted write (disk-full/OOM/SIGKILL) leaves the previous snapshot or none, never a truncated one. And restoreState now fails open: a corrupt/truncated/unknown-schema snapshot logs a warning to stderr and starts empty rather than returning the error and aborting runServe. So a bad file can never wedge the stop→start path.

Verified end-to-end: garbling snapshot.json and running start --persist → server comes up (status running), no manual deletion needed; a normal cycle leaves valid JSON. Added TestRestoreStateIgnoresUnreadableFile (corrupt + missing → nil). 493792f

[NitinKumar004]

L1 (Low) — dead code. Nothing ever creates or writes an assets/ directory (object bodies are inlined as base64 in the single snapshot.json), so this RemoveAll cleans up a path that never exists, and assetsDir/assetsDirName (lines 26, 74) have no producer. Please drop the two symbols and this block — it reads as leftover scaffolding for a bodies-on-disk layout that wasn't taken.

[thzgajendra]

Removed — dropped assetsDir, assetsDirName, and the RemoveAll block. You're right, object bodies are base64-inlined in the single snapshot.json, so nothing ever writes an assets/ dir; it was leftover scaffolding for a bodies-on-disk layout we didn't take. 493792f

[thzgajendra]

Thanks for the thorough review — all points addressed in 493792f.

• M1 (atomic write + fail-open): WriteFile → temp file + os.Rename (atomic same-FS); restoreState now warns and starts empty on a corrupt/truncated/unknown-schema snapshot instead of aborting. E2E-verified a garbled snapshot.json no longer wedges start. New test TestRestoreStateIgnoresUnreadableFile.
• L1 (dead code): removed assetsDir/assetsDirName + the RemoveAll block.
• L2 (GSIs): now captured/restored from the table config (DescribeTable→CreateTable, both carry GSIs); new TestExportRestorePreservesGSIs proves the driver-level round-trip. One thing worth flagging that testing surfaced: the DynamoDB wire CreateTable intentionally ignores inline GlobalSecondaryIndexes (pre-existing, per the comment at dynamodb_lifecycle_test.go:1083), so GSIs created via the AWS API never reach the table config in the first place — not a persistence regression, and out of scope to fix here. The capture is correct and effective for the driver/in-process path; I softened the doc wording to "any secondary indexes present in a table's configuration" so it doesn't over-claim.
• L3 (metadata-only): explicit doc note that restored objects come back as zero-byte keys until re-uploaded.

Gate green: build / vet / gofmt / go test -race ./persist/... ./cmd/cloudemu/... / golangci-lint --new-from-rev = 0.

[NitinKumar004]

Approving — all findings resolved

Verified the hardening commit (493792f9) in an isolated worktree at the PR head; gate green (build / vet / go test -race ./persist/... ./cmd/cloudemu/ ok both / gofmt clean).

• M1 (both halves) fixed. Snapshot.WriteFile now writes to a temp file in the same dir and os.Renames onto the target — atomic on one filesystem, with clean temp-cleanup on every error path — so an interrupted write can never leave a truncated snapshot. And restoreState now fails open: a corrupt / unreadable / unknown-schema snapshot warns and starts empty instead of wedging startup on the stop→start path. Covered by TestRestoreStateIgnoresUnreadableFile. (Nice detail dropping the now-unused filePerm const — CreateTemp is 0600 by default.)
• L1 fixed — the dead assets/ cleanup (assetsDir/assetsDirName + the RemoveAll) is gone.
• L2 fixed — table secondary indexes are captured (Table.GSIs) and restored via CreateTable; TestExportRestorePreservesGSIs proves the round-trip.
• L3 — docs now note that --persist-metadata-only restores zero-byte objects.

One non-blocking, forward-looking heads-up: fail-open + overwrite-on-stop means that once SchemaVersion is ever bumped, an old-but-valid snapshot would be rejected → silent empty start → the next stop overwrites it → old data lost. Harmless today (schema v1, no migration), and fail-open was the right call for the corrupt-file case — just worth revisiting when v2 lands (e.g. rename a schema-rejected file aside rather than overwrite it).

Clean, well-tested turnaround — thanks. LGTM.