(fix): release workflow binary memory after use during init

[justinkaseman]

Problem

During startup all workflows are loaded concurrently (up to maxConcurrency=12 in flight simultaneously). Each goroutine calls tryEngineCreate, which:

1. Decodes spec.Workflow (hex-encoded compressed WASM → 2× the compressed binary size as a Go string)
2. hex.DecodeString(decodedBinary) (the compressed binary, 1× size)
3. Calls engineFactory where host.NewModule compiles the binary into the wasmtime engine
4. Blocks on initDone while trigger subscriptions complete

spec.Workflow and decodedBinary are both pinned in tryEngineCreate's stack frame throughout the entire initDone waiting on trigger subscriptions (seconds per workflow).

12 goroutines x 3x compressed WASM each = 36x compressed WASM pinned continuously in Go heap

This can cause problems if we are running close to the maximum workflows.

Fix

This change releases them after they are finished being used milliseconds into the 15-second window.

After each batch the GC is also prompted to clean up.

[github-actions]

👋 justinkaseman, thanks for creating this pull request!

To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team.

Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks!

[github-actions]

✅ No conflicts with other open PRs targeting develop

[github-actions]

I see you updated files related to core. Please run make gocs in the root directory to add a changeset as well as in the text include at least one of the following tags:

• #added For any new functionality added.
• #breaking_change For any functionality that requires manual action for the node to boot.
• #bugfix For bug fixes.
• #changed For any change to the existing functionality.
• #db_update For any feature that introduces updates to database schema.
• #deprecation_notice For any upcoming deprecation functionality.
• #internal For changesets that need to be excluded from the final changelog.
• #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
• #removed For any functionality/config that is removed.
• #updated For any functionality that is updated.
• #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

[Copilot]

Pull request overview

Risk Rating: MEDIUM (touches workflow engine creation path; small diff but affects startup behavior and memory/liveness characteristics)

Reduces memory retained during concurrent workflow engine initialization by clearing large binary references earlier in tryEngineCreate, so they don’t remain reachable while waiting for initDone.

Changes:

• Clear spec.Workflow (hex-encoded binary) immediately after decoding to []byte.
• Clear decodedBinary after engineFactory returns to shorten the lifetime of the local reference.

[cl-sonarqube-production]

Quality Gate failed

Failed conditions
4.3% Technical Debt Ratio on New Code (required ≤ 4%)

See analysis details on SonarQube

[trunk-io]

     

View Full Report ↗︎ ⋅ Docs

[mchain0]

the spikes were observed even before introducing concurrency. still the diagnosis and the solution seems like a proper direction. it would be good to present a load-tests chart confirming smoother memory profile on reboot