[Bug] Queue workers use stale cached flow code after flow is updated — jobs hang until visibility timeout

[suphakonb]

What happened?

After updating a queue-bound flow's code (via UI "Save Changes" or PUT /api/flows/{id}),
queue workers continue executing the old stale code on every subsequent job.

The old code contained a publish step that called an unreachable internal host
(http://runloop-engine:8080). This causes Node.js http.request() to hang
indefinitely — because req.setTimeout() does NOT fire during TCP connection
establishment (only on idle connected sockets).

The queue's visibility timeout (300s) eventually kills the hung job,
producing the misleading error: "Execution cancelled".

Run Now (manual trigger) always uses fresh code from DB → completes in < 10s ✅
Queue-triggered jobs use cached old code → hang for exactly 5m 0s ❌

The DB confirmed the new code was saved correctly:

• GET /runloop/rl/api/flows/{id} returns the updated code
• updatedAt: 2026-05-07T09:49:16.693Z matches the last save

Disabling and re-enabling the queue (Workers=0 → Workers=1) does NOT fix it.
New goroutines still load from cache, not fresh from DB.

Steps to reproduce

1. Create a queue bound to a flow containing Node.js code that calls an
   unreachable host (e.g. http://runloop-engine:8080)
2. Enqueue a job → observe it hang for exactly visibility seconds → fails
   with "Execution cancelled"
3. Update the flow code via UI (SAVE CHANGES) to remove the broken call
4. Confirm via GET /api/flows/{id} that DB has the new code
5. Click Run Now on the flow → ✅ completes in < 10 seconds (uses new DB code)
6. Enqueue a new queue job → ❌ still hangs 5 minutes (still running old code)
7. Edit queue → disable (Enabled=false) → Save → re-enable → Save
8. Enqueue again → ❌ still hangs 5 minutes (cache survives worker restart)

Only confirmed workaround:
Create a brand-new flow via POST /api/flows with the same code,
then rebind with PATCH /api/queues/{name} using the new flowId.
Fresh workers have no cache → load code correctly from DB ✓

Version / commit

• RunLoop version: v0.1.0 BETA - Engine: ONLINE (observed at community.oneweb.tech) - Node.js runtime inside executor: v24.14.1 - Queue backend: PostgreSQL - Concurrency: 1 | max_attempts: 3 | visibility: 300s

How are you running RunLoop?

Local dev (npm run dev)

Logs / errors

Anything else?

Summary

Queue workers cache the bound flow's code when they start up.
After updating the flow code via the UI ("Save Changes") or the PUT /api/flows/{id} API,
queue workers continue executing the old (stale) code until the engine process is restarted.

"Run Now" (manual trigger) always uses the latest code from DB — working correctly.
Only queue-triggered executions are affected.

---

Environment

• RunLoop version: v0.1.0 BETA
• Queue backend: PostgreSQL
• Affected queue: pipeline-tasks
• Flow: Pipeline Executor (o6m9d5kp6yxgn0pqalg2qxhhx)

---

Steps to Reproduce

1. Create a queue bound to flow FlowA
2. FlowA has Node.js code that calls http://internal-host:8080 (unreachable)
3. Enqueue a job → it hangs for exactly visibility seconds → fails with "Execution cancelled"
4. Update FlowA code via UI (Save Changes) to remove the broken call
5. Confirm via GET /api/flows/{id} that DB has new code (updatedAt is recent)
6. Click "Run Now" → ✅ completes in < 10 seconds (uses new code from DB)
7. Enqueue another queue job → ❌ still hangs for 5 minutes (still using old cached code)
8. Disable queue (set Enabled=false), re-enable (Enabled=true) → ❌ still same behavior
9. Only fix found: create a brand new flow with POST /api/flows, then PATCH /api/queues/{name} with new flowId

---

Expected Behavior

When flow code is updated (via UI or API), queue workers should use the new code
on the next job pickup — without requiring engine restart or queue recreation.

The PATCH /api/queues/{name} API docs state:

"Changes apply on next worker pickup"

This should also apply to flow code changes, not just config parameters.

---

Actual Behavior

Queue workers continue running stale cached code indefinitely.
The only workarounds are:

• Restart the RunLoop engine process
• Create a new flow + rebind queue via PATCH /api/queues/{name} with new flowId

---

Root Cause Analysis

The RunLoop engine appears to load the flow definition (including Node.js code)
into the worker goroutine's memory at queue startup or first job pickup,
then caches it for all subsequent jobs.

Evidence:

• GET /runloop/rl/api/flows/{id} confirms DB has correct code
• updatedAt: 2026-05-07T09:49:16.693Z on flow matches our last save
• "Run Now" executions use fresh code (no caching)
• Queue executions use old code (with stale publish call to unreachable host)
• Last error on job after queue disable: "load flow: context cancelled" — confirms worker loads flow code at pickup time, but context was cancelled due to disable

---

Impact

• All queue jobs fail at exactly visibility seconds (300s default)
• Error appears as "Execution cancelled" — misleading, actual cause is stale code hang
• System appears to work (no startup error) but silently degrades

---

Suggested Fix

Option 1 (Preferred): Hot-reload on flow update
When PUT /api/flows/{id} or PATCH /api/queues/{name} is called,
signal active workers to reload the flow definition from DB on next job pickup.

Option 2: Force-reload button
Add a "Reload Workers" button on the Queue detail page that signals workers to reload flow code.

Option 3 (Workaround, already works):
PATCH /api/queues/{name} with a new flowId — forces workers to use updated definition.
Document this as the official workaround until hot-reload is implemented.

---

Additional: req.setTimeout() Warning

The affected flow used Node.js req.setTimeout(30000, ...) expecting it to protect against hangs.
However, req.setTimeout() is a socket idle timeout — it does NOT fire during TCP connection establishment.

If a target host drops TCP SYN packets (unreachable, firewall, wrong hostname),
req.setTimeout() will never fire, causing an infinite hang.

Recommendation: Add a documentation note or example in the Node.js executor
showing proper TCP connect timeout using the socket event:

req.on('socket', function(socket) {
  socket.setTimeout(connectTimeoutMs);
  socket.on('timeout', function() { req.destroy(new Error('Connect timeout')); });
});

[hacka0wi]

Did a deep read on the queue path before reaching for a fix. Sharing what I found because the conclusion is different from the title — and the difference matters for what we're actually shipping.

TL;DR

The "stale cached flow code" hypothesis doesn't survive a code read. The queue handler reloads the flow's flow_config from Postgres on every single delivery — there's no cache to invalidate. The symptom you observed (queue jobs hang for exactly 5 m 0 s while Run Now succeeds in <10 s after the same edit) almost certainly comes from the old broken messages still sitting in the queue and being redelivered after the visibility window expires.

PR #15 ships the operator escape hatch for that scenario.

Where the fresh-load happens

internal/queue/manager.go:163-195:

```go
return func(ctx context.Context, msg *Message) error {
// Load the flow config fresh on each delivery so queue workers pick
// up flow edits without a restart.
flowConfig, err := m.loadFlowConfig(ctx, q.FlowID)
...
}
```

loadFlowConfig is just SELECT flow_config FROM flows WHERE id=\$1. No in-memory cache, no prepared-statement plan cache that would survive a row update, no goroutine-captured snapshot. Each delivery hits the DB.

I also checked FlowExecutor — its struct has no per-flow cache, the only caches are cbManager (circuit breakers, keyed by node id) and plugins (registered plugin handlers, not flow code). Neither stores user flow nodes or scripts.

What probably happened in your repro

Re-reading steps 1–8 with this in mind:

1. Step 2: Enqueue msg#1 with the broken publish step → engine starts the Node.js subprocess → http.request() hangs because TCP-connect timeouts don't fire in Node by default → 300 s later visibility expires → msg#1 goes back to PENDING.
2. Step 6: You enqueue msg#2. The PG backend's claim is ORDER BY priority DESC, created_at (backend_postgres.go:144-149) — strictly FIFO when priorities are equal. msg#1 (older) wins every claim against msg#2.
3. So what you observed as "msg#2 hangs 5 min" was actually msg#1 being redelivered with the same broken script. After 3 retries (max_attempts) it dead-letters, and then msg#2 finally gets picked up — by that time you've already concluded the queue is broken.

The "rebind to a new flow id" workaround works because you also recreate the queue in that flow, which means job_queue_items rows reference an old queue_name and get ignored by the new consumer.

What I'm testing for

If you can paste the output of these two queries while the bug is reproducing, that confirms (or kills) the theory:

```sql
-- How many old broken msgs are still pending?
SELECT id, status, attempts, locked_until, created_at
FROM job_queue_items
WHERE queue_name = ''
ORDER BY created_at;

-- And which one is actively locked right now?
SELECT id, status, locked_by, locked_until
FROM job_queue_items
WHERE queue_name = '' AND status = 'PROCESSING';
```

What PR #15 does

POST /api/queues/:name/purge — deletes all PENDING rows, leaves in-flight PROCESSING rows alone, returns the deleted count. Use after a flow fix to clear the backlog instead of waiting ~15 min for max_attempts to drain.

If after running purge you still see queue jobs running old code, then there is a real cache somewhere I missed and I'd love the reproducer above to find it.

[hacka0wi]

Update — v0.1.2 is deployed on community.oneweb.tech with the operator escape hatch from PR #15.

How to clear stuck queue jobs

```bash
curl -X POST -H "Authorization: Bearer "
https://community.oneweb.tech/runloop/api/queues//purge
```

Returns {"success":true,"deleted":N} where N is the number of PENDING rows removed. PROCESSING rows are left alone so we never yank a row from under an in-flight worker.

Smoke test on prod just now

Caught a real-world stuck row on pipeline-tasks while validating the deploy:

```json
// before
{"PENDING": 1, "DLQ": 5, "oldestPendingSec": 26438} // 7.3 hours old (!)

// POST /api/queues/pipeline-tasks/purge
{"deleted": 1, "success": true}

// after
{"PENDING": 0, "DLQ": 5, "oldestPendingSec": 0}
```

That's exactly the symptom you described — a message stuck for hours after the queue went sideways.

What I'd love from you

If you can reproduce the original repro now (broken flow → enqueue → fix flow → enqueue again), please run:

```sql
-- snapshot of the queue while the new enqueue is "still hanging"
SELECT id, status, attempts, locked_until, created_at
FROM job_queue_items
WHERE queue_name = ''
ORDER BY created_at;
```

If the snapshot shows the new enqueue with attempts=0 while an older row is PROCESSING, the queue is doing what I described in the previous comment (stuck-in-retry, not cache). One POST .../purge should clear it.

If instead you see only one row and it's the new enqueue but the script logs old code, then there really is a cache I missed and I'd want to dig into the engine pod's runtime state next. Either way, the queue table snapshot tells us which.

Closing this out is up to you once you've verified — leaving the issue open until then.