Summary
Nothing serializes dependency installs for a component. When two installs of the same component overlap, npm's reify tears down a tree the other is building, fails with ENOTEMPTY, and aborts mid-teardown — leaving the component with no node_modules at all.
Observed on 5.1.x on a two-node hosted cluster. Three deploy_component attempts for one component landed within 13 minutes; the first two exited 0, the third died:
npm error code ENOTEMPTY
npm error syscall rmdir
npm error path /home/harperdb/harper/components/<component>/node_modules/node-llama-cpp/dist
npm error errno -39
npm error ENOTEMPTY: directory not empty, rmdir '.../node_modules/node-llama-cpp/dist'
End state: package-lock.json written, node_modules gone. The component kept running only because the worker still had the modules loaded in memory — the loss surfaced on the next restart, days later.
Why nothing prevents it
installApplications() (components/Application.ts) pushes every app into applicationInstallationPromises and awaits Promise.allSettled, so installs run concurrently by design.deployLifecycle (components/deployLifecycle.ts) ref-counts in-flight deploys per component name and exists to quiet file watchers. It deliberately allows overlap for the same name (0→1 fires start, 1→0 fires end, intermediate transitions are silent), so it is not — and was never intended as — a mutex around installs.- Operation-driven deploys each call
prepareApplication() independently, so two calls for the same component have nothing in common to contend on.
Retrying a deploy that appears stuck is the natural operator reaction, which makes this easy to trigger.
Secondary: the failed install leaks a child process
After the failure the container held a [npm install] <defunct> zombie, still unreaped ~24 hours later. nonInteractiveSpawn rejects on timeout via childProcess.kill(); something on the abort path isn't reaping the child.
Suggested direction
Serialize installs per component directory — a mutex or queue keyed on application.dirPath, so a second install of the same component waits rather than racing. A failed install should also not be able to leave a component with fewer dependencies than it started with; staging into a temp dir and swapping, or restoring on failure, would make the operation non-destructive.
Related: #1974 (a destroyed install can never recover) and #1975 (it reports healthy while broken). This bug creates the damage; those two explain why it persisted unnoticed.
🤖 Filed by Claude on behalf of @heskew
Summary
Nothing serializes dependency installs for a component. When two installs of the same component overlap, npm's reify tears down a tree the other is building, fails with
ENOTEMPTY, and aborts mid-teardown — leaving the component with nonode_modulesat all.Observed on 5.1.x on a two-node hosted cluster. Three
deploy_componentattempts for one component landed within 13 minutes; the first two exited 0, the third died:End state:
package-lock.jsonwritten,node_modulesgone. The component kept running only because the worker still had the modules loaded in memory — the loss surfaced on the next restart, days later.Why nothing prevents it
installApplications()(components/Application.ts) pushes every app intoapplicationInstallationPromisesand awaitsPromise.allSettled, so installs run concurrently by design.deployLifecycle(components/deployLifecycle.ts) ref-counts in-flight deploys per component name and exists to quiet file watchers. It deliberately allows overlap for the same name (0→1fires start,1→0fires end, intermediate transitions are silent), so it is not — and was never intended as — a mutex around installs.prepareApplication()independently, so two calls for the same component have nothing in common to contend on.Retrying a deploy that appears stuck is the natural operator reaction, which makes this easy to trigger.
Secondary: the failed install leaks a child process
After the failure the container held a
[npm install] <defunct>zombie, still unreaped ~24 hours later.nonInteractiveSpawnrejects on timeout viachildProcess.kill(); something on the abort path isn't reaping the child.Suggested direction
Serialize installs per component directory — a mutex or queue keyed on
application.dirPath, so a second install of the same component waits rather than racing. A failed install should also not be able to leave a component with fewer dependencies than it started with; staging into a temp dir and swapping, or restoring on failure, would make the operation non-destructive.Related: #1974 (a destroyed install can never recover) and #1975 (it reports healthy while broken). This bug creates the damage; those two explain why it persisted unnoticed.
🤖 Filed by Claude on behalf of @heskew