Server build task stuck forever

[activatedgeek]

• code-server version: 1.691-vsc1.33.0

• OS Version: CentOS Linux release 7.6.1810

• node version: 8.10.0

• gcc version: 8.2.0

• yarn version: 1.15.2

This behavior also remains on the latest release 1.604-vsc1.32.0

Description

Running the server build task gets stuck. This is the log for a while now.

yarn run v1.15.2
$ ts-node -r tsconfig-paths/register build/tasks.ts build:server:binary
INFO  BUILD:SERVER:BINARY Starting...
INFO  VSCODE:INSTALL Starting...
INFO  VSCODE:INSTALL Completed! time=242716ms {"time":{"expected":9007199254740991,"ms":1554574909554}}
INFO  BUILD:BOOTSTRAP-FORK Starting...
INFO  BUILD:WEB Starting...
INFO  BUILD:SERVER:BUNDLE Starting...
INFO  BUILD:SERVER:BUNDLE Executing 'npm ["run","build"]' {"options":{"cwd":"/home/code-server/packages/server"}}
INFO  BUILD:APP:BROWSER Starting...
INFO  VSCODE:PATCH Starting...
INFO  BUILD:APP:BROWSER Executing 'npm ["run","build"]' {"options":{"cwd":"/home/code-server/packages/app/browser"}}
INFO  VSCODE:PATCH Executing 'git ["apply","--unidiff-zero","/home/code-server/scripts/vscode.patch"]' {"options":{"cwd":"/home/code-server/lib/vscode"}}
INFO  VSCODE:PATCH Completed! time=1993ms {"time":{"expected":9007199254740991,"ms":1554575152394}}
INFO  BUILD:BOOTSTRAP-FORK Executing 'npm ["run","build:bootstrap-fork"]' {"options":{"cwd":"/home/code-server/packages/vscode"}}
INFO  BUILD:WEB Executing 'npm ["run","build"]' {"options":{"cwd":"/home/code-server/packages/web"}}
INFO  BUILD:APP:BROWSER Completed! time=41679ms {"time":{"expected":9007199254740991,"ms":1554575152394}}
INFO  BUILD:SERVER:BINARY Still running...
INFO  BUILD:BOOTSTRAP-FORK Still running...
INFO  BUILD:WEB Still running...
INFO  BUILD:SERVER:BUNDLE Still running...
INFO  BUILD:SERVER:BINARY Still running...
INFO  BUILD:BOOTSTRAP-FORK Still running...
INFO  BUILD:WEB Still running...
INFO  BUILD:SERVER:BUNDLE Still running...
INFO  BUILD:SERVER:BINARY Still running...
INFO  BUILD:BOOTSTRAP-FORK Still running...
INFO  BUILD:WEB Still running...
INFO  BUILD:SERVER:BUNDLE Still running...
INFO  BUILD:SERVER:BINARY Still running...
INFO  BUILD:BOOTSTRAP-FORK Still running...
INFO  BUILD:WEB Still running...
INFO  BUILD:SERVER:BUNDLE Still running...
INFO  BUILD:SERVER:BINARY Still running...
INFO  BUILD:BOOTSTRAP-FORK Still running...
INFO  BUILD:WEB Still running...
INFO  BUILD:SERVER:BUNDLE Still running...
INFO  BUILD:SERVER:BINARY Still running...
INFO  BUILD:BOOTSTRAP-FORK Still running...
INFO  BUILD:WEB Still running...
INFO  BUILD:SERVER:BUNDLE Still running...
INFO  BUILD:SERVER:BINARY Still running...
INFO  BUILD:BOOTSTRAP-FORK Still running...
INFO  BUILD:WEB Still running...
INFO  BUILD:SERVER:BUNDLE Still running...
INFO  BUILD:SERVER:BINARY Still running...
INFO  BUILD:BOOTSTRAP-FORK Still running...
INFO  BUILD:WEB Still running...
INFO  BUILD:SERVER:BUNDLE Still running...
INFO  BUILD:SERVER:BINARY Still running...
INFO  BUILD:BOOTSTRAP-FORK Still running...
INFO  BUILD:WEB Still running...
INFO  BUILD:SERVER:BUNDLE Still running...
INFO  BUILD:SERVER:BINARY Still running...
INFO  BUILD:BOOTSTRAP-FORK Still running...
INFO  BUILD:WEB Still running...
INFO  BUILD:SERVER:BUNDLE Still running...
INFO  BUILD:SERVER:BINARY Still running...
INFO  BUILD:BOOTSTRAP-FORK Still running...
INFO  BUILD:WEB Still running...
INFO  BUILD:SERVER:BUNDLE Still running...
INFO  BUILD:SERVER:BINARY Still running...
INFO  BUILD:BOOTSTRAP-FORK Still running...
INFO  BUILD:WEB Still running...
INFO  BUILD:SERVER:BUNDLE Still running...
INFO  BUILD:SERVER:BINARY Still running...
INFO  BUILD:BOOTSTRAP-FORK Still running...
INFO  BUILD:WEB Still running...
INFO  BUILD:SERVER:BUNDLE Still running...
INFO  BUILD:SERVER:BINARY Still running...
INFO  BUILD:BOOTSTRAP-FORK Still running...
INFO  BUILD:WEB Still running...
INFO  BUILD:SERVER:BUNDLE Still running...

Steps to Reproduce

1. Run yarn
2. Run yarn task build:server:binary

[kylecarbs]

Very interesting. What happens if you go into the packages/web directory and run npm run build?

[activatedgeek]

Here's the error log. This keeps repeating for a long time (I believe it logs for each of the 63 threads in a non-deterministic order).

> @coder/web@ build /home/code-server/packages/web
> ../../node_modules/.bin/cross-env UV_THREADPOOL_SIZE=100 node --max-old-space-size=32384 ../../node_modules/webpack/bin/webpack.js --config ./webpack.config.js

Happy[ts]: Version: 5.0.1. Threads: 63
Happy[ts]: All set; signaling webpack to proceed.
HappyThread[ts:32]: unable to send to worker! Error [ERR_IPC_CHANNEL_CLOSED]: channel closed
    at ChildProcess.target.send (internal/child_process.js:588:16)
    at processSend (/home/code-server/node_modules/happypack/lib/processSend.js:8:15)
    at Object.UnbufferedFd.send (/home/code-server/node_modules/happypack/lib/UnbufferedFd.js:9:3)
    at send (/home/code-server/node_modules/happypack/lib/HappyThread.js:38:20)
    at Object.compile (/home/code-server/node_modules/happypack/lib/HappyThread.js:146:7)
    at Object.compile (/home/code-server/node_modules/happypack/lib/HappyThreadPool.js:82:14)
    at HappyPlugin.compile (/home/code-server/node_modules/happypack/lib/HappyPlugin.js:194:14)
    at Object.HappyLoader (/home/code-server/node_modules/happypack/lib/HappyLoader.js:31:15)
    at LOADER_EXECUTION (/home/code-server/node_modules/loader-runner/lib/LoaderRunner.js:119:14)
    at runSyncOrAsync (/home/code-server/node_modules/loader-runner/lib/LoaderRunner.js:120:4)
    at iterateNormalLoaders (/home/code-server/node_modules/loader-runner/lib/LoaderRunner.js:229:2)
    at Array.<anonymous> (/home/code-server/node_modules/loader-runner/lib/LoaderRunner.js:202:4)
    at Storage.finished (/home/code-server/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:43:16)
    at provider (/home/code-server/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:79:9)
    at /home/code-server/node_modules/graceful-fs/graceful-fs.js:90:16
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:511:3)
HappyThread[ts:34]: unable to send to worker! Error [ERR_IPC_CHANNEL_CLOSED]: channel closed
    at ChildProcess.target.send (internal/child_process.js:588:16)
    at processSend (/home/code-server/node_modules/happypack/lib/processSend.js:8:15)
    at Object.UnbufferedFd.send (/home/code-server/node_modules/happypack/lib/UnbufferedFd.js:9:3)
    at send (/home/code-server/node_modules/happypack/lib/HappyThread.js:38:20)
    at Object.compile (/home/code-server/node_modules/happypack/lib/HappyThread.js:146:7)
    at Object.compile (/home/code-server/node_modules/happypack/lib/HappyThreadPool.js:82:14)
    at HappyPlugin.compile (/home/code-server/node_modules/happypack/lib/HappyPlugin.js:194:14)
    at Object.HappyLoader (/home/code-server/node_modules/happypack/lib/HappyLoader.js:31:15)
    at LOADER_EXECUTION (/home/code-server/node_modules/loader-runner/lib/LoaderRunner.js:119:14)
    at runSyncOrAsync (/home/code-server/node_modules/loader-runner/lib/LoaderRunner.js:120:4)
    at iterateNormalLoaders (/home/code-server/node_modules/loader-runner/lib/LoaderRunner.js:229:2)
    at Array.<anonymous> (/home/code-server/node_modules/loader-runner/lib/LoaderRunner.js:202:4)
    at Storage.finished (/home/code-server/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:43:16)
    at provider (/home/code-server/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:79:9)
    at /home/code-server/node_modules/graceful-fs/graceful-fs.js:90:16
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:511:3)
HappyThread[ts:35]: unable to send to worker! Error [ERR_IPC_CHANNEL_CLOSED]: channel closed
    at ChildProcess.target.send (internal/child_process.js:588:16)
    at processSend (/home/code-server/node_modules/happypack/lib/processSend.js:8:15)
    at Object.UnbufferedFd.send (/home/code-server/node_modules/happypack/lib/UnbufferedFd.js:9:3)
    at send (/home/code-server/node_modules/happypack/lib/HappyThread.js:38:20)
    at Object.compile (/home/code-server/node_modules/happypack/lib/HappyThread.js:146:7)
    at Object.compile (/home/code-server/node_modules/happypack/lib/HappyThreadPool.js:82:14)
    at HappyPlugin.compile (/home/code-server/node_modules/happypack/lib/HappyPlugin.js:194:14)
    at Object.HappyLoader (/home/code-server/node_modules/happypack/lib/HappyLoader.js:31:15)
    at LOADER_EXECUTION (/home/code-server/node_modules/loader-runner/lib/LoaderRunner.js:119:14)
    at runSyncOrAsync (/home/code-server/node_modules/loader-runner/lib/LoaderRunner.js:120:4)
    at iterateNormalLoaders (/home/code-server/node_modules/loader-runner/lib/LoaderRunner.js:229:2)
    at Array.<anonymous> (/home/code-server/node_modules/loader-runner/lib/LoaderRunner.js:202:4)
    at Storage.finished (/home/code-server/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:43:16)
    at provider (/home/code-server/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:79:9)
    at /home/code-server/node_modules/graceful-fs/graceful-fs.js:90:16
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:511:3)

[activatedgeek]

@kylecarbs any luck here? Thanks!

[activatedgeek]

@kylecarbs Ping!

[nol166]

What happens if you remove --max-old-space-size=32384 from the package.json in packages/web?

[activatedgeek]

@nol166 The same error remains.

[fengzilong]

the same situation occurs on my mac when building image from the Dockfile

OS Version: macOS 10.13.2
Docker version: 17.09.1-ce-mac42

I have been waiting for a long time, but it seems never end

[lucacasonato]

AFAIK this happens when you don't have enough RAM available. I had set docker to use 2GB and it didn't work. After setting the RAM to 8GB it did. I did some more testing and the max seems to be between 2 and 3 GB of RAM.

[fengzilong]

Yeah, you are right. After setting the RAM to 8GB, it doesn't stuck anymore

[kylecarbs]

It's very interesting that the process never seems to die if its out of mem...

[activatedgeek]

I have like >32G of free memory on my remote workstation.

[sr229]

Fixed on latest tree. We already use a new build pipeline. This should take you no more than 2 minutes to build on a decent machine.