Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Large builds consistently hang and crash the daemon #6968

Closed
anthonyalayo opened this issue Sep 7, 2023 · 4 comments
Closed

Large builds consistently hang and crash the daemon #6968

anthonyalayo opened this issue Sep 7, 2023 · 4 comments
Labels
area/build status/triage version/4.22.1

Comments

@anthonyalayo
Copy link

anthonyalayo commented Sep 7, 2023 •
edited

Description

I am working on a relatively large project, built using C++.

When attempting to run my build script,

RUN ./build.sh

Docker always crashes during the build step. The daemon becomes completely unresponsive, and I need to force quit all processes and restart.

The message I get on my build terminal is:

ERROR: failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF

Here are some miscellaneous messages I am seeing that would hopefully point in the right direction.

~/Library/Containers/com.docker.docker/Data/log/host ❯ cat * | grep -i EOF                                                                                                                                                      04:11:28 PM
2023/09/07 16:03:18 INFO: [transport] transport: closing: EOF
2023/09/07 16:03:18 INFO: [transport] transport: closing: EOF
[2023-09-07T23:03:18.857530000Z][com.docker.backend.docker][W] Resyncer volumes/grpcfuse: while watching docker events: unexpected EOF
[2023-09-07T23:03:18.858171000Z][com.docker.backend.docker][W] Resyncer idle: while watching docker events: unexpected EOF
2023/09/07 16:03:18 INFO: [transport] transport: closing: EOF
[2023-09-07T23:03:18.858118000Z][com.docker.backend.docker][W] Resyncer ports: while watching docker events: unexpected EOF
time="2023-09-07T23:03:18Z" level="info" msg="GoBackend: [2023-09-07T23:03:18.869312000Z][com.docker.backend.gvisor/virtualnetwork][E] EOF
time="2023-09-07T23:03:18Z" level="info" msg="GoBackend: [2023-09-07T23:03:18.877588000Z][com.docker.backend.apiproxy][W] ignored error: unexpected EOF
[2023-09-07T23:03:18.877650000Z][com.docker.backend.apiproxy][W] error reading response from engine for ""/events?filters=%7B%22event%22%3A%5B%22load%22%2C%22pull%22%2C%22tag%22%2C%22import%22%5D%7D"": unexpected EOF
[2023-09-07T23:03:18.877703000Z][com.docker.backend.apiproxy][W] ignored error: unexpected EOF
[2023-09-07T23:03:18.877728000Z][com.docker.backend.apiproxy][W] ignored error: unexpected EOF
[2023-09-07T23:03:18.877755000Z][com.docker.backend.apiproxy][W] ignored error: unexpected EOF
[2023-09-07T23:03:18.877705000Z][com.docker.backend.logwrite][W] unable to download logs from VM. Will retry in 5s. Error was: EOF
[2023-09-07T23:03:18.877844000Z][com.docker.backend.apiproxy][W] error reading response from engine for ""/events?filters=%7B%22event%22%3A%5B%22load%22%2C%22pull%22%2C%22tag%22%2C%22import%22%5D%7D"": unexpected EOF
[2023-09-07T23:03:18.877798000Z][com.docker.backend.apiproxy][W] ignored error: unexpected EOF
[2023-09-07T23:03:18.878109000Z][com.docker.backend.tasks.build][I] 2023-09-07 23:03:18 [I] error receiving build history event desktop-linux/desktop-linux: rpc error: code = Unavailable desc = error reading from server: EOF
[2023-09-07T23:03:18.878438000Z][com.docker.backend.tasks.build][I] 2023-09-07 23:03:18 [I] docker events error: unexpected EOF
time="2023-09-07T23:03:23Z" level="info" msg="GoBackend: [2023-09-07T23:03:23.900069000Z][com.docker.backend.main][E] otel-collector terminated unexpectedly: [2023-09-07T23:03:23.296135083Z][otel-collector][I] EOF
time="2023-09-07T23:07:21Z" level="info" msg="GoBackend: [2023-09-07T23:07:21.337797000Z][com.docker.backend.httpproxy][I] HTTP 120  : 400: reading request: EOF
time="2023-09-07T23:08:34Z" level="info" msg="GoBackend: [2023-09-07T23:08:34.483547000Z][com.docker.backend.httpproxy][I] HTTP 123  : 400: reading request: EOF
time="2023-09-07T23:09:27Z" level="info" msg="GoBackend: [2023-09-07T23:09:27.788540000Z][com.docker.backend.main][E] otel-collector terminated unexpectedly: [2023-09-07T23:09:27.788385333Z][otel-collector][I] EOF
time="2023-09-07T22:16:59Z" level="info" msg="GoBackend: [2023-09-07T22:16:59.963018000Z][com.docker.backend.httpproxy][I] HTTP 102  : 400: reading request: EOF
time="2023-09-07T22:17:00Z" level="info" msg="GoBackend: [2023-09-07T22:17:00.016201000Z][com.docker.backend.httpproxy][I] HTTP 103  : 400: reading request: EOF
time="2023-09-07T22:06:13Z" level="info" msg="GoBackend: [2023-09-07T22:06:13.845915000Z][com.docker.backend.main][E] otel-collector terminated unexpectedly: [2023-09-07T22:06:13.845610208Z][otel-collector][I] EOF
time="2023-09-07T22:16:47Z" level="info" msg="GoBackend: [2023-09-07T22:16:47.032024000Z][com.docker.backend.main][E] otel-collector terminated unexpectedly: [2023-09-07T22:16:47.031447125Z][otel-collector][I] EOF
[2023-09-07T23:07:21.337797000Z][com.docker.backend.httpproxy][I] HTTP 120  : 400: reading request: EOF
[2023-09-07T23:08:34.483547000Z][com.docker.backend.httpproxy][I] HTTP 123  : 400: reading request: EOF
[2023-09-07T23:09:27.788540000Z][com.docker.backend.main][E] otel-collector terminated unexpectedly: [2023-09-07T23:09:27.788385333Z][otel-collector][I] EOF
[2023-09-07T23:03:18.857530000Z][com.docker.backend.docker][W] Resyncer volumes/grpcfuse: while watching docker events: unexpected EOF
[2023-09-07T23:03:18.858171000Z][com.docker.backend.docker][W] Resyncer idle: while watching docker events: unexpected EOF
[2023-09-07T23:03:18.858118000Z][com.docker.backend.docker][W] Resyncer ports: while watching docker events: unexpected EOF
[2023-09-07T23:03:18.869312000Z][com.docker.backend.gvisor/virtualnetwork][E] EOF
[2023-09-07T23:03:18.877588000Z][com.docker.backend.apiproxy][W] ignored error: unexpected EOF
[2023-09-07T23:03:18.877650000Z][com.docker.backend.apiproxy][W] error reading response from engine for "/events?filters=%7B%22event%22%3A%5B%22load%22%2C%22pull%22%2C%22tag%22%2C%22import%22%5D%7D": unexpected EOF
[2023-09-07T23:03:18.877703000Z][com.docker.backend.apiproxy][W] ignored error: unexpected EOF
[2023-09-07T23:03:18.877728000Z][com.docker.backend.apiproxy][W] ignored error: unexpected EOF
[2023-09-07T23:03:18.877755000Z][com.docker.backend.apiproxy][W] ignored error: unexpected EOF
[2023-09-07T23:03:18.877705000Z][com.docker.backend.logwrite][W] unable to download logs from VM. Will retry in 5s. Error was: EOF
[2023-09-07T23:03:18.877844000Z][com.docker.backend.apiproxy][W] error reading response from engine for "/events?filters=%7B%22event%22%3A%5B%22load%22%2C%22pull%22%2C%22tag%22%2C%22import%22%5D%7D": unexpected EOF
[2023-09-07T23:03:18.877798000Z][com.docker.backend.apiproxy][W] ignored error: unexpected EOF
[2023-09-07T23:03:18.878109000Z][com.docker.backend.tasks.build][I] 2023-09-07 23:03:18 [I] error receiving build history event desktop-linux/desktop-linux: rpc error: code = Unavailable desc = error reading from server: EOF
[2023-09-07T23:03:18.878438000Z][com.docker.backend.tasks.build][I] 2023-09-07 23:03:18 [I] docker events error: unexpected EOF
[2023-09-07T23:03:23.900069000Z][com.docker.backend.main][E] otel-collector terminated unexpectedly: [2023-09-07T23:03:23.296135083Z][otel-collector][I] EOF
[2023-09-07T22:06:13.845915000Z][com.docker.backend.main][E] otel-collector terminated unexpectedly: [2023-09-07T22:06:13.845610208Z][otel-collector][I] EOF
[2023-09-07T22:16:47.032024000Z][com.docker.backend.main][E] otel-collector terminated unexpectedly: [2023-09-07T22:16:47.031447125Z][otel-collector][I] EOF
[2023-09-07T22:16:59.963018000Z][com.docker.backend.httpproxy][I] HTTP 102  : 400: reading request: EOF
[2023-09-07T22:17:00.016201000Z][com.docker.backend.httpproxy][I] HTTP 103  : 400: reading request: EOF
2023-09-07 23:03:18 [I] error receiving build history event desktop-linux/desktop-linux: rpc error: code = Unavailable desc = error reading from server: EOF
2023-09-07 23:03:18 [I] docker events error: unexpected EOF

~/Library/Containers/com.docker.docker/Data/log/host ❯ cat * | grep -i ERROR                                                                                                                                                                                                                                    04:14:52 PM
time="2023-09-07T23:10:01Z" level="info" msg="GoBackend: [2023-09-07T23:10:01.062092000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
time="2023-09-07T23:10:01Z" level="info" msg="GoBackend: [2023-09-07T23:10:01.080020000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
time="2023-09-07T23:10:01Z" level="info" msg="GoBackend: [2023-09-07T23:10:01.099232000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
time="2023-09-07T23:10:01Z" level="info" msg="GoBackend: [2023-09-07T23:10:01.117363000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
[2023-09-07T23:10:05.603245000Z][com.docker.backend.apiproxy][W] Error forwarding raw stream from dockerd: write unix <HOME>/.docker/run/docker.sock->: write: broken pipe
[2023-09-07T23:10:05.603275000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
time="2023-09-07T23:10:05Z" level="info" msg="GoBackend: [2023-09-07T23:10:05.603782000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
time="2023-09-07T23:10:05Z" level="info" msg="GoBackend: [2023-09-07T23:10:05.671343000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
time="2023-09-07T23:10:05Z" level="info" msg="GoBackend: [2023-09-07T23:10:05.690679000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
time="2023-09-07T23:10:05Z" level="info" msg="GoBackend: [2023-09-07T23:10:05.711173000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
time="2023-09-07T23:10:05Z" level="info" msg="GoBackend: [2023-09-07T23:10:05.728969000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
time="2023-09-07T23:11:11Z" level="info" msg="GoBackend: [2023-09-07T23:11:11.698341000Z][com.docker.backend.apiproxy][W] Error forwarding raw stream from dockerd: write unix <HOME>/.docker/run/docker.sock->: write: broken pipe
[2023-09-07T23:11:11.698360000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
time="2023-09-07T23:11:11Z" level="info" msg="GoBackend: [2023-09-07T23:11:11.710968000Z][com.docker.backend.apiproxy][W] Error forwarding raw stream from dockerd: write unix <HOME>/.docker/run/docker.sock->: write: broken pipe
[2023-09-07T23:11:11.711012000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
time="2023-09-07T23:11:11Z" level="info" msg="GoBackend: [2023-09-07T23:11:11.740704000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
time="2023-09-07T23:11:16Z" level="info" msg="GoBackend: time=""2023-09-07T16:11:16-07:00"" level=info msg=""error copying: readfrom tcp 127.0.0.1:6443->127.0.0.1:59169: write tcp 127.0.0.1:6443->127.0.0.1:59169: write: broken pipe""
time="2023-09-07T23:01:48Z" level="info" msg="GoBackend: [2023-09-07T23:01:48.693348000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
time="2023-09-07T23:01:48Z" level="info" msg="GoBackend: [2023-09-07T23:01:48.693550000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
time="2023-09-07T23:01:48Z" level="info" msg="GoBackend: [2023-09-07T23:01:48.772699000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
time="2023-09-07T23:01:48Z" level="info" msg="GoBackend: [2023-09-07T23:01:48.791205000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
time="2023-09-07T23:01:48Z" level="info" msg="GoBackend: [2023-09-07T23:01:48.810748000Z][com.docker.backend.apiproxy][W] Error forwarding raw stream from dockerd: write unix <HOME>/.docker/run/docker.sock->: write: broken pipe
[2023-09-07T23:01:48.810793000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
time="2023-09-07T23:01:48Z" level="info" msg="GoBackend: [2023-09-07T23:01:48.829357000Z][com.docker.backend.apiproxy][W] Error closing raw stream to client: close unix <HOME>/.docker/run/docker.sock->: shutdown: socket is not connected
[2023-09-07T23:03:08.850271000Z][com.docker.backend.procd][I] received state {IsPaused:false PauseText: PauseError: DockerAPI:running Docker:running Idle:running State:running Mode:linux}
[2023-09-07T23:03:08.850293000Z][com.docker.backend.procd][I] received state {IsPaused:false PauseText:Docker Desktop is resuming... PauseError: DockerAPI:running Docker:running Idle:running State:running Mode:linux}
[2023-09-07T23:03:08.852510000Z][com.docker.backend.procd][I] received state {IsPaused:false PauseText:Docker Desktop is running PauseError: DockerAPI:running Docker:running Idle:running State:running Mode:linux}
[2023-09-07T23:03:08.852528000Z][com.docker.backend.procd][I] received state {IsPaused:false PauseText:Docker Desktop is resuming... PauseError: DockerAPI:running Docker:running Idle:running State:running Mode:linux}
[2023-09-07T23:03:18.855699000Z][com.docker.backend.socketforward][W] host: volume-contents: error accepting local connection: context canceled
2023/09/07 16:03:18 INFO: [transport] transport: loopyWriter exiting with error: transport closed by client
2023/09/07 16:03:18 INFO: [transport] transport: loopyWriter exiting with error: transport closed by client
[2023-09-07T23:03:18.857653000Z][com.docker.backend.socketforward][W] host: lifecycle-server: error accepting local connection: context canceled
[2023-09-07T23:03:18.857701000Z][com.docker.backend.socketforward][W] host: docker-guest-proxy: error accepting local connection: context canceled
[2023-09-07T23:03:18.858064000Z][com.docker.backend.socketforward][W] host: nat-control: error accepting local connection: context canceled
[2023-09-07T23:03:18.858514000Z][com.docker.backend.socketforward][W] host: dns-forwarder: error accepting local connection: context canceled
2023/09/07 16:03:18 INFO: [transport] transport: loopyWriter exiting with error: transport closed by client
[2023-09-07T23:03:18.861568000Z][com.docker.backend.socketforward][W] host: diagnosticd: error accepting local connection: context canceled
[2023-09-07T23:03:18.861916000Z][com.docker.backend.socketforward][W] host: otel-collector-public: error accepting local connection: context canceled
[2023-09-07T23:03:18.862945000Z][com.docker.backend.socketforward][W] host: docker-api-proxy-control: error accepting local connection: context canceled
[2023-09-07T23:03:18.862919000Z][com.docker.backend.socketforward][W] host: debug-socketforwarder: error accepting local connection: context canceled
[2023-09-07T23:03:18.863284000Z][com.docker.backend.socketforward][W] host: devenv-volumes: error accepting local connection: context canceled
[2023-09-07T23:03:18.863211000Z][com.docker.backend.socketforward][W] host: debug-shell: error accepting local connection: context canceled
[2023-09-07T23:03:18.863614000Z][com.docker.backend.socketforward][W] host: filesystem-test: error accepting local connection: context canceled
[2023-09-07T23:03:18.863273000Z][com.docker.backend.socketforward][W] host: container-filesystem: error accepting local connection: context canceled
[2023-09-07T23:03:18.863551000Z][com.docker.backend.socketforward][W] host: otel-collector: error accepting local connection: context canceled
[2023-09-07T23:03:18.863648000Z][com.docker.backend.socketforward][W] host: artifactory: error accepting local connection: context canceled
[2023-09-07T23:03:18.863852000Z][com.docker.backend.socketforward][W] host: filesystem-event: error accepting local connection: context canceled
time="2023-09-07T23:03:18Z" level="info" msg="GoBackend: [2023-09-07T23:03:18.877588000Z][com.docker.backend.apiproxy][W] ignored error: unexpected EOF
[2023-09-07T23:03:18.877650000Z][com.docker.backend.apiproxy][W] error reading response from engine for ""/events?filters=%7B%22event%22%3A%5B%22load%22%2C%22pull%22%2C%22tag%22%2C%22import%22%5D%7D"": unexpected EOF
[2023-09-07T23:03:18.877703000Z][com.docker.backend.apiproxy][W] ignored error: unexpected EOF
[2023-09-07T23:03:18.877728000Z][com.docker.backend.apiproxy][W] ignored error: unexpected EOF
[2023-09-07T23:03:18.877755000Z][com.docker.backend.apiproxy][W] ignored error: unexpected EOF
[2023-09-07T23:03:18.877705000Z][com.docker.backend.logwrite][W] unable to download logs from VM. Will retry in 5s. Error was: EOF
[2023-09-07T23:03:18.877844000Z][com.docker.backend.apiproxy][W] error reading response from engine for ""/events?filters=%7B%22event%22%3A%5B%22load%22%2C%22pull%22%2C%22tag%22%2C%22import%22%5D%7D"": unexpected EOF
[2023-09-07T23:03:18.877914000Z][com.docker.backend.apiproxy][W] Error closing raw stream to dockerd: flushing frame host:27 Shutdown: write unix ->: use of closed network connection
time="2023-09-07T23:03:18Z" level="info" msg="GoBackend: [2023-09-07T23:03:18.877981000Z][com.docker.backend.apiproxy][W] ignored error: writing frame host:27 Close: write unix ->: use of closed network connection
[2023-09-07T23:03:18.878035000Z][com.docker.backend.apiproxy][W] Error closing raw stream to dockerd: writing frame host:1041 Shutdown: write unix ->: use of closed network connection
[2023-09-07T23:03:18.878053000Z][com.docker.backend.apiproxy][W] Error closing raw stream to dockerd: writing frame host:1042 Shutdown: write unix ->: use of closed network connection
[2023-09-07T23:03:18.877798000Z][com.docker.backend.apiproxy][W] ignored error: unexpected EOF
[2023-09-07T23:03:18.877674000Z][com.docker.backend.procd][I] received state {IsPaused:false PauseText:Docker Desktop is running PauseError: DockerAPI:running Docker:stopped Idle:running State:stopped Mode:linux}
[2023-09-07T23:03:18.878099000Z][com.docker.backend.apiproxy][W] ignored error: writing frame host:54 Close: write unix ->: use of closed network connection
[2023-09-07T23:03:18.878181000Z][com.docker.backend.apiproxy][W] ignored error: writing frame host:1042 Close: write unix ->: use of closed network connection
[2023-09-07T23:03:18.878023000Z][com.docker.backend.apiproxy][W] ignored error: writing frame host:1080 Close: write unix ->: use of closed network connection
[2023-09-07T23:03:18.878293000Z][com.docker.backend.apiproxy][W] ignored error: writing frame host:1041 Close: write unix ->: use of closed network connection                                                                                                                                                                                                                         

~/Library/Containers/com.docker.docker/Data/log/host ❯ cat * | grep -i FAILED                                                                                                                                                                                                                                   04:14:54 PM
time="2023-09-07T23:03:18Z" level="info" msg="GoBackend: [2023-09-07T23:03:18.854105000Z][com.docker.backend.Engines][I] graceful shutdown failed: lifecycle-server failed to shutdown the VM: Post ""http://ipc/vm/shutdown"": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
time="2023-09-07T23:09:22Z" level="error" msg="IPCClient: 0d4f0400-GoBackend (1516bc05) C<-S GET /ping (2.061009ms): Request failed: no response"
time="2023-09-07T23:09:22Z" level="error" msg="IPCClient: 0d4f0400-GoBackend (bca012eb) C<-S GET /ping (838.041306ns): Request failed: no response"
time="2023-09-07T23:09:23Z" level="error" msg="IPCClient: 0d4f0400-GoBackend (327c177c) C<-S GET /ping (1.050949ms): Request failed: no response"
time="2023-09-07T23:09:23Z" level="error" msg="IPCClient: 0d4f0400-GoBackend (26952316) C<-S GET /ping (1.582026ms): Request failed: no response"
2023/09/07 16:09:23 WARNING: [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
2023/09/07 16:09:23 WARNING: [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
2023/09/07 16:09:23 WARNING: [core] [Channel #5 SubChannel #6] grpc: addrConn.createTransport failed to connect to {
2023/09/07 16:09:23 WARNING: [core] [Channel #11 SubChannel #12] grpc: addrConn.createTransport failed to connect to {
time="2023-09-07T22:06:09Z" level="error" msg="IPCClient: d949354b-GoBackend (4443194a) C<-S GET /ping (2.074003ms): Request failed: no response"
time="2023-09-07T22:06:09Z" level="error" msg="IPCClient: d949354b-GoBackend (b96c71fb) C<-S GET /ping (784.993172ns): Request failed: no response"
2023/09/07 15:06:10 WARNING: [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
2023/09/07 15:06:10 WARNING: [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
2023/09/07 15:06:10 WARNING: [core] [Channel #5 SubChannel #6] grpc: addrConn.createTransport failed to connect to {
2023/09/07 15:06:10 WARNING: [core] [Channel #11 SubChannel #12] grpc: addrConn.createTransport failed to connect to {
time="2023-09-07T22:16:42Z" level="error" msg="IPCClient: 16b7e005-GoBackend (444d333a) C<-S GET /ping (1.556993ms): Request failed: no response"
time="2023-09-07T22:16:42Z" level="error" msg="IPCClient: 16b7e005-GoBackend (fbb79a13) C<-S GET /ping (803.947449ns): Request failed: no response"
2023/09/07 15:16:42 WARNING: [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
2023/09/07 15:16:42 WARNING: [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
2023/09/07 15:16:42 WARNING: [core] [Channel #5 SubChannel #6] grpc: addrConn.createTransport failed to connect to {
2023/09/07 15:16:42 WARNING: [core] [Channel #11 SubChannel #12] grpc: addrConn.createTransport failed to connect to {

Reproduce

Running docker build is all it takes for the hang and freeze to occur.

Expected behavior

The build can complete and the docker daemon doesn't freeze.

docker version

Client:
 Cloud integration: v1.0.35-desktop+001
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:32:30 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.22.1 (118664)
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:35:38 2023
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.5
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.1
    Path:     /Users/anthony.alayo/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2-desktop.1
    Path:     /Users/anthony.alayo/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/anthony.alayo/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/anthony.alayo/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.6
    Path:     /Users/anthony.alayo/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/anthony.alayo/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/anthony.alayo/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  0.20.0
    Path:     /Users/anthony.alayo/.docker/cli-plugins/docker-scout

Server:
 Containers: 38
  Running: 35
  Paused: 0
  Stopped: 3
 Images: 18
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 5.15.49-linuxkit-pr
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 6
 Total Memory: 7.765GiB
 Name: docker-desktop
 ID: fbfbb3b7-049c-49bc-a10c-2085c685f8aa
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 168
  Goroutines: 208
  System Time: 2023-09-07T23:17:23.804674845Z
  EventsListeners: 11
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: true
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

3620C384-21E6-44D7-AD3B-F3FF4207A9FD/20230907231903

Additional Info

No response
@anthonyalayo anthonyalayo mentioned this issue Sep 7, 2023
Closed
@djs55
Copy link
Contributor

djs55 commented Sep 8, 2023

@anthonyalayo could you upload a diagnostic, ideally when it's in the crashed state?

The normal cause of a large crashed build is the OOM killer because the VM ran out of memory. Could you check your settings and increase the maximum allocation?

@anthonyalayo
Copy link
Author

anthonyalayo commented Sep 8, 2023 •
edited

Thanks @djs55 , I think that may in fact be the case. The diagnostic attached on the ticket was uploaded through Docker's support feature, but I did just create another diagnostic that is hopefully cleaner.

The ID is: 3620C384-21E6-44D7-AD3B-F3FF4207A9FD/20230908204901.

Update: Interestingly, this attempt with Kubernetes disabled (which I do need enabled) had a slightly different effect. I still received:

ERROR: failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF

But it wasn't completely hosed. I was able to run other commands after it without having to force kill the daemon.

@anthonyalayo
Copy link
Author

Any idea what could be the problem @djs55 ?

@anthonyalayo
Copy link
Author

I just finished speaking with Docker support in parallel, and they did identify the issue as being out of memory. I didn't know how I could prove it before, but now I can. For anyone else who stumbles upon this issue, Activity Monitor on OSX will have an entry for "Virtual Machine Service".

image

If you see a similar failure to mine, hopefully this saves you some time. I wasn't aware that the VM would be consuming resources for the docker build process. Closing up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/build status/triage version/4.22.1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@djs55 @lorenrh @anthonyalayo