🐛 Bug Report — Runtime APIs: uncaught exception 'disconnected' with websockets on macOS

[aroman]

macOS ventura 13.5

✘ [ERROR] Uncaught (in response) Error: Network connection lost.
LLVMSymbolizer: error reading file: Unknown architecture named:
workerd/server/server.c++:2533: error: Uncaught exception: kj/async.c++:212: disconnected: worker_do_not_log; Request failed due to internal error
stack: 1035daf07 1044e28c3 1044e93c7 1044df95b 102c4081f 102c43f4f 102c43d83 102c43d6b 104551ba7 104551e03 10455055b 10455032b 102c381ab 18ea13f27 1044d9f1c 103602430 102ff9cf8 102ffa08b 102ee7144 1035c2d3c 1035c57b4
?? at ??:0:0

this log is unactionable to me — is this a bug in my workers code? in workerd? what does "internal error" mean (internal to what)?

doesn't help that it's not actually symbolicated, not sure how to get that working.

[takestairs]

I met it, too. The error message just like yours. The only difference is windows 10.

workerd/server/server.c++:2533: error: Uncaught exception: kj/async.c++:212: disconnected: remote.worker_do_not_log; Request failed due to internal error
stack: 7ff634911540 7ff6349304bb 7ff63491b02d 7ff634c812ff 7ff63491ac85 7ff63491c33f 7ff634921f36 7ff6349190b7 7ff633b2af52 7ff633b2e6d9 7ff633b2e572 7ff633b2e55c 7ff63499038c 7ff634996e66 7ff634990909 0 0 0

[ds300]

I'm seeing a similar error in local dev. Here's the full logs with --log-level debug

workerd/util/symbolizer.c++:99: warning: Not symbolizing stack traces because $LLVM_SYMBOLIZER is not set. To symbolize stack traces, set $LLVM_SYMBOLIZER to the location of the llvm-symbolizer binary. When running tests under bazel, use `--test_env=LLVM_SYMBOLIZER=<path>`.
workerd/io/worker-entrypoint.c++:217: info: exception = kj/compat/http.c++:3751: disconnected: WebSocket was aborted
stack: 105241cf8 10524208b 10512f144
workerd/io/worker-entrypoint.c++:217: info: exception = kj/async.c++:212: disconnected: operation canceled
stack: 105822f07 10672a8c3 1067313c7 10672795b 104e8881f 104e8bf4f 104e8bd83 104e8bd6b 106799ba7 106799e03 10679855b 10679832b 104e801ab 19a807e4f 106721f1c 10584a430 105241cf8 10524208b 10512f144
workerd/server/server.c++:2533: error: Uncaught exception: kj/async.c++:212: disconnected: worker_do_not_log; Request failed due to internal error
stack: 105822f07 10672a8c3 1067313c7 10672795b 104e8881f 104e8bf4f 104e8bd83 104e8bd6b 106799ba7 10

It happens often, but seemingly not all the time, when a websocket is closed (not sure by which end)

[kentonv]

Sorry, workerd's logging is still very bad. The logging was actually not designed to be actionable to application developers, instead it was designed to be actionable to us -- the engineers working on workerd itself. For the time being I would recommend ignoring workerd logs unless you are seeing a problem separately and trying to debug it.

[aroman]

@kentonv understood, is there any way to turn these logs off then? they're extremely noisy, triply so when you actually have LLVM_SYMBOLIZER set:

LLVMSymbolizer: error reading file: Unknown architecture named:
workerd/server/server.c++:2533: error: Uncaught exception: kj/async.c++:212: disconnected: worker_do_not_log; Request failed due to internal error
stack: 1030a2f07 103faa8c3 103fb13c7 103fa795b 10270881f 10270bf4f 10270bd83 10270bd6b 104019ba7 104019e03 10401855b 10401832b 1027001ab 18ea13f27 103fa1f1c 1030ca430 102ac1cf8 102ac208b 1029af144 10308ad3c 10308d7b4
?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

?? at ??:0:0

[paulgb]

Agreed, I'd love to have a way to turn workerd logs off. I run workerd in CI/CD tests and they are so noisy that it is difficult to diagnose actual issues.