sccache occasionally crashes on Windows: "An existing connection was forcibly closed by the remote host. (os error 10054)"

[petamas]

We use sccache on Windows to build on our CI servers, and we really like it, as it sped up our builds by a factor of 2. However, sometimes (about once a month) it crashes mid-build with the following message:

sccache: error: failed to execute compile
sccache: caused by: error reading compile response from server
sccache: caused by: Failed to read response header
sccache: caused by: An existing connection was forcibly closed by the remote host. (os error 10054)

When we last ran into this issue, we set up some logging this way:

SCCACHE_LOG="debug,sccache::config=off" // we want to disable the "could not read config file" message, as it clutters up the output
SCCACHE_ERROR_LOG="C:/...snip..."
RUST_LOG_STYLE="never"

On the client side, it resulted in only one more line (printed for every sccache invocation, including the failing one):

[2022-01-15T04:14:10Z DEBUG sccache::commands] Server sent CompileStarted

At the end of the server's logfile, we can see this (I removed the actual argument lists as they may contain sensitive data):

[2022-01-15T04:14:14Z DEBUG sccache::server] handle_client: compile
[2022-01-15T04:14:14Z DEBUG sccache::server] check_compiler: Supported compiler
[2022-01-15T04:14:14Z DEBUG sccache::server] parse_arguments: Ok: [...snip...]
[2022-01-15T04:14:14Z DEBUG sccache::compiler::compiler] [qml_NumberText_qml.cpp.obj]: get_cached_or_compile: [...snip...]
[2022-01-15T04:14:14Z DEBUG sccache::compiler::msvc] preprocess: Some(...snip...)	
[2022-01-15T04:14:14Z DEBUG tokio_reactor] adding I/O source: 27221032976
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 16
[2022-01-15T04:14:14Z DEBUG tokio_reactor] adding I/O source: 27225227266
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 2
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 15
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 24
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 23
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 27
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 22
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 3
...snip...
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 19
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 21
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 15
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 24
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 0
[2022-01-15T04:14:14Z DEBUG tokio_reactor::registration] scheduling Read for: 0
thread '<unnamed>' panicked at 'internal error: entered unreachable code', C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\mio-named-pipes-0.1.7\src\lib.rs:633:14
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "PoisonError { inner: .. }"', C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\mio-named-pipes-0.1.7\src\lib.rs:297:46
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "PoisonError { inner: .. }"', C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\mio-named-pipes-0.1.7\src\lib.rs:467:43
stack backtrace:
   0:     0x7ff60374d8b9 - <unknown>
   1:     0x7ff60376feeb - <unknown>
   2:     0x7ff603743438 - <unknown>
   3:     0x7ff603752744 - <unknown>
   4:     0x7ff603752328 - <unknown>
   5:     0x7ff603752fff - <unknown>
   6:     0x7ff603752b65 - <unknown>
   7:     0x7ff60374e19f - <unknown>
   8:     0x7ff603752b19 - <unknown>
   9:     0x7ff60376e3c0 - <unknown>
  10:     0x7ff60376e203 - <unknown>
  11:     0x7ff6033f1869 - <unknown>
  12:     0x7ff60315050b - <unknown>
  13:     0x7ff60315810b - <unknown>
  14:     0x7ff60303688c - <unknown>
  15:     0x7ff60310d4eb - <unknown>
  16:     0x7ff60311189c - <unknown>
  17:     0x7ff602fd2878 - <unknown>
  18:     0x7ff602fcade6 - <unknown>
  19:     0x7ff60323f14b - <unknown>
  20:     0x7ff603237c7b - <unknown>
  21:     0x7ff6031e4293 - <unknown>
  22:     0x7ff6031fd25a - <unknown>
  23:     0x7ff6037f4b10 - <unknown>
  24:     0x7ff6037f444e - <unknown>
  25:     0x7ff6037f23c0 - <unknown>
  26:     0x7ff6037f3ebd - <unknown>
  27:     0x7ff6037f2839 - <unknown>
  28:     0x7ffbfabba78d - _chkstk
  29:     0x7ffbfab4535a - RtlUnwindEx
  30:     0x7ff6037f26ca - <unknown>
  31:     0x7ff6037f3545 - <unknown>
  32:     0x7ff6037f389b - <unknown>
  33:     0x7ff6037f3fb9 - <unknown>
  34:     0x7ff6037f2839 - <unknown>
  35:     0x7ffbfabba70d - _chkstk
  36:     0x7ffbfab449c3 - RtlImageNtHeaderEx
  37:     0x7ffbfab466d9 - RtlRaiseException
  38:     0x7ffbf7144f38 - RaiseException
  39:     0x7ff6037f2b78 - <unknown>
  40:     0x7ff603761ba1 - <unknown>
  41:     0x7ff603761b29 - <unknown>
  42:     0x7ff603753208 - <unknown>
  43:     0x7ff6037530b6 - <unknown>
  44:     0x7ff603752b65 - <unknown>
  45:     0x7ff60374e19f - <unknown>
  46:     0x7ff603752b19 - <unknown>
  47:     0x7ff60376e3c0 - <unknown>
  48:     0x7ff60376e203 - <unknown>
  49:     0x7ff6033f07dc - <unknown>
  50:     0x7ff6033f03e7 - <unknown>
  51:     0x7ff6033e227f - <unknown>
  52:     0x7ff6033dfcad - <unknown>
  53:     0x7ff602fc599f - <unknown>
  54:     0x7ff60304e1f4 - <unknown>
  55:     0x7ff603316457 - <unknown>
  56:     0x7ff60307b86d - <unknown>
  57:     0x7ff6031b4f15 - <unknown>
  58:     0x7ff6032c8e24 - <unknown>
  59:     0x7ff6033a5767 - <unknown>
  60:     0x7ff6033554a6 - <unknown>
  61:     0x7ff603337af9 - <unknown>
  62:     0x7ff603230f8d - <unknown>
  63:     0x7ff603124081 - <unknown>
  64:     0x7ff603317c6e - <unknown>
  65:     0x7ff6032fafbb - <unknown>
  66:     0x7ff603230e6d - <unknown>
  67:     0x7ff603383360 - <unknown>
  68:     0x7ff60307bccd - <unknown>
  69:     0x7ff603367a20 - <unknown>
  70:     0x7ff6032c767e - <unknown>
  71:     0x7ff603183cfa - <unknown>
  72:     0x7ff6031fd07a - <unknown>
  73:     0x7ff6031d1344 - <unknown>
  74:     0x7ff6032518ba - <unknown>
  75:     0x7ff603443a05 - <unknown>
  76:     0x7ff60343c0bf - <unknown>
  77:     0x7ff6032bc080 - <unknown>
  78:     0x7ff6031fa377 - <unknown>
  79:     0x7ff6032be73f - <unknown>
  80:     0x7ff60325084b - <unknown>
  81:     0x7ff603073d3f - <unknown>
  82:     0x7ff603297c2e - <unknown>
  83:     0x7ff6032953c3 - <unknown>
  84:     0x7ff603188500 - <unknown>
  85:     0x7ff60327f76b - <unknown>
  86:     0x7ff60317eed2 - <unknown>
  87:     0x7ff602ffedd7 - <unknown>
  88:     0x7ff602fc1076 - <unknown>
  89:     0x7ff602fc104c - <unknown>
  90:     0x7ff603753333 - <unknown>
  91:     0x7ff602fc1037 - <unknown>
  92:     0x7ff6037f1260 - <unknown>
  93:     0x7ffbf9bc84d4 - BaseThreadInitThunk
  94:     0x7ffbfab61781 - RtlUserThreadStart
thread panicked while panicking. aborting.

Any idea what may have caused these crashes, or what other logging we could turn on to gather more information?

[luser]

That's deep in the bowels of mio-named-pipes. It'd be worthwhile to see if there's a newer release that we could update sccache to use.

[dsanders11]

@luser, it looks like mio-named-pipes was removed as a dependency in 144eb0f.

I'm also seeing this crash, much more regularly, when using sccache with Chromium. It looks like the last release was a year ago, are there plans to cut a new release any time soon? At least with mio-named-pipes no longer being present the crash will either be fixed, or shift into mio itself.