[Scheduler] Exception notifying context listener

[jerrymarino]

Seeing some errors in the scheduler - directly proceeding a write failure:

[INFO ] io.grpc.Context$ExecutableListener deliver - Exception notifying context listener	
    io.grpc.StatusRuntimeException: UNKNOWN	
    at io.grpc.Status.asRuntimeException(Status.java:539)	
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)	
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:576)	
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)	
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:757)	
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:736)	
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)	
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)	
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)	
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)	
    at java.base/java.lang.Thread.run(Thread.java:1589)

Worker will tank with "Decompression error: Corrupted block detected"

[SEVERE ] build.buildfarm.common.services.WriteStreamObserver writeData - error writing data for uploads/6a747d66-b968-4cb7-9436-9c0cfe0540a7/compressed-blobs/zstd/0f09f96d2d3195689a3884604fc9ff1a218e1ab01b12578ecf28a817635ae1b9/1640367446
    java.io.IOException: Decompression error: Corrupted block detected
    at com.github.luben.zstd.ZstdInputStreamNoFinalizer.readInternal(ZstdInputStreamNoFinalizer.java:171)
    at com.github.luben.zstd.ZstdInputStreamNoFinalizer.read(ZstdInputStreamNoFinalizer.java:123)
    at com.google.protobuf.ByteString.readChunk(ByteString.java:540)
    at com.google.protobuf.ByteString.readFrom(ByteString.java:517)
    at com.google.protobuf.ByteString.readFrom(ByteString.java:485)
    at build.buildfarm.common.ZstdDecompressingOutputStream.write(ZstdDecompressingOutputStream.java:63)
    at build.buildfarm.cas.cfc.CASFileCache$8.write(CASFileCache.java:3008)
    at build.buildfarm.cas.cfc.WriteOutputStream.write(WriteOutputStream.java:43)
    at build.buildfarm.cas.cfc.WriteOutputStream.write(WriteOutputStream.java:43)
    at build.buildfarm.cas.cfc.CASFileCache$5.write(CASFileCache.java:1084)
    at build.buildfarm.cas.cfc.WriteOutputStream.write(WriteOutputStream.java:43)
    at build.buildfarm.cas.cfc.CASFileCache$UniqueWriteOutputStream.write(CASFileCache.java:817)
    at build.buildfarm.cas.cfc.CASFileCache$UniqueWriteOutputStream.write(CASFileCache.java:809)
    at com.google.protobuf.ByteString$LiteralByteString.writeTo(ByteString.java:1381)
    at build.buildfarm.common.services.WriteStreamObserver.writeData(WriteStreamObserver.java:420)
    at build.buildfarm.common.services.WriteStreamObserver.handleWrite(WriteStreamObserver.java:392)
    at build.buildfarm.common.services.WriteStreamObserver.handleRequest(WriteStreamObserver.java:307)
    at build.buildfarm.common.services.WriteStreamObserver.initialize(WriteStreamObserver.java:250)
    at build.buildfarm.common.services.WriteStreamObserver.onUncommittedNext(WriteStreamObserver.java:128)
    at build.buildfarm.common.services.WriteStreamObserver.onNext(WriteStreamObserver.java:107)
    at build.buildfarm.common.services.WriteStreamObserver.onNext(WriteStreamObserver.java:56)
    at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262)
    at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.message

[jerrymarino]

For reference the error comes from here: https://github.com/grpc/grpc-java/blob/56d1c63802fd9706fece4b144969bfc9bdcfc99c/context/src/main/java/io/grpc/Context.java#L1075