Many WARNING: DEADLINE_EXCEEDED: write logs from buildfarm-server.

[monaka]

I found many log lines from my bazel-buildfarm-server Pod.

Logs

bazel-buildfarm-server-6f886dbf9f-qn4vd buildfarm-server May 02, 2024 8:58:57 AM build.buildfarm.common.grpc.StubWriteOutputStream$2 onError                          
bazel-buildfarm-server-6f886dbf9f-qn4vd buildfarm-server WARNING: DEADLINE_EXCEEDED: write(uploads/85e8ddf8-870c-4774-b6c3-eebd4ec342da/blobs/bea4e1324009e61e800fcdbf
792a3ce8bcb02e512dd2c0523c13274758a12308/3084) on worker 172.20.136.234:8982 after 0 bytes of content                                                                 
bazel-buildfarm-server-6f886dbf9f-qn4vd buildfarm-server May 02, 2024 8:59:45 AM build.buildfarm.common.grpc.StubWriteOutputStream$2 onError                          
bazel-buildfarm-server-6f886dbf9f-qn4vd buildfarm-server WARNING: DEADLINE_EXCEEDED: write(uploads/c5b6d3f6-5948-4c4b-a961-310111a3368f/blobs/9e72388485c414119fc9b3f6
393a1269a682e8f6ab66fe94ac67d1354f87001c/3259) on worker 172.20.136.234:8982 after 0 bytes of content                                                                 
bazel-buildfarm-server-6f886dbf9f-qn4vd buildfarm-server May 02, 2024 9:02:28 AM build.buildfarm.common.grpc.StubWriteOutputStream$2 onError                          
bazel-buildfarm-server-6f886dbf9f-qn4vd buildfarm-server WARNING: DEADLINE_EXCEEDED: write(uploads/907c3a40-5312-4214-b59e-83ca9fa72589/blobs/f2b86a63079bcf83a10dd48a
f818d277031cb2811a3233b2f12da685490408e9/3431) on worker 172.20.136.234:8982 after 0 bytes of content

I couldn't find any logs from workers.
I checked the cache in workers and it looks like some are stored.

% kubectl exec -it -n bazel-buildfarm bazel-buildfarm-shard-worker-0 -- ls /tmp/worker/cache | wc -l
  387851

My environment is the same as #1716 (but just restarted Pods).
It uses AWS EBS GP3 as storage for caches.
So storage performance will be enough.

Could you give some hints to suppress WARNING: DEADLINE_EXCEEDED ? Thank you.

[monaka]

Is this issue similar to #1412 ?

[monaka]

I found a workaround.

kubectl exec -it -n bazel-buildfarm bazel-buildfarm-shard-worker-0 -- rm -fr /tmp/worker/cache
kubectl exec -it -n bazel-buildfarm bazel-buildfarm-shard-worker-0 -- mkdir /tmp/worker/cache
kubectl exec -it -n bazel-buildfarm bazel-buildfarm-shard-worker-1 -- rm -fr /tmp/worker/cache
kubectl exec -it -n bazel-buildfarm bazel-buildfarm-shard-worker-1 -- mkdir /tmp/worker/cache
kubectl rollout restart sts -n bazel-buildfarm bazel-buildfarm-shard-worker

I close this issue for now even though I'm not sure of the root cause.
I will reopen if I can find how to reproduce it.