x/build/cmd/coordinator: mind combined memory usage when concurrently processing large payloads

[dmitshur]

As part of WriteTgz/PutTar operations, the coordinator handles payloads that may be large (for example, a Go tree tarball is 23 MB, and the x/website tree is even larger). If the entire payload size is read into memory during the processing of these operations, then multiple operations made in parallel can lead to coordinator using up much more memory than it does under a more average workload, and exceeding its current 8 GB memory limit.

For now, we can double that limit to give it more room to handle workloads that may happen as part of normal Go development and debugging, without getting killed due to OOM.

Keeping this issue open to see what else we may need to do. There are various opportunities to investigate and consider:

• see if these operations can be optimized to stream data rather than hold all bytes in memory during processing
• limit concurrency of these requests
• use disk when processing large tarballs
• keep a larger memory allocation

CC @golang/release, @thanm, @aclements.

[gopherbot]

Change https://go.dev/cl/383874 mentions this issue: cmd/coordinator: double memory limit

[dmitshur]

We've recently found that the large number of builders and golang.org/x repos can sometimes lead to a large amount of parallel work, which in turn consumes more memory than coordinator currently has.

As next steps, I'll send a CL to give coordinator more memory and later we can collect pprof samples to better understand how to reduce the memory use under such workloads.

[gopherbot]

Change https://go.dev/cl/409594 mentions this issue: cmd/coordinator: triple the memory allocation