Description
Describe the bug
Both the compress_build_logs and upload_logs_to_binary_cache config options try to exclusively handle log files fetched by Hydra from its builders. Both do different things with the log file, but both unlink() the original file once they're done, so it should be pretty obvious they can never work together.
This is especially bad since compress_build_logs is enabled by default.
To Reproduce
Enable both settings, notice that some logs are not uploaded to the cache, and notice that hydra-notify fails to bzip2 a large percentage of logs.
Expected behavior
Either make those not conflict, or error out when this configuration is found.
Screenshots
n/a
Hydra Server:
hydra.nixos.org
Additional context
This is likely the reason why hydra.nixos.org has had randomly missing logs for years. We've found 72M stranded .bz2 log files on the local drive of the Hydra server, which are all log files that likely haven't been uploaded to S3...