Investigate hydra-queue-runner S3 downloads

[edolstra]

Need to figure out why this happens:

Mar 09 18:57:48 chef hydra-queue-runner[11303]: uploaded ‘s3://nix-cache/nar/190m0683g4c5gqsy69ixc0hqm53kgblj2y6dj9cracf4k8vys65k.nar.xz’ (426701500 bytes) in 83605 ms
...
Mar 09 18:58:08 chef hydra-queue-runner[11303]: fetching ‘s3://nix-cache/nar/190m0683g4c5gqsy69ixc0hqm53kgblj2y6dj9cracf4k8vys65k.nar.xz’...
Mar 09 19:00:34 chef hydra-queue-runner[11303]: downloaded ‘s3://nix-cache/nar/190m0683g4c5gqsy69ixc0hqm53kgblj2y6dj9cracf4k8vys65k.nar.xz’ (426701500 bytes) in 145942 ms
...
Mar 09 19:00:41 chef hydra-queue-runner[11303]: exporting path ‘/nix/store/8w4bmwqk4cpg8zi43lmzz35jqki500fw-iso9660-image’ (439354208 bytes)
...
Mar 10 03:57:47 chef hydra-queue-runner[11303]: loading build 33064542 (nixos:release-16.03-small:nixos.iso_minimal.x86_64-linux)
Mar 10 03:57:47 chef hydra-queue-runner[11303]: fetching ‘s3://nix-cache/8w4bmwqk4cpg8zi43lmzz35jqki500fw.narinfo’...
Mar 10 03:57:47 chef hydra-queue-runner[11303]: downloaded ‘s3://nix-cache/8w4bmwqk4cpg8zi43lmzz35jqki500fw.narinfo’ (713 bytes) in 126 ms
...
Mar 10 04:05:10 chef hydra-queue-runner[11303]: exporting path ‘/nix/store/8w4bmwqk4cpg8zi43lmzz35jqki500fw-iso9660-image’ (439354208 bytes)

The first download is probably due to caching of negative lookups in Cloudfront, causing a build machine to be unable to fetch the path directly. This can be fixed by having the build machines access the underlying S3 bucket directly, rather than Cloudfront. (This would also save some money for EC2 build machines.)

The second download is probably due to the queue monitor thread encountering a new build of a cached path, requiring it to fetch the path to determine the build products. This could probably be avoided by copying the build products from the previous build in the Hydra database.

[edolstra]

Queue runner status after 76,000s of uptime:

      "s3" : {
         "costDollarApprox" : 12.7399,
         "get" : 346118,
         "getBytes" : 143014408725,
         "getSpeed" : 3.77933,
         "getTimeMs" : 36088182,
         "head" : 117607,
         "put" : 113421,
         "putBytes" : 389521856099,
         "putSpeed" : 2.44061,
         "putTimeMs" : 152206399
      }

[domenkozar]

So this is fixed?