Bazel uploads AC entries that reference non-existent (zero-size) CAS blob

[scele]

Description of the problem / feature request:

Bazel remote cache grpc backend seems to optimize away uploading the zero-sized CAS blob (sha256 = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855). This means that all remote cache implementations must add special logic to handle that digest, for example: buchgr/bazel-remote#233

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I'm running buildbarn/bb-storage@fb0be44 as a remote cache at grpc://localhost:9092.

$ cat BUILD 
cc_binary(
    name = "test",
    srcs = ["test.cc"],
)

$ bazel clean && bazel build //:test --remote_cache=grpc://localhost:9092 --experimental_remote_grpc_log=$PWD/grpc.log
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
INFO: Invocation ID: 8e4eba6d-d513-4afb-8260-63a2b182eb21
INFO: Analyzed target //:test (13 packages loaded, 47 targets configured).
INFO: Found 1 target...
Target //:test up-to-date:
  bazel-bin/test
INFO: Elapsed time: 0.524s, Critical Path: 0.25s
INFO: 2 processes: 2 linux-sandbox.
INFO: Build completed successfully, 6 total actions

Here is the decoded remote grpc log:

Click to expand

metadata {
  tool_details {
    tool_name: "bazel"
    tool_version: "2.2.0"
  }
  action_id: "capabilities"
  tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
  correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "build.bazel.remote.execution.v2.Capabilities/GetCapabilities"
details {
  get_capabilities {
    request {
      instance_name: "local"
    }
    response {
      cache_capabilities {
        digest_function: MD5
        digest_function: SHA1
        digest_function: SHA256
        digest_function: UNKNOWN_ENUM_VALUE_DigestFunction_5
        digest_function: UNKNOWN_ENUM_VALUE_DigestFunction_6
        action_cache_update_capabilities {
          update_enabled: true
        }
        symlink_absolute_path_strategy: ALLOWED
      }
      low_api_version {
        major: 2
      }
      high_api_version {
        major: 2
      }
    }
  }
}
start_time {
  seconds: 1585913715
  nanos: 962000000
}
end_time {
  seconds: 1585913715
  nanos: 966000000
}

---------------------------------------------------------
metadata {
  tool_details {
    tool_name: "bazel"
    tool_version: "2.2.0"
  }
  action_id: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
  tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
  correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
  code: 5
  message: "https://our-build-cache/ac/b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
}
method_name: "build.bazel.remote.execution.v2.ActionCache/GetActionResult"
details {
  get_action_result {
    request {
      instance_name: "local"
      action_digest {
        hash: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
        size_bytes: 142
      }
    }
  }
}
start_time {
  seconds: 1585913716
  nanos: 222000000
}
end_time {
  seconds: 1585913717
  nanos: 53000000
}

---------------------------------------------------------
metadata {
  tool_details {
    tool_name: "bazel"
    tool_version: "2.2.0"
  }
  action_id: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
  tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
  correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "build.bazel.remote.execution.v2.ContentAddressableStorage/FindMissingBlobs"
details {
  find_missing_blobs {
    request {
      instance_name: "local"
      blob_digests {
        hash: "f5a4eb0b457a32baefb3f8463dc35f3413fcf1aa9bd65905bcfc224ebe97f259"
        size_bytes: 1224
      }
      blob_digests {
        hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
      }
      blob_digests {
        hash: "decc7c70fee9e3d6598b7f5af926de0a5341a094ccc6bc5f1efb7c2072bb0df9"
        size_bytes: 88
      }
      blob_digests {
        hash: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
        size_bytes: 142
      }
      blob_digests {
        hash: "d70d1a8abd6791629f51490151fd7d2ee34b89e8147ba081ff2a4e1fa79a21ed"
        size_bytes: 989
      }
    }
    response {
      missing_blob_digests {
        hash: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
        size_bytes: 142
      }
      missing_blob_digests {
        hash: "d70d1a8abd6791629f51490151fd7d2ee34b89e8147ba081ff2a4e1fa79a21ed"
        size_bytes: 989
      }
      missing_blob_digests {
        hash: "decc7c70fee9e3d6598b7f5af926de0a5341a094ccc6bc5f1efb7c2072bb0df9"
        size_bytes: 88
      }
      missing_blob_digests {
        hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
      }
      missing_blob_digests {
        hash: "f5a4eb0b457a32baefb3f8463dc35f3413fcf1aa9bd65905bcfc224ebe97f259"
        size_bytes: 1224
      }
    }
  }
}
start_time {
  seconds: 1585913717
  nanos: 143000000
}
end_time {
  seconds: 1585913719
  nanos: 726000000
}

---------------------------------------------------------
metadata {
  tool_details {
    tool_name: "bazel"
    tool_version: "2.2.0"
  }
  action_id: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
  tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
  correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "google.bytestream.ByteStream/Write"
details {
  write {
    resource_names: "local/uploads/b41d91ce-f43e-4d20-bf4d-740faae6f32f/blobs/f5a4eb0b457a32baefb3f8463dc35f3413fcf1aa9bd65905bcfc224ebe97f259/1224"
    num_writes: 1
    bytes_sent: 1224
    response {
      committed_size: 1224
    }
  }
}
start_time {
  seconds: 1585913719
  nanos: 729000000
}
end_time {
  seconds: 1585913720
  nanos: 463000000
}

---------------------------------------------------------
metadata {
  tool_details {
    tool_name: "bazel"
    tool_version: "2.2.0"
  }
  action_id: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
  tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
  correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "google.bytestream.ByteStream/Write"
details {
  write {
    resource_names: "local/uploads/40f630ae-528e-41a5-bb77-d924e7dc01b2/blobs/decc7c70fee9e3d6598b7f5af926de0a5341a094ccc6bc5f1efb7c2072bb0df9/88"
    num_writes: 1
    bytes_sent: 88
    response {
      committed_size: 88
    }
  }
}
start_time {
  seconds: 1585913719
  nanos: 728000000
}
end_time {
  seconds: 1585913720
  nanos: 479000000
}

---------------------------------------------------------
metadata {
  tool_details {
    tool_name: "bazel"
    tool_version: "2.2.0"
  }
  action_id: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
  tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
  correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "google.bytestream.ByteStream/Write"
details {
  write {
    resource_names: "local/uploads/2f6c53ee-4bc1-4000-a476-ffc67bd074a0/blobs/d70d1a8abd6791629f51490151fd7d2ee34b89e8147ba081ff2a4e1fa79a21ed/989"
    num_writes: 1
    bytes_sent: 989
    response {
      committed_size: 989
    }
  }
}
start_time {
  seconds: 1585913719
  nanos: 727000000
}
end_time {
  seconds: 1585913720
  nanos: 485000000
}

---------------------------------------------------------
metadata {
  tool_details {
    tool_name: "bazel"
    tool_version: "2.2.0"
  }
  action_id: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
  tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
  correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "google.bytestream.ByteStream/Write"
details {
  write {
    resource_names: "local/uploads/1ac78d17-a236-42b4-9215-522abf656ad5/blobs/b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1/142"
    num_writes: 1
    bytes_sent: 142
    response {
      committed_size: 142
    }
  }
}
start_time {
  seconds: 1585913719
  nanos: 727000000
}
end_time {
  seconds: 1585913720
  nanos: 496000000
}

---------------------------------------------------------
metadata {
  tool_details {
    tool_name: "bazel"
    tool_version: "2.2.0"
  }
  action_id: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
  tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
  correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "build.bazel.remote.execution.v2.ActionCache/UpdateActionResult"
details {
  update_action_result {
    request {
      instance_name: "local"
      action_digest {
        hash: "b2392350ab3f76698c8487fe49635fdbb461b958457fcc569349ec7769640ef1"
        size_bytes: 142
      }
      action_result {
        output_files {
          path: "bazel-out/k8-fastbuild/bin/_objs/test/test.pic.o"
          digest {
            hash: "f5a4eb0b457a32baefb3f8463dc35f3413fcf1aa9bd65905bcfc224ebe97f259"
            size_bytes: 1224
          }
        }
        output_files {
          path: "bazel-out/k8-fastbuild/bin/_objs/test/test.pic.d"
          digest {
            hash: "decc7c70fee9e3d6598b7f5af926de0a5341a094ccc6bc5f1efb7c2072bb0df9"
            size_bytes: 88
          }
        }
        stdout_digest {
          hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
        }
        stderr_digest {
          hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
        }
      }
    }
    response {
      output_files {
        path: "bazel-out/k8-fastbuild/bin/_objs/test/test.pic.o"
        digest {
          hash: "f5a4eb0b457a32baefb3f8463dc35f3413fcf1aa9bd65905bcfc224ebe97f259"
          size_bytes: 1224
        }
      }
      output_files {
        path: "bazel-out/k8-fastbuild/bin/_objs/test/test.pic.d"
        digest {
          hash: "decc7c70fee9e3d6598b7f5af926de0a5341a094ccc6bc5f1efb7c2072bb0df9"
          size_bytes: 88
        }
      }
      stdout_digest {
        hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
      }
      stderr_digest {
        hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
      }
    }
  }
}
start_time {
  seconds: 1585913720
  nanos: 497000000
}
end_time {
  seconds: 1585913721
  nanos: 419000000
}

---------------------------------------------------------
metadata {
  tool_details {
    tool_name: "bazel"
    tool_version: "2.2.0"
  }
  action_id: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
  tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
  correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
  code: 5
  message: "https://our-build-cache/ac/3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
}
method_name: "build.bazel.remote.execution.v2.ActionCache/GetActionResult"
details {
  get_action_result {
    request {
      instance_name: "local"
      action_digest {
        hash: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
        size_bytes: 142
      }
    }
  }
}
start_time {
  seconds: 1585913721
  nanos: 427000000
}
end_time {
  seconds: 1585913722
  nanos: 309000000
}

---------------------------------------------------------
metadata {
  tool_details {
    tool_name: "bazel"
    tool_version: "2.2.0"
  }
  action_id: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
  tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
  correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "build.bazel.remote.execution.v2.ContentAddressableStorage/FindMissingBlobs"
details {
  find_missing_blobs {
    request {
      instance_name: "local"
      blob_digests {
        hash: "cb32cfd8d6feb84baf601bdc875110a64149892e9fee976478c490cdb2e6c74c"
        size_bytes: 7856
      }
      blob_digests {
        hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
      }
      blob_digests {
        hash: "5d42134e4bf9156e629091745de7d258decaec422022b1b1399df3c81c4291bd"
        size_bytes: 358
      }
      blob_digests {
        hash: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
        size_bytes: 142
      }
    }
    response {
      missing_blob_digests {
        hash: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
        size_bytes: 142
      }
      missing_blob_digests {
        hash: "5d42134e4bf9156e629091745de7d258decaec422022b1b1399df3c81c4291bd"
        size_bytes: 358
      }
      missing_blob_digests {
        hash: "cb32cfd8d6feb84baf601bdc875110a64149892e9fee976478c490cdb2e6c74c"
        size_bytes: 7856
      }
      missing_blob_digests {
        hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
      }
    }
  }
}
start_time {
  seconds: 1585913722
  nanos: 437000000
}
end_time {
  seconds: 1585913724
  nanos: 692000000
}

---------------------------------------------------------
metadata {
  tool_details {
    tool_name: "bazel"
    tool_version: "2.2.0"
  }
  action_id: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
  tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
  correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "google.bytestream.ByteStream/Write"
details {
  write {
    resource_names: "local/uploads/403d946f-38b2-45f0-8afa-f9ae704160ac/blobs/cb32cfd8d6feb84baf601bdc875110a64149892e9fee976478c490cdb2e6c74c/7856"
    num_writes: 1
    bytes_sent: 7856
    response {
      committed_size: 7856
    }
  }
}
start_time {
  seconds: 1585913724
  nanos: 695000000
}
end_time {
  seconds: 1585913725
  nanos: 430000000
}

---------------------------------------------------------
metadata {
  tool_details {
    tool_name: "bazel"
    tool_version: "2.2.0"
  }
  action_id: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
  tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
  correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "google.bytestream.ByteStream/Write"
details {
  write {
    resource_names: "local/uploads/c39d05f9-349e-4318-b6eb-0c97a9f34cea/blobs/3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0/142"
    num_writes: 1
    bytes_sent: 142
    response {
      committed_size: 142
    }
  }
}
start_time {
  seconds: 1585913724
  nanos: 693000000
}
end_time {
  seconds: 1585913725
  nanos: 430000000
}

---------------------------------------------------------
metadata {
  tool_details {
    tool_name: "bazel"
    tool_version: "2.2.0"
  }
  action_id: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
  tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
  correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "google.bytestream.ByteStream/Write"
details {
  write {
    resource_names: "local/uploads/59868396-c124-49ed-ba9e-e50a2b1f2de1/blobs/5d42134e4bf9156e629091745de7d258decaec422022b1b1399df3c81c4291bd/358"
    num_writes: 1
    bytes_sent: 358
    response {
      committed_size: 358
    }
  }
}
start_time {
  seconds: 1585913724
  nanos: 694000000
}
end_time {
  seconds: 1585913725
  nanos: 443000000
}

---------------------------------------------------------
metadata {
  tool_details {
    tool_name: "bazel"
    tool_version: "2.2.0"
  }
  action_id: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
  tool_invocation_id: "549975db-fdf6-4fcf-8dd8-e1232062e7f6"
  correlated_invocations_id: "6814a271-36e6-4fc7-ac25-ac5ecce1c1a7"
}
status {
}
method_name: "build.bazel.remote.execution.v2.ActionCache/UpdateActionResult"
details {
  update_action_result {
    request {
      instance_name: "local"
      action_digest {
        hash: "3b4517619716fe39beeadadef3ef076d697c39a92a89d1541aaef81fc578e2e0"
        size_bytes: 142
      }
      action_result {
        output_files {
          path: "bazel-out/k8-fastbuild/bin/test"
          digest {
            hash: "cb32cfd8d6feb84baf601bdc875110a64149892e9fee976478c490cdb2e6c74c"
            size_bytes: 7856
          }
          is_executable: true
        }
        stdout_digest {
          hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
        }
        stderr_digest {
          hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
        }
      }
    }
    response {
      output_files {
        path: "bazel-out/k8-fastbuild/bin/test"
        digest {
          hash: "cb32cfd8d6feb84baf601bdc875110a64149892e9fee976478c490cdb2e6c74c"
          size_bytes: 7856
        }
        is_executable: true
      }
      stdout_digest {
        hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
      }
      stderr_digest {
        hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
      }
    }
  }
}
start_time {
  seconds: 1585913725
  nanos: 444000000
}
end_time {
  seconds: 1585913725
  nanos: 658000000
}

---------------------------------------------------------

The log shows that even though the remote cache reports e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 as missing, bazel does not attempt to upload it. Yet, it is referencing it from the ActionResult messages that it uploads.

In normal use this would not be noticed, since on a remote cache hit, I believe bazel grpc cache backend will notice that the referenced CAS blob has zero size and will not attempt to download it. However, if I try to consume the "corrupt" AC entry using the http protocol, bazel will try to download the e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 CAS blob, gets back 404, and will fall back to local execution.

What operating system are you running Bazel on?

Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:        18.04
Codename:       bionic

What's the output of bazel info release?

release 2.2.0

Have you found anything relevant by searching the web?

Seems that some remote cache implementations have added special handling for this, see e.g. buchgr/bazel-remote#233

[EdSchouten]

Relatedly, I've also received reports that Bazel does not upload empty files that are specified inside input roots of build actions.

Note that what Bazel does would be perfectly fine, as long as we make sure that the REv2 specification captures this. Right now REv2 mentions no such thing.

[mostynb]

Submitted a PR to clarify this in the spec: bazelbuild/remote-apis#131

[EdSchouten]

I think this issue can be closed now, as bazelbuild/remote-apis#131 has been merged.

[scele]

Yep, closing!

[coeuvre]

This behavior is a side-effect of cc2b3ec:

                if (committedOffset.get() < chunker.getSize()) {
                  return ctx.call(() -> callAndQueryOnFailure(committedOffset, progressiveBackoff));
                }

chunker.getSize() will return 0 if the blob is zero-size in which case Bazel wouldn't call the upload function.

IIUC, the change didn't intend to optimize away uploading zero-sized blob. Otherwise, we should add the special case handling at RemoteCache just like what we did for downloading.

That said, the behavior is existed for a long time and the spec is updated accordingly. I am going to add the special case handling at RemoteCache so Bazel won't upload zero-sized blob to other remote cache backends as well.