Tracking issue for "Remote Builds without the Bytes"

[buchgr]

This issue tracks the progress towards Remote Builds without the Bytes.

Milestones

• Implement basic support for not downloading intermediate artifacts (In Bazel 0.25)
  The feature can be enabled by adding the flag --experimental_remote_download_outputs=minimal to the remote caching / execution invocation. The feature mostly works, but comes with the limitations stated in the subsequent tasks.
• Fix local actions that need to fetch remote outputs (In Bazel 0.25)
  While Bazel can run most actions remotely there are a few dozen or so action implementations that currently can't. This can be problematic because such actions often require Bazel to download remote files which this change tries to avoid. An example of such an action is the SymlinkAction.
• Implement fetching of top level artifacts (In Bazel 0.26)
  Implement --experimental_remote_download_outputs=toplevel as described in the design document.
• Support downloading unzipped undeclared test outputs
  See --remote_download_toplevel --nozip_undeclared_test_outputs does not download any undeclared test outputs for passing tests #17884 for details
• Store per output file metadata in the persistent action cache
  The persistent action cache stores the state of Bazel's output base. It does not know about remotely stored files. We need to extend it to do so. See Store remote metadata in the on disk action cache #8248
• Builds without the Bytes fails on missing AC result
  See Builds without the Bytes fails on missing AC result #10880 for details
• Support remote cache eviction
  As detailed in the design document it's possible for Bazel to wrongly think that an output file exists on the remote system because it has been evicted ("garbage collected") in the meantime. If a Bazel action then depends on an evicted file there is currently no other way but to fail the build/test. See Remote Builds without the Bytes should support remote output eviction #8250
• Add up- and download stats to the profiler
  A user should have insight into what is being up/downloaded and why.
• Be compatible with the Build Event Service local file uploader (In Bazel 0.28.0)
  This change currently doesn't interact well with the BEP and BES. See Remote Builds without the Bytes should work with BES #8249
• Don't create a runfiles tree for remote actions
  For a test that runs remotely we currently create a runfiles tree in the output base with dangling symlinks that's unused. We need to fix the SymlinkTreeStrategy.
• Download targets of symlinks in runfiles
  See --remote_download_toplevel does not download symlink targets of runfiles #18249 for details
• Support symlinks into tree artifacts
  See Action input symlink into directory doesn't work with --remote_download_minimal #15678 for details
• ctx.actions.symlink actions are not constant time
  See FR?: ctx.actions.symlink action time scales w/ size of input; should be constant #14125 for details
• Support unresolved symlinks in action outputs
  See Symlinks in action outputs are not yet supported by --experimental_remote_download_outputs=minimal #13355 for details
• Do not create runfile trees unnecessarily when building without the bytes
  See Do not create runfile trees unnecessarily when building without the bytes #18580 for details
• Bazel query silently thrashes the analysis cache when --remote_download_minimal is enabled
  See Bazel query silently thrashes the analysis cache #10902 for details
• Build event file creation fails with symlinks when using --remote_download_minimal
  See Build event file creation fails with symlinks when using --remote_download_minimal #11942 for details
• Support for mobile-install command
  See mobile-install fails with BWOB #14523 for details
• Work seamlessly with IDEs

Related Issues

#1922 #5505

[iirina]

Please assign a priority (see the maintainers guide).

[ittaiz]

@buchgr Do I understand correctly that Fix local actions that need to fetch remote outputs is a pre-requisite for developer machines to have cache hits from CI?
Also can you elaborate a bit about Work seamlessly with IDEs? What won't work off the bat?
Alternatively will we be able to run on CI with --experimental_remote_fetch_outputs=minimal and developer machines without and still have cache hits?