Proposal: add support for artifact aliasing - #419
Conversation
This document proposes artifact aliasing in Bazel, allowing content to be addressable under multiple exec paths without copying. It details the API, semantics, and implications for backward compatibility and laydown strategies.
Adds ctx.actions.copy(output, target_file): makes target_file's content available at output (a second exec path), sharing its digest, with no byte copy and no spawn. Unlike ctx.actions.symlink, which plants a followable symbolic link that resolves back to the target's location, ctx.actions.copy only declares the relationship and defers materialization to sandbox/spawn creation -- the execution strategy places the target's content at the output's own path (hard link, copy, or the shared digest on remote execution), so its realpath stays inside the consuming tree (which tools such as Node.js require). Strategies that cannot materialize content (a purely symlinking sandbox or non-sandboxed local execution) fall back to a symlink. The original is never a declared input of the consumer -- it sees only the copy. Reuses the existing resolved-path metadata rather than adding new machinery: FileArtifactValue.isContentCopy() marks the value (ordinary symlinks are unchanged), a declaration-only CopyAction injects it, and the remote prefetcher and the local sandbox stage the content from the recorded source. Design: bazelbuild/proposals#419
Adds ctx.actions.copy(output, target_file): makes target_file's content available at output (a second exec path), sharing its digest, with no byte copy and no spawn. Unlike ctx.actions.symlink, which plants a followable symbolic link that resolves back to the target's location, ctx.actions.copy only declares the relationship and defers materialization to sandbox/spawn creation -- the execution strategy places the target's content at the output's own path (hard link, copy, or the shared digest on remote execution), so its realpath stays inside the consuming tree (which tools such as Node.js require). Strategies that cannot materialize content (a purely symlinking sandbox or non-sandboxed local execution) fall back to a symlink. The original is never a declared input of the consumer -- it sees only the copy. Reuses the existing resolved-path metadata rather than adding new machinery: FileArtifactValue.isContentCopy() marks the value (ordinary symlinks are unchanged), a declaration-only CopyAction injects it, and the remote prefetcher and the local sandbox stage the content from the recorded source. Design: bazelbuild/proposals#419
Adds ctx.actions.copy(output, target_file): makes target_file's content available at output (a second exec path), sharing its digest, with no byte copy and no spawn. Unlike ctx.actions.symlink, which plants a followable symbolic link that resolves back to the target's location, ctx.actions.copy only declares the relationship and defers materialization to sandbox/spawn creation -- the execution strategy places the target's content at the output's own path (hard link, copy, or the shared digest on remote execution), so its realpath stays inside the consuming tree (which tools such as Node.js require). Strategies that cannot materialize content (a purely symlinking sandbox or non-sandboxed local execution) fall back to a symlink. The original is never a declared input of the consumer -- it sees only the copy. Reuses the existing resolved-path metadata rather than adding new machinery: FileArtifactValue.isContentCopy() marks the value (ordinary symlinks are unchanged), a declaration-only CopyAction injects it, and the remote prefetcher and the local sandbox stage the content from the recorded source. Design: bazelbuild/proposals#419
Adds ctx.actions.copy(output, target_file): makes target_file's content available at output (a second exec path), sharing its digest, with no byte copy and no spawn. Unlike ctx.actions.symlink, which plants a followable symbolic link that resolves back to the target's location, ctx.actions.copy only declares the relationship and defers materialization to sandbox/spawn creation -- the execution strategy places the target's content at the output's own path (hard link, copy, or the shared digest on remote execution), so its realpath stays inside the consuming tree (which tools such as Node.js require). Strategies that cannot materialize content (a purely symlinking sandbox or non-sandboxed local execution) fall back to a symlink. The original is never a declared input of the consumer -- it sees only the copy. Reuses the existing resolved-path metadata rather than adding new machinery: FileArtifactValue.isContentCopy() marks the value (ordinary symlinks are unchanged), a declaration-only CopyAction injects it, and the remote prefetcher and the local sandbox stage the content from the recorded source. Design: bazelbuild/proposals#419
Adds ctx.actions.copy(output, target_file): makes target_file's content available at output (a second exec path), sharing its digest, with no byte copy and no spawn. Unlike ctx.actions.symlink, which plants a followable symbolic link that resolves back to the target's location, ctx.actions.copy only declares the relationship and defers materialization to sandbox/spawn creation -- the execution strategy places the target's content at the output's own path (hard link, copy, or the shared digest on remote execution), so its realpath stays inside the consuming tree (which tools such as Node.js require). Strategies that cannot materialize content (a purely symlinking sandbox or non-sandboxed local execution) fall back to a symlink. The original is never a declared input of the consumer -- it sees only the copy. Reuses the existing resolved-path metadata rather than adding new machinery: FileArtifactValue.isContentCopy() marks the value (ordinary symlinks are unchanged), a declaration-only CopyAction injects it, and the remote prefetcher and the local sandbox stage the content from the recorded source. Design: bazelbuild/proposals#419
Adds ctx.actions.copy(output, target_file): makes target_file's content available at output (a second exec path), sharing its digest, with no byte copy and no spawn. Unlike ctx.actions.symlink, which plants a followable symbolic link that resolves back to the target's location, ctx.actions.copy only declares the relationship and defers materialization to sandbox/spawn creation -- the execution strategy places the target's content at the output's own path (hard link, copy, or the shared digest on remote execution), so its realpath stays inside the consuming tree (which tools such as Node.js require). Strategies that cannot materialize content (a purely symlinking sandbox or non-sandboxed local execution) fall back to a symlink. The original is never a declared input of the consumer -- it sees only the copy. Reuses the existing resolved-path metadata rather than adding new machinery: FileArtifactValue.isContentCopy() marks the value (ordinary symlinks are unchanged), a declaration-only CopyAction injects it, and the remote prefetcher and the local sandbox stage the content from the recorded source. Design: bazelbuild/proposals#419
Add artifact aliasing proposal with API and semantics.
Explain how tree aliases handle symlinks and content staging.
|
Why not link this from the main list in |
|
potentially duplicated of #396 ? |
Will do, i missed that. |
Far be it from me to be overly critical, but you haven't yet? |
|
Fixed, and fixed an earlier issue with sandbox proposal. thanks @lberki Don't mind me asking, what's the next step to get this to be accepted? |
|
We'll have to find a reviewer (and some capacity to review the change). Let me see what I can do. |
|
Since this was sent as Markdown and not as a Google doc, should we comment on this PR or create a separate GitHub discussion? There is precedent for both :-) Edit: Left some early general comments here: #396 (comment) |
This document proposes artifact aliasing in Bazel, allowing content to be addressable under multiple exec paths without copying. It details the API, semantics, and implications for backward compatibility and laydown strategies.