[wanda] Normalize file mode in digest to 0644/0755 based on executable bit

[andrew-anyscale]

Git only tracks two file modes (100644 and 100755), but the digest previously included the full mode from os.Lstat(), which varies with the local umask (e.g. 0644 under umask 0022 vs 0664 under 0002). This caused digest mismatches across environments for identical content.

Now record() normalizes to 0644 (non-executable) or 0755 (executable) based solely on the owner-execute bit, matching git's two-state model.

Topic: raymake-digest

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
Signed-off-by: andrew andrew@anyscale.com

[andrew-anyscale]

Reviews in this chain:
└#468 [wanda] Normalize file mode in digest to 0644/0755 based on executable bit

[andrew-anyscale]

#	head	base	diff	date	summary
0	b75ca482	ed021020	diff	Mar 19 21:18 PM	2 files changed, 88 insertions(+), 26 deletions(-)
1	c3465507	4e874c20	diff	Mar 19 21:19 PM	0 files changed
2	a0e7974d	4e874c20	diff	Mar 19 21:21 PM	0 files changed
3	362b6ca8	4e874c20	diff	Mar 19 21:23 PM	0 files changed
4	348a63ff	690b2754	diff	Mar 19 21:28 PM	3 files changed, 66 insertions(+), 2 deletions(-)
5	f96f9660	374fcd96	diff	Mar 19 21:35 PM	1 file changed, 1 insertion(+), 5 deletions(-)
6	b82d2fc2	0f318da6	rebase	Mar 19 21:38 PM	0 files changed
7	39731ae6	6eb1cc52	rebase	Mar 19 21:44 PM	0 files changed
8	e3cea2e4	92a7f782	rebase	Mar 20 7:09 AM	0 files changed
9	830f3d4b	0331d931	rebase	Mar 20 10:29 AM	0 files changed
10	d2ddbb83	9007be10	diff	Mar 23 12:15 PM	0 files changed
11	75d66bab	9007be10	diff	Mar 24 13:12 PM	5 files changed, 30 insertions(+), 150 deletions(-)
12	1989bb77	9007be10	diff	Mar 27 13:52 PM	2 files changed, 8 insertions(+), 3 deletions(-)

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical inconsistency in how wanda generates cache keys between its digest and build commands. By implementing a mechanism to pre-compute and utilize dependency digests consistently across both operations, it ensures that the cache keys are identical, leading to more reliable and predictable build caching behavior. The changes primarily involve enhancing the build session to handle dependency graph traversal and digest management, thereby improving the overall integrity of the wanda build system.

Highlights

• Digest Consistency: The wanda digest subcommand now computes cache keys that match those generated by wanda build, resolving a previous inconsistency caused by differing dependency resolution and Docker image ID handling.
• Dependency Digest Pre-computation: A new computeDepDigests function has been introduced to walk the dependency graph in topological order and pre-compute each specification's input digest, storing them for consistent use.
• Digest-based Base Resolution: The resolveBases function now utilizes these pre-computed dependency digests for wanda-built dependencies, bypassing Docker/registry resolution and ensuring that the correct, consistent digest is used in cache key computation.
• Updated Digest Command Logic: The Digest command has been refactored to directly output the pre-computed root digest from the build session, aligning its behavior with the build process.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request aims to align the cache key generation between wanda build and wanda digest by pre-computing dependency digests. The changes look good overall, but I've found a couple of issues. There's a potential panic in the Digest command when no NamePrefix is configured, which I've marked as critical. Additionally, there's a bug in how dependency digests are looked up when image tags are present. I've provided suggestions for both issues.

[aslonnie]

it is by design that it is using the output result's digest rather than the input's digest. this is so that when A depends on B, even when B's input changed, as long as B's output has not changed, then the building of A can still be a cache hit.

I think this does raises a new issue, which is how to do pure input based caching.. like for ray core build cache..

we can either:

• save both indexes, one is for the action, one is for the entire dep graph
• take a shortcut and limit digests to only single step actions.. and we can also pin the base image (manylinux base) with image digest.

[aslonnie]

we can maybe talk more in person tomorrow.

[aslonnie]

hmm... when in the tarball file's record, can we still use 644 and 755 ?

otherwise this 0 and 1 mode bits is a bit weird.

essentially:

// we only check if a file has exec bit or not
// as a file in git only saves exec bit; other mode bits are determined
// by user's umask set up and might differ from system to system
mode := 0o600 // or should this be 0x644 ?
if (stat.Mode() & 0o100) != 0 {
  mode = 0o700 // or should this be 0x755 ?
}

I still feel that what actually matters more is the control of uid and gid.

[andrew-anyscale]

I still feel that what actually matters more is the control of uid and gid.

This is to resolve the issue of CI's digest not being reproducible locally. I.e., on the same commit and ensuring both rayci and epoch are set correctly, it was impossible to get a matching digest across CI and locally without manually updating permissions of files in the repo