Fix: IntrinsicError when a non-directory PATH entry is used with system_binary

[jbylund]

Closes #21990

Problem

Pants crashed with IntrinsicError: Not a directory (os error 20) when a
system_binary target (or any binary lookup) included a non-directory entry in
its search path. Two distinct shapes of the same underlying bug:

1. foo/bar on PATH where foo is a file. Calling path_metadata_request
   on foo/bar hits ENOTDIR in the OS. That error was not caught in the Rust
   layer, so it surfaced as an uncaught IntrinsicError.

2. A symlink-to-file on PATH — the case from the issue report. The first
   path_metadata_request succeeds and returns PathMetadataKind::SYMLINK,
   but the Python code then tried to look up symlink/binary_name, which also
   hits ENOTDIR.

Fix

src/rust/fs/src/posixfs.rs — path_metadata now accepts a follow_symlinks
flag. When true, it calls tokio::fs::metadata (which follows symlinks) instead
of tokio::fs::symlink_metadata, so a symlink-to-directory appears as a
Directory rather than a Symlink. It also continues to treat ENOTDIR as absent
(Ok(None)), fixing the file-component-on-PATH crash.

src/rust/fs/src/lib.rs — The Vfs trait's path_metadata signature is
updated to accept follow_symlinks: bool. The in-memory DigestTrie implementation
accepts but ignores the flag.

src/rust/engine/src/nodes/path_metadata.rs — PathMetadataNode now carries
follow_symlinks and passes it through to the VFS call.

src/rust/engine/src/intrinsics/digests.rs — The path_metadata_request
intrinsic reads follow_symlinks from the Python PathMetadataRequest object and
passes it to PathMetadataNode.

src/python/pants/engine/fs.py — PathMetadataRequest gains a
follow_symlinks: bool = False field.

src/python/pants/core/util_rules/system_binaries.py — The initial lookup of
each PATH entry uses follow_symlinks=True. A symlink-to-directory is returned as
PathMetadataKind::DIRECTORY and handled correctly; a symlink-to-file is returned
as PathMetadataKind::FILE and skipped.

src/python/pants/core/util_rules/system_binaries_test.py — Three regression
tests alongside the existing test_find_binary_file_path:

• test_find_binary_symlink_to_dir_on_path — symlink on PATH pointing to a directory containing the binary is found correctly
• test_find_binary_symlink_to_file_on_path — symlink on PATH pointing to a file is skipped gracefully
• test_find_binary_file_component_on_path — path like not_a_dir/subdir where not_a_dir is a file is skipped gracefully

[tdyas]

Calling filesystem APIs from Pants rule code is forbidden since it prevents the Pants engine from knowing about the filesystem accesses. If you need to check whether the symlink target is a directory, then aggregate those requests and call the path_metadata_request rule again just as is done with file_metadata_requests.

[tdyas]

A better solution would be to add a follow_symlinks option to PathMetadataRequest and then use that option here. This would leave the Pants engine in control of filesystem accesses.

The relevant code in the Rust layer is

pants/src/rust/fs/src/posixfs.rs

Line 248 in 7c08ed5

match tokio::fs::symlink_metadata(&abs_path).await {

.

Instead of always calling tokio::fs::symlink_metadata, that code can switch on follow_symlinks and call either tokio::fs::symlink_metadata (if false) or tokio::fs::metadata (if true).