[raycicmd] Implicit dimension matching for array-to-array depends_on

[andrew-anyscale]

Implement ($) resolution: when an array step depends on another array step via ($), each expanded variant depends only on the target variant with matching values on shared dimensions.

Guards:

• ($) from a non-array step errors (must use (*))
• ($) with no overlapping dimensions errors (must use (*))
• Value mismatch in implicit matching produces a clear error

Topic: implicit-dim-matching
Relative: bracket-depends-on
Signed-off-by: andrew andrew@anyscale.com

[andrew-anyscale]

Reviews in this chain:
└#471 [raycicmd] Implicit dimension matching for array-to-array depends_on

[andrew-anyscale]

#	head	base	diff	date	summary
0	56bfbd04	b2ec364a	diff	Mar 20 16:20 PM	2 files changed, 750 insertions(+), 16 deletions(-)
1	03cd98f2	b2ec364a	diff	Mar 20 16:44 PM	1 file changed, 1 insertion(+), 1 deletion(-)
2	7efcbc4a	2ca9c576	diff	Mar 20 17:10 PM	2 files changed, 76 insertions(+), 245 deletions(-)
3	bd4e5bdf	3b3be0c4	diff	Mar 20 17:37 PM	2 files changed, 307 insertions(+), 261 deletions(-)
4	339ff12e	66c37638	rebase	Mar 20 17:55 PM	0 files changed
5	c0fdd3e1	91b2dcd9	diff	Mar 20 17:57 PM	0 files changed
6	551d7bbe	f2f670d9	rebase	Mar 20 18:35 PM	0 files changed
7	c47615af	fb306e41	diff	Mar 20 21:50 PM	2 files changed, 27 insertions(+), 27 deletions(-)
8	5be334f0	28bf469b	diff	Mar 23 8:10 AM	0 files changed
9	b474b417	5528dfca	rebase	Mar 26 10:40 AM	0 files changed
10	3debefd7	34605949	diff	Mar 26 11:23 AM	0 files changed

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the depends_on mechanism within array steps by introducing implicit dimension matching. This intelligent dependency resolution ensures that expanded pipeline steps only link to relevant variants of their array dependencies, thereby optimizing pipeline execution by reducing unnecessary inter-variant dependencies and improving overall parallelism. It also provides an explicit way to override this behavior when a full fan-out is still required.

Highlights

• Implicit Dimension Matching: Introduced implicit dimension matching for array-to-array depends_on references. When two array steps share dimension names (e.g., 'python', 'cuda'), an expanded variant of the dependent step will now automatically depend only on the variant of its dependency that matches on those shared dimensions, rather than all variants.
• Improved Parallelism: This change unblocks parallelism in pipeline DAGs by preventing unnecessary waiting. For example, a Python 3.11 build will no longer wait for Python 3.10 or 3.12 builds of its dependency if only the 3.11 variant is relevant.
• Opt-Out Mechanism: Added an explicit opt-out mechanism using the depends_on: - step-name: all syntax. This allows users to force the previous 'fan-out to all variants' behavior if implicit matching is not desired.
• Enhanced arraySelector: The arraySelector struct was updated with a matchAll field to explicitly indicate when all variants of a dependency should be matched, supporting the new opt-out behavior.
• Robust Error Handling: Improved error messages for implicit dimension matching failures, guiding users to use the all selector if a matching variant cannot be found.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces implicit dimension matching for array-to-array dependencies, a valuable feature for improving pipeline parallelism. The implementation is well-structured, adding logic for implicit filters while also providing a clear opt-out mechanism. The changes are supported by a comprehensive and robust test suite that covers a wide range of scenarios and edge cases. The code quality is high, and I have one suggestion to enhance documentation for better long-term maintainability.

[gemini-code-assist]

The comment for the filter field is a bit brief given its nuanced behavior. When filter is nil, it signifies a plain dependency, which can either trigger implicit dimension matching or result in a fan-out to all variants if implicit matching doesn't apply. Clarifying this behavior in the comment will improve maintainability for anyone working on this code in the future.

Suggested change

	filter map[string][]string // nil = implicit matching may apply
	filter map[string][]string // dimension constraints; nil for a plain dependency, which may trigger implicit matching or fan-out to all variants

[andrew-anyscale]

Added skip-prcheck since a majority is testing.

[aslonnie]

PR description looks weird.

why skip-prcheck label is added for this PR?

[andrew-anyscale]

added skip-prcheck since a majority is testing.