Implement cycle detection in transitive_targets, and tolerate cycles in file-addresses. #10409
Conversation
| Address("", target_name="t2.txt", generated_base_target_name="t2"), | ||
| ] | ||
|
|
||
| def do_test_failed_cycle(self, address_str: str, cyclic_address_str: str) -> None: |
There was a problem hiding this comment.
Nit: consider renaming to assert_failed_cycle() for alignment with Pytest and unittest.
|
|
||
| return TransitiveTargets(tuple(tt.root for tt in tts), FrozenOrderedSet(dependencies)) | ||
| transitive_targets = TransitiveTargets(tuple(targets), FrozenOrderedSet(visited)) | ||
| _detect_cycles(tuple(t.address for t in targets), dependency_mapping) |
There was a problem hiding this comment.
I suspect running DFS has a real performance impact on mission-critical code. But I also don't know of another way to detect these cycles at the same time as we're resolvings deps via BFS..
Any idea how bad the impact is when running ./pants dependencies --transitive ::?
One thing Benjy and I started to discuss this morning is whether to have a global flag --tolerate-cycles. Gordon gave feedback that he wouldn't want to tolerate cycles with dep inference / generated subtargets either. We would only run this code if you have --no-tolerate-cycles.
There was a problem hiding this comment.
It appears to add about 4.8% to the runtime.
One thing Benjy and I started to discuss this morning is whether to have a global flag --tolerate-cycles.
I'd prefer something that could be toggled on a target by target basis, but failing that, maybe a ternary of --tolerate-cycles=file|target|none.
There was a problem hiding this comment.
I think that would be fine with me to be a target by target basis. I'd rather have that than always no matter what tolerating generated subtargets.
There was a problem hiding this comment.
As long as there's a way to opt-in to (or out of) enforcing cycle hygiene.
stuhood
force-pushed
the
stuhood/high-level-cycle-tolerance
branch
from
e5d4502
to
01f7336
Compare
Yes. |
…in file-addresses. [ci skip-rust-tests]
[ci skip-rust-tests]
[ci skip-rust-tests]
stuhood
force-pushed
the
stuhood/high-level-cycle-tolerance
branch
from
01f7336
to
43aa188
Compare
### Problem "weak" `Get`s were a potential solution for optionally allowing for cycles in the runtime `Graph`, but they were only partially implemented (see #10229), and not particularly trustworthy. We ended up going with a higher level solution in #10409, but the weak-`Get`s code still exists, and increases the complexity of the `graph` crate unnecessarily. ### Solution Revert #10230. Closes #10229 and closes #10394. [ci skip-build-wheels]
Problem
After more deeply investigating the issues with #10230 (demonstrated in #10393 and its revert), it doesn't seem like the right idea to rely on the engine's cycle detection (which the implementation of #10230 showed should primarily be used for deadlock detection) to expose high level cycle tolerance for the build graph.
Solution
Move to iterative transitive target expansion (à la #10046), and cycle detect afterward using DFS with a stack of visited entries.
Result
Fixes #10059, and closes #10229. A followup will back out portions of #10230 (but not all of it, as there were a number of other improvements mixed in).
[ci skip-rust-tests]