Release-age cooldown bypass ignored when package is first resolved as transitive dependency

[andre-motta]

Summary

When a package is declared as a top-level requirement with an == pin (to bypass the release-age cooldown), the bypass is ignored if the same package was already resolved as a transitive dependency earlier in the bootstrap process. This is due to the session-level cache in BootstrapRequirementResolver not including req_type in its cache key.

Root Cause

In src/fromager/bootstrap_requirement_resolver.py, the resolution cache key is (str(req), pre_built):

self._resolved_requirements: dict[
    tuple[str, bool], tuple[tuple[str, Version], ...]
] = {}

The req_type (TOP_LEVEL vs INSTALL) is not part of the cache key. When a package is first encountered as a transitive dependency (RequirementType.INSTALL), the cooldown is enforced and the result is cached. When the same package is later encountered as a top-level requirement (RequirementType.TOP_LEVEL) with an == pin, the cached (cooldown-blocked) result is returned without re-evaluating resolve_package_cooldown().

Reproduction

1. Package B depends on A>=1.0
2. Package A==2.0.0 was published within the last X days (within cooldown window)
3. Declare both A==2.0.0 and B as top-level requirements
4. If B is processed first, A>=1.0 resolves as transitive → cooldown blocks version 2.0.0 → result cached
5. When A==2.0.0 is later processed as top-level, cache returns the blocked result → bypass never applies

Expected: Top-level A==2.0.0 should bypass the cooldown regardless of processing order.

Actual: Bypass only works if A==2.0.0 is resolved before any transitive dependency pulls in A.

Suggested Fix

Include req_type in the cache key:

cache_key = (str(req), pre_built, req_type)

This ensures transitive and top-level resolutions are cached separately, preserving the req_type context needed for cooldown bypass decisions.

Workaround

Move the cooldown-bypassing top-level requirement to the very first position in the requirements file so it is resolved before any transitive dependency can pull it in.

[tiran]

@andre-motta @rd4398 @dhellmann The suggested fix to update the cache key, is not the best solution. In your example we would end up with A==1.0.0 and A==2.0.0. I would expect that Fromager only builds A==2.0.0. After all, version 2.0.0 of A satisfies all requirements and the cooldown rule.

What if we change the resolution for top-level requirements from depth-first (DFS) to breadth-first (BFS)? If we resolve all top-level dependencies first, then we avoid the bug. On top of that, we also verify that all top-level dependencies resolve at all.

[andre-motta]

@andre-motta @rd4398 @dhellmann The suggested fix to update the cache key, is not the best solution. In your example we would end up with A==1.0.0 and A==2.0.0. I would expect that Fromager only builds A==2.0.0. After all, version 2.0.0 of A satisfies all requirements and the cooldown rule.

What if we change the resolution for top-level requirements from depth-first (DFS) to breadth-first (BFS)? If we resolve all top-level dependencies first, then we avoid the bug. On top of that, we also verify that all top-level dependencies resolve at all.

That would actually solve so many problems (BFS to DFS).
Failures would come from top level requirements first which helps in debugging the RHAI pipeline.

[dhellmann]

@andre-motta @rd4398 @dhellmann The suggested fix to update the cache key, is not the best solution. In your example we would end up with A==1.0.0 and A==2.0.0. I would expect that Fromager only builds A==2.0.0. After all, version 2.0.0 of A satisfies all requirements and the cooldown rule.
What if we change the resolution for top-level requirements from depth-first (DFS) to breadth-first (BFS)? If we resolve all top-level dependencies first, then we avoid the bug. On top of that, we also verify that all top-level dependencies resolve at all.

That would actually solve so many problems (BFS to DFS). Failures would come from top level requirements first which helps in debugging the RHAI pipeline.

We already resolve top-level dependencies to verify that they can be resolved before starting bootstrapping here.

The results are not all saved to the stack for processing, though. We use that stage to filter out anything that doesn't resolve and then put the rest of them on the stack and process them just like any other package. We're relying on the cache in the resolver to give us the same list of responses later when the _phase_resolve() method resolves the requirement again.

I think that means the previous fix doesn't solve the problem fully. The underlying issue is not just the ordering and it's not just that sometimes we have top-level requirements that would allow newer packages. It's that when we resolve a rule, we're only looking at the current context, and not accounting for other versions that may have been found in other contexts.

In multi-version mode, with a top-level rule A==2.0.0 and B with dependency A>1.0 we should build A versions 1.0.0 and 2.0.0. And we should also use version 2.0.0 when building B, to ensure compatibility. Saving the type of requirement with the rule in the cache doesn't allow that, because we would re-resolve the A>1.0 requirement without bypassing the date check and get 1.0.0. Changing the order of processing doesn't resolve that, because the cache won't find version 2.0.0 with a different rule for A because the rule is part of the cache key.

I think the correct fix is to have the resolver cache all known versions of a package, using only the name of the package as the cache key. It should also track which requirement rules have been used to compute a set of versions for a given package, so that we can avoid recomputing the same rule over and over. The resolver logic would then be something like

1. If the rule has not been seen before, resolve it by going out to the network then extend the cache contents with the new versions.
2. Apply the rule to all known versions from the cache for the package to compute the widest available set of versions based on the current rule and all previous rules.

And then I think we need to apply that logic to all top-level dependencies before we start building anything so that the cache is fully populated before we hit any transitive dependency rules.

That will result in a rule like A finding 1.0.0, A==2.0.0 finding 2.0.0 and then when B has transitive dependency A>1.0 both versions will be in the cache and we'll be able to use 2.0.0 to build B.

[rd4398]

@andre-motta @rd4398 @dhellmann The suggested fix to update the cache key, is not the best solution. In your example we would end up with A==1.0.0 and A==2.0.0. I would expect that Fromager only builds A==2.0.0. After all, version 2.0.0 of A satisfies all requirements and the cooldown rule.

What if we change the resolution for top-level requirements from depth-first (DFS) to breadth-first (BFS)? If we resolve all top-level dependencies first, then we avoid the bug. On top of that, we also verify that all top-level dependencies resolve at all.

I have done some research and expressed my concerns with BFS approach along with few other points to consider in my comment here: #1188 (comment)

[rd4398]

@andre-motta @rd4398 @dhellmann The suggested fix to update the cache key, is not the best solution. In your example we would end up with A==1.0.0 and A==2.0.0. I would expect that Fromager only builds A==2.0.0. After all, version 2.0.0 of A satisfies all requirements and the cooldown rule.
What if we change the resolution for top-level requirements from depth-first (DFS) to breadth-first (BFS)? If we resolve all top-level dependencies first, then we avoid the bug. On top of that, we also verify that all top-level dependencies resolve at all.

That would actually solve so many problems (BFS to DFS). Failures would come from top level requirements first which helps in debugging the RHAI pipeline.

We already resolve top-level dependencies to verify that they can be resolved before starting bootstrapping here.

The results are not all saved to the stack for processing, though. We use that stage to filter out anything that doesn't resolve and then put the rest of them on the stack and process them just like any other package. We're relying on the cache in the resolver to give us the same list of responses later when the _phase_resolve() method resolves the requirement again.

I think that means the previous fix doesn't solve the problem fully. The underlying issue is not just the ordering and it's not just that sometimes we have top-level requirements that would allow newer packages. It's that when we resolve a rule, we're only looking at the current context, and not accounting for other versions that may have been found in other contexts.

In multi-version mode, with a top-level rule A==2.0.0 and B with dependency A>1.0 we should build A versions 1.0.0 and 2.0.0. And we should also use version 2.0.0 when building B, to ensure compatibility. Saving the type of requirement with the rule in the cache doesn't allow that, because we would re-resolve the A>1.0 requirement without bypassing the date check and get 1.0.0. Changing the order of processing doesn't resolve that, because the cache won't find version 2.0.0 with a different rule for A because the rule is part of the cache key.

I think the correct fix is to have the resolver cache all known versions of a package, using only the name of the package as the cache key. It should also track which requirement rules have been used to compute a set of versions for a given package, so that we can avoid recomputing the same rule over and over. The resolver logic would then be something like

1. If the rule has not been seen before, resolve it by going out to the network then extend the cache contents with the new versions.
2. Apply the rule to all known versions from the cache for the package to compute the widest available set of versions based on the current rule and all previous rules.

And then I think we need to apply that logic to all top-level dependencies before we start building anything so that the cache is fully populated before we hit any transitive dependency rules.

That will result in a rule like A finding 1.0.0, A==2.0.0 finding 2.0.0 and then when B has transitive dependency A>1.0 both versions will be in the cache and we'll be able to use 2.0.0 to build B.

Okay, this makes sense. I will change my PR to implement this