solver: Release unused refs in LoadWithParents #3815
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before this any refs that were loaded but not put into results would be leaked.
cc @tonistiigi noticed in certain load tests I was running w/ Dagger that cache refs were leaking, this fixes it. I am still trying to understand the exact circumstances it arises in.I know for sure it only happens when using a cacheManager constructed from a remote cache import. And I also think it might happen when there are multiple results that end up with the same layers (i.e. two image refs resolve to the same set of layers). However, the context in which this was triggered involved cache manifest json constructed by custom code, not by the normal buildkit code, so I don't know if it's possible to trigger this using a cache manifest generated by buildkit upstream.Either way, I think this fix is desirable upstream since we should never leave refs unreleased anyways.
EDIT: @tonistiigi I was able to reproduce this same bug purely with Buildkit, see the newly added integration test, which fails without this change present.
The problem happens when a remote cache import has multiple records with the same layers. In this case the resultID is calculated as a hash of the layer digests, so both records are associated with the same resultID. This causes them both to be loaded in
LoadWithParents
in the remote cache result storage.filterResults
then only chooses one of them to use for the returnedresults
list, but it never releases the refs for the others it didn't end up choosing.