fix(source_metadata): deduplicate metadata requests across environments#5647
Open
mohitdebian wants to merge 5 commits intoprefix-dev:mainfrom
Open
fix(source_metadata): deduplicate metadata requests across environments#5647mohitdebian wants to merge 5 commits intoprefix-dev:mainfrom
mohitdebian wants to merge 5 commits intoprefix-dev:mainfrom
Conversation
…ts to avoid cache conflicts
…latform deduplication
…ign with backend metadata cache handling
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When the same source package is used across multiple pixi environments (for example
defaultandtest), each environment may requestSourceMetadataconcurrently.Previously,
CommandDispatcherProcessordeduplicated requests using the fullSourceMetadataSpec:However,
SourceMetadataSpecembedsBuildBackendMetadataSpec, which contains environment-specific fields such as:channelschannel_configbuild_environmentvariant_configurationvariant_filesThese fields often differ between environments even when they refer to the exact same source package. Because of this, the dispatcher treated the requests as different tasks and spawned multiple metadata computations.
When those tasks attempted to write to the disk cache concurrently, they could race on
try_write, resulting in:Changes in this PR
This PR introduces a normalized in-memory deduplication key:
SourceMetadataKeySourceMetadataKeyflattens the fields fromSourceMetadataSpecthat determine the identity of the metadata request. This allows the dispatcher to deduplicate requests without relying on the nested structure ofSourceMetadataSpec.The key includes:
packagemanifest_sourcepreferred_build_sourceenabled_protocolsbuild_environmentchannelschannel_configvariant_configurationvariant_filesAdditional changes:
SourceMetadataKeystruct insource_metadata/mod.rsSourceMetadataSpec::dedup_key()to construct the normalized keyon_source_metadatato compute the key once and reuse itgen_source_metadata_idto accept the precomputed keyWriteResult::Conflictso that the metadata already written to the cache is reusedEffect
Before:
After:
This ensures metadata for the same source package is computed only once per dispatcher execution and prevents redundant cache writes.
Fixes #5626
How Has This Been Tested?
The changes were verified locally with:
cargo check cargo test cargo clippy --all-targets --all-features cargo fmtAdditionally verified that multiple environments requesting the same source metadata attach to the same dispatcher task instead of spawning duplicate computations.
AI Disclosure
Tools: Claude
Checklist:
schema/model.py