You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ABFS URLs encode the container in URL userinfo, for example abfss://container@account.dfs.core.windows.net/path. The process-wide object store cache
previously keyed stores by scheme, host, and port only, so different containers in the same
storage account shared one cache entry. Since each Azure object store is bound to the container
from its URL, a later read could reuse the first container's store and return data from the wrong
container.
What changes are included in this PR?
Preserve URL userinfo when building object store keys for abfs and abfss, while retaining
the existing host-only behavior for other schemes.
Update the cache documentation to describe container-scoped keys.
Add an offline regression test that prepares two containers in the same account with identical
configuration and verifies that they resolve to distinct object store instances.
Nice targeted fix. I confirmed the analysis by reading native/core/src/parquet/objectstore/azure.rs — MicrosoftAzureBuilder::parse_url binds the container into the store instance and the resource Path is container-relative, so a store genuinely cannot be reused across containers. Position::BeforeUsername..AfterPort correctly includes the userinfo when present and collapses to BeforeHost when it isn't, so non-Azure schemes are untouched.
A few small things worth considering:
DataFusion's own registry has the same blind spot.RuntimeEnv::register_object_store uses get_url_key, which also strips userinfo (datafusion-execution/src/object_store.rs). So within a single RuntimeEnv, only one abfss store per account can be registered — a second registration would overwrite the first at the DF layer. Today this is fine because Comet constructs a fresh RuntimeEnv per Parquet file read, as the module doc calls out. But that invariant is now load-bearing for correctness on Azure. A sentence tying the two together in the module doc block would help a future reader who tries to share a RuntimeEnv across container reads.
Two small extensions to the new test, if you want them:
Assert that a second call with the same container returns the same Arc, so a future change that accidentally invalidates the same-URL cache path is also caught.
Add an s3://bucket@…-style case to pin down that the non-Azure branch still keys by host only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #4993.
Rationale for this change
ABFS URLs encode the container in URL userinfo, for example
abfss://container@account.dfs.core.windows.net/path. The process-wide object store cachepreviously keyed stores by scheme, host, and port only, so different containers in the same
storage account shared one cache entry. Since each Azure object store is bound to the container
from its URL, a later read could reuse the first container's store and return data from the wrong
container.
What changes are included in this PR?
abfsandabfss, while retainingthe existing host-only behavior for other schemes.
configuration and verifies that they resolve to distinct object store instances.
How are these changes tested?
cargo test --manifest-path native/Cargo.toml -p datafusion-comet --lib(
129 passed; 4 ignored)cargo clippy --manifest-path native/Cargo.toml -p datafusion-comet --lib --tests -- -D warningscargo fmt --manifest-path native/Cargo.toml --all -- --checkThe regression constructs the Azure stores locally and does not issue network requests.