Skip to content

fix: Only strip config key prefix when present - #755

Draft
kylebarron wants to merge 1 commit into
mainfrom
fix/config-key-prefix-stripping
Draft

fix: Only strip config key prefix when present#755
kylebarron wants to merge 1 commit into
mainfrom
fix/config-key-prefix-stripping

Conversation

@kylebarron

Copy link
Copy Markdown
Member

Problem

IntoPyObject for the S3/GCS/Azure config keys assumed every key as_ref()s to a prefixed string, and used .expect() to strip it:

self.0.as_ref().strip_prefix("aws_").expect("Expected config prefix to start with aws_")

But object_store delegates client and encryption keys to inner enums — Self::Client(opt) => opt.as_ref(), Self::Encryption(opt) => opt.as_ref() — which return unprefixed names like allow_http. So reading .config on a store built with any client option panicked:

>>> S3Store("bucket", allow_http=True).config
pyo3_runtime.PanicException: Expected config prefix to start with aws_

All three stores were affected (aws_, google_, azure_).

Fix

Strip the prefix only when it's actually present, matching what object_store's own FromStr does (s.strip_prefix("aws_").unwrap_or(s)).

Also: aws_endpoint_url_s3 round-trip

While auditing every config key across the three stores for the same asymmetry, aws_endpoint_url_s3 turned out to be the only key with no unprefixed alias upstream — from_str accepts aws_endpoint_url_s3 but not endpoint_url_s3, unlike its sibling endpoint_url_sts which has both. Since we emit keys unprefixed, the value we produced couldn't be fed back in:

>>> store = S3Store("bucket", aws_endpoint_url_s3="https://example.com")
>>> store.config
{'bucket': 'bucket', 'endpoint_url_s3': 'https://example.com'}
>>> pickle.loads(pickle.dumps(store))
UnknownConfigurationKeyError: Configuration key: 'endpoint_url_s3' is not valid for store 'S3'

PyAmazonS3ConfigKey::extract now retries the parse with an aws_ prefix on failure. Worth an upstream PR adding the missing alias too.

Testing

Regression tests added for each store. I also scanned every config key alias in all three stores for these two failure modes; no others remain.

Note

Found while investigating #753. This does not fix that issue — AWS_ENDPOINT_URL_S3 beating an explicit endpoint= kwarg is a separate problem (Endpoint and S3Endpoint are distinct upstream keys, and build() does self.s3_endpoint.or(self.endpoint) unconditionally). It does, however, make the aws_endpoint_url_s3= workaround usable with .config/pickling.

🤖 Generated with Claude Code

`IntoPyObject` for the S3/GCS/Azure config keys assumed every key
`as_ref()`s to a prefixed string and used `.expect()` to strip it. But
object_store delegates client and encryption keys to inner enums —
`Self::Client(opt) => opt.as_ref()` — which return unprefixed names like
`allow_http`. Reading `.config` on a store built with any client option
therefore panicked:

    S3Store("bucket", allow_http=True).config
    PanicException: Expected config prefix to start with aws_

Strip the prefix only when it is actually present, matching what
object_store's own `FromStr` does (`strip_prefix("aws_").unwrap_or(s)`).

Separately, `aws_endpoint_url_s3` is the one key with no unprefixed
alias upstream, so the `endpoint_url_s3` we emit could not be parsed
back in — breaking `S3Store(config=store.config)` and pickling. Retry
the parse with the `aws_` prefix on failure.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant