Skip to content

feat(s3): support credential-scoped endpoint and OSS storage provider#2019

Merged
benflexcompute merged 1 commit into
mainfrom
ben/oss-credential-endpoint-support
Apr 24, 2026
Merged

feat(s3): support credential-scoped endpoint and OSS storage provider#2019
benflexcompute merged 1 commit into
mainfrom
ben/oss-credential-endpoint-support

Conversation

@benflexcompute
Copy link
Copy Markdown
Collaborator

@benflexcompute benflexcompute commented Apr 24, 2026

Summary

  • Add endpoint and storageProvider fields to _UserCredential, letting the STS response redirect the S3 client to a non-AWS backend.
  • When storageProvider == OSS, disable AWS integrity-check headers and switch to virtual-hosted addressing; preserves existing path-style behavior for other S3-compatible stores (s3proxy, MinIO) via Env.current.s3_endpoint_url.
  • Apply user_credential.endpoint to the boto3 client; Env.current.s3_endpoint_url still takes precedence when set.

Test plan

  • Default AWS flow (no storageProvider, no endpoint) still builds client unchanged.
  • storageProvider=OSS path: virtual-hosted addressing + checksums disabled.
  • S3-compatible dev path (Env.current.s3_endpoint_url set, no storageProvider): path-style + checksums disabled.
  • user_credential.endpoint set: client uses that endpoint_url.

🤖 Generated with Claude Code


Note

Medium Risk
Medium risk because it changes how the boto3 S3 client is configured (endpoint selection, checksum behavior, and addressing style), which can impact all uploads/downloads and compatibility with different S3 backends.

Overview
Adds optional endpoint and storageProvider fields to the STS userCredentials payload (_UserCredential) and threads them into S3 client construction.

When storageProvider is OSS, the client now disables AWS checksum/integrity behavior and switches to virtual-hosted bucket addressing; other S3-compatible endpoints configured via Env.current.s3_endpoint_url keep the existing path-style + checksum-disabled behavior.

If a credential-scoped endpoint is provided, it is used as the boto3 endpoint_url unless Env.current.s3_endpoint_url is set (which still overrides).

Reviewed by Cursor Bugbot for commit 26fc6aa. Bugbot is set up for automated code reviews on this repo. Configure here.

Add `endpoint` and `storageProvider` fields to `_UserCredential` so the
STS response can point the S3 client at alternative backends. When
`storageProvider == OSS`, disable AWS checksum headers and switch to
virtual-hosted addressing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown
Contributor

Coverage report (flow360)

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  flow360/cloud
  s3_utils.py 154-164, 187-191
Project Total  

This report was generated by python-coverage-comment-action

@benflexcompute benflexcompute merged commit 45677bd into main Apr 24, 2026
21 checks passed
@benflexcompute benflexcompute deleted the ben/oss-credential-endpoint-support branch April 24, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants