Skip to content

fix(storage): retry temporary OSS failures - #695

Merged
JingsongLi merged 4 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/oss-retry-layer
Aug 9, 2026
Merged

fix(storage): retry temporary OSS failures#695
JingsongLi merged 4 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/oss-retry-layer

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Purpose

OSS can return temporary failures such as HTTP 503. The OSS service classifies these errors as temporary, but the operator had no retry layer, so metadata and data operations failed immediately.

Changes

  • Attach OpenDAL RetryLayer to OSS operators with exponential backoff and jitter.
  • Support fs.oss.retry.count (default 10) and fs.oss.retry.interval.millisecond (default 500).
  • Add a regression test that returns OSS QpsLimitExceeded once and verifies the read succeeds on retry.
  • Document the retry options.

Tests

  • cargo test --offline -p paimon io::storage_oss::tests
  • cargo check --offline -p paimon --no-default-features --features storage-oss
  • cargo clippy --offline -p paimon --no-default-features --features storage-oss -- -D warnings
  • cargo fmt --all -- --check
  • git diff --check

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as draft August 8, 2026 16:43
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review August 9, 2026 02:33

@QuakeWang QuakeWang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit ee7057b into apache:main Aug 9, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants