Skip to content

feat: Add Operator Cache for databend #18196

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jun 20, 2025

Conversation

Xuanwo
Copy link
Member

@Xuanwo Xuanwo commented Jun 19, 2025

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Add Operator Cache for databend.

Databend uses OpenDAL to access object storage. When accessing through an IAM role, OpenDAL internally uses reqsign to call AWS's API and obtain a temporary token. However, this API has a ratelimit. If users are frequently creating external tables, this can often cause query failures.

This PR address this by introduce an operator cache which can resuse the same operator if inputs are the same.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

Xuanwo added 2 commits June 19, 2025 15:10
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Jun 19, 2025
@BohuTANG BohuTANG requested a review from dantengsky June 20, 2025 08:26
Signed-off-by: Xuanwo <github@xuanwo.io>
@dantengsky dantengsky merged commit fb2a0de into databendlabs:main Jun 20, 2025
148 of 150 checks passed
@Xuanwo Xuanwo deleted the cache-operator branch June 25, 2025 03:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants