Skip to content

feat(tools): add async token-bucket rate limiter middleware for tool execution - #39673

Closed
glatinone wants to merge 3 commits into
langgenius:mainfrom
glatinone:feat/tool-rate-limiter-middleware
Closed

feat(tools): add async token-bucket rate limiter middleware for tool execution#39673
glatinone wants to merge 3 commits into
langgenius:mainfrom
glatinone:feat/tool-rate-limiter-middleware

Conversation

@glatinone

Copy link
Copy Markdown

This PR introduces an opt-in Async Token-Bucket Rate Limiter and resilience middleware for tool execution:

  • New middleware: api/core/tools/middleware/rate_limiter.py (RPM/TPM configurable, async-safe)
  • Tool engine integration to acquire tokens before outbound calls
  • Exponential backoff with jitter on HTTP 429/503 and respect Retry-After
  • Unit tests: api/tests/unit_tests/core/tools/middleware/test_rate_limiter.py

Backwards compatible and disabled by default; enables safer multi-agent concurrency and reduces upstream 429s.

@dosubot dosubot Bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Jul 28, 2026
@glatinone

Copy link
Copy Markdown
Author

Closing this PR to avoid adding unnecessary review overhead for maintainers. Thank you for your time!

@glatinone glatinone closed this Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant