[VQueues] Introducing UserLimiter by AhmedSoliman · Pull Request #4607 · restatedev/restate

AhmedSoliman · 2026-04-17T13:39:50Z

Stack created with Sapling. Best reviewed with ReviewStack.

github-actions · 2026-04-17T14:15:59Z

Test Results

7 files + 2 7 suites +2 2m 47s ⏱️ -48s
47 tests + 20 47 ✅ + 20 0 💤 ±0 0 ❌ ±0
200 runs +146 200 ✅ +146 0 💤 ±0 0 ❌ ±0

Results for commit a4c8893. ± Comparison against base commit 6d70da2.

This pull request removes 25 and adds 45 tests. Note that renamed tests count towards both.

dev.restate.sdktesting.tests.AwakeableIngressEndpointTest ‑ completeWithFailure(Client)
dev.restate.sdktesting.tests.AwakeableIngressEndpointTest ‑ completeWithSuccess(Client)
dev.restate.sdktesting.tests.AwakeableLeaderTransferTest ‑ awakeableCompletionsAreNotLostDuringLeaderTransfer(Client, ContainerHandle)
dev.restate.sdktesting.tests.IngressTest ‑ idempotentInvokeSend(Client)
dev.restate.sdktesting.tests.IngressTest ‑ idempotentInvokeService(Client)
dev.restate.sdktesting.tests.IngressTest ‑ idempotentInvokeVirtualObject(Client)
dev.restate.sdktesting.tests.IngressTest ‑ idempotentSendThenAttachWIthIdempotencyKey(Client)
dev.restate.sdktesting.tests.IngressTest ‑ privateService(URI, Client)
dev.restate.sdktesting.tests.InvokerMemoryTest ‑ allInvocationsCompleteUnderMemoryPressure(Client, URI, int)
dev.restate.sdktesting.tests.InvokerMemoryTest ‑ allStatefulInvocationsCompleteUnderMemoryPressure(Client, URI, int)
…

dev.restate.sdktesting.tests.CallOrdering ‑ ordering(boolean[], Client)[1]
dev.restate.sdktesting.tests.CallOrdering ‑ ordering(boolean[], Client)[2]
dev.restate.sdktesting.tests.CallOrdering ‑ ordering(boolean[], Client)[3]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromAdminAPI(BlockingOperation, Client, URI)[1]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromAdminAPI(BlockingOperation, Client, URI)[2]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromAdminAPI(BlockingOperation, Client, URI)[3]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromContext(BlockingOperation, Client)[1]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromContext(BlockingOperation, Client)[2]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromContext(BlockingOperation, Client)[3]
dev.restate.sdktesting.tests.Combinators ‑ awakeableOrTimeoutUsingAwakeableTimeoutCommand(Client)
…

♻️ This comment has been updated with latest results.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 47324891fd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-04-21T11:41:41Z

+                });
+                // todo: This is temporary until we wrap the returned permit into an InvokePermit
+                // that invoker permit will carry the inner permit as opaque type.
+                let (permit, memory_lease) = run_permit.take_invoker_permit();


Keep user limiter permits alive for invocation lifetime

This extracts only the invoker semaphore/memory lease and then drops run_permit at the end of the match arm, which triggers ReservedResources::drop and sends PermitReleased for user resources immediately. As a result, LimitKeyConcurrency counters are decremented right after dispatch, so long-running invocations no longer consume user-concurrency slots and configured user limits are effectively bypassed under concurrent load.

Useful? React with 👍 / 👎.

Coming soon...

tillrohrmann

Thanks a lot for adding support for the user limiter @AhmedSoliman. The changes look great. +1 for merging :-)

As a follow-up, we need to figure out how to gate this feature.

tillrohrmann · 2026-04-22T08:22:50Z

+// Copyright (c) 2023 - 2026 Restate Software, Inc., Restate GmbH.
+// All rights reserved.
+//
+// Use of this software is governed by the Business Source License
+// included in the LICENSE file.
+//
+// As of the Change Date specified in that file, in accordance with
+// the Business Source License, use of this software will be governed
+// by the Apache License, Version 2.0.


As you've already raised the question, we need to find a way how to make this feature (the hierarchical limits) only accessible for enterprise users. Some variants that claude has seen in the wild are:

Cockroachdb: They put everything under their Cockroackdb Software License which allows free usage as long as your revenue is below $10 million. They enforce it by requiring a license key and users of the free tier can ask for a free tier license key. Before, cockroachdb was putting enterprise features under a different license but they moved away from it in favor of the CSL.

Redpanda: They put the enterprise features under a different license (redpanda community license) and require a license key for continued usage. If users don't have a valid license key, then the feature will degrade gracefully after a period of 30 days. In our case, this could mean that we no longer accept the addition/modification of hierarchical limits.

Gitlab: They have dedicated enterprise features as extensions to their open source features. All enterprise features live under a separate directory ee and they build a community edition and enterprise edition. For the latter, they replace the community edition features with their ee implementations. They also check license keys if ee features are used at runtime.

We don't have to answer this question right now but until we release, we probably need to find an answer.

One theme that seems to be very common is to prevent enterprise feature usage by having a license key.

Personally, I like the Redpanda model which allows users to try enterprise features out (w/o a high entrance barrier) and if they like them, then they have an incentive to get a paid subscription. But that's ultimately more of a product decision.

I'll leave things as is, let's discuss internally.

AhmedSoliman force-pushed the pr4607 branch 2 times, most recently from 69fa9ff to 595a94c Compare April 17, 2026 13:59

AhmedSoliman force-pushed the pr4607 branch from 595a94c to 7e8cee2 Compare April 17, 2026 16:17

AhmedSoliman mentioned this pull request Apr 17, 2026

Add user-limits and lock stats to WaitStats #4609

Draft

AhmedSoliman force-pushed the pr4607 branch 2 times, most recently from 350a09e to 6eb0d7c Compare April 20, 2026 15:58

AhmedSoliman mentioned this pull request Apr 20, 2026

[VQueues] Provisions to add user limits to datafusion #4615

Open

AhmedSoliman force-pushed the pr4607 branch from 6eb0d7c to 4732489 Compare April 20, 2026 16:40

AhmedSoliman requested a review from tillrohrmann April 21, 2026 11:28

AhmedSoliman marked this pull request as ready for review April 21, 2026 11:28

chatgpt-codex-connector Bot reviewed Apr 21, 2026

View reviewed changes

AhmedSoliman force-pushed the pr4607 branch from 4732489 to db52203 Compare April 21, 2026 15:51

AhmedSoliman force-pushed the pr4607 branch 3 times, most recently from c203f6f to 454db8a Compare April 22, 2026 09:54

tillrohrmann approved these changes Apr 22, 2026

View reviewed changes

[VQueues] Introducing UserLimiter

a4c8893

AhmedSoliman force-pushed the pr4607 branch from 454db8a to a4c8893 Compare April 22, 2026 13:00

AhmedSoliman merged commit a4c8893 into main Apr 22, 2026
65 checks passed

AhmedSoliman deleted the pr4607 branch April 22, 2026 13:32

github-actions Bot locked and limited conversation to collaborators Apr 22, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[VQueues] Introducing UserLimiter#4607

[VQueues] Introducing UserLimiter#4607
AhmedSoliman merged 1 commit intomainfrom
pr4607

AhmedSoliman commented Apr 17, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 17, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Apr 21, 2026

Uh oh!

AhmedSoliman Apr 21, 2026

Uh oh!

Uh oh!

tillrohrmann left a comment

Uh oh!

tillrohrmann Apr 22, 2026

Uh oh!

AhmedSoliman Apr 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

AhmedSoliman commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

AhmedSoliman Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tillrohrmann left a comment

Choose a reason for hiding this comment

Uh oh!

tillrohrmann Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

AhmedSoliman Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AhmedSoliman commented Apr 17, 2026 •

edited

Loading

github-actions Bot commented Apr 17, 2026 •

edited

Loading