[adapter] Batch-allocate user IDs to reduce persist writes during DDL#35460
Merged
mtabebe merged 2 commits intoMaterializeInc:mainfrom Mar 13, 2026
Merged
[adapter] Batch-allocate user IDs to reduce persist writes during DDL#35460mtabebe merged 2 commits intoMaterializeInc:mainfrom
mtabebe merged 2 commits intoMaterializeInc:mainfrom
Conversation
…MaterializeInc#35310) # Problem: Every DDL statement (CREATE TABLE, CREATE VIEW, etc.) performs a persist write + timestamp oracle call to allocate a single user ID. This represents one extra round trip to the timestamp oracle, in addition to the necessary catalog operation for DDL. # Solution: - Add `IdPool` struct with `allocate()`, `allocate_many()`, and `refill()` methods to pre-allocate user IDs in a contiguous range - Add `USER_ID_POOL_BATCH_SIZE` dyncfg (default 512) to control batch size - Add `user_id_pool` field to `Coordinator` initialized as empty - Add Coordinator wrapper methods `allocate_user_id()` and `allocate_user_ids(count)` that draw from the pool and auto-refill from catalog when exhausted - Replace all call sites that used the two-line pattern of `get_catalog_write_ts()` + `catalog().allocate_user_id(ts)` with the new pooled methods - This eliminates the timestamp oracle call, and catalog operation # Testing: - New `IdPool` unit tests covering empty pool, single/batch allocation, refill, mixed allocation, and invalid range panic - New SLT regression test (`test/sqllogictest/id_pool.slt`) exercising tables, views, materialized views, indexes, types, secrets, and drop/recreate cycles - New MZCompose restart test to ensure IDs remain unique across the restart # Perf On local machine: 10% saving: from 160ms to 145 ms on a empty catalog Closes SQL-115
|
Thanks for opening this PR! Here are a few tips to help make the review process smooth for everyone. PR title guidelines
Pre-merge checklist
|
Contributor
Author
Problem: In the preflight phase of a zero down time upgrade, get_next_ids() is used to determine if there has been ongoing DDL. Since the IdPool batch-allocates IDs, the counter can advance far ahead of actually-created items. New objects created from the pool then had IDs below the captured baseline and were missed by DDL detection, so the standby could fail to see new tables/views. MaterializeInc/database-issues#11232 Solution: - Compute next user item ID and next user replica ID from the max existing IDs in the catalog (transaction get_items / get_cluster_replicas) instead of the allocator counter. Testing: - New cargo test: mz-catalog test_persist_ddl_detection_with_batch_allocated_ids - New MZ Compose test: workflow_ddl_detection_with_id_pool
mtabebe
force-pushed
the
ma/dd/batch-id-fix
branch
from
1edf51f to
1913e45
Compare
def-
reviewed
Mar 12, 2026
Contributor
def-
left a comment
def-
left a comment
There was a problem hiding this comment.
Thanks for adding a 0dt test!
aljoscha
approved these changes
Mar 13, 2026
Contributor
aljoscha
left a comment
aljoscha
left a comment
There was a problem hiding this comment.
The approach looks good, but I think there's a cargo test failure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits: the original from #35310
And a new commit that fixes the issue by finding the max used ID from the catalog.
Keeping commits separate to make it easier to re-review.