Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

blue/green: Update items in the transaction all at once #26361

Merged
merged 3 commits into from Apr 1, 2024

Conversation

ParkMyCar
Copy link
Member

@ParkMyCar ParkMyCar commented Apr 1, 2024

This PR slightly refactors the RenameSchema op to update the durable transaction with the new items all at once instead of calling tx.update_item(...) for each item. update_item(...) iterates through the entire catalog every time it's called so it can be quite slow.

Creating two schemas with 100 views each, and then 500 views depending on the 100 views in a schema we exhibit the following performance.

main

materialize=> ALTER SCHEMA blue SWAP WITH green;
ALTER SCHEMA
Time: 37301.170 ms (00:37.301)

This PR

materialize=> ALTER SCHEMA blue SWAP WITH green;
ALTER SCHEMA
Time: 470.736 ms

Motivation

Greatly improves the speed of ALTER SCHEMA ... SWAP WITH ...

Checklist

  • This PR has adequate test coverage / QA involvement has been duly considered.
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
  • This PR includes the following user-facing behavior changes:
    • Performance improvement for ALTER SCHEMA ... SWAP WITH ...

@ParkMyCar ParkMyCar requested a review from jkosh44 April 1, 2024 20:06
@ParkMyCar ParkMyCar requested a review from a team as a code owner April 1, 2024 20:06
Copy link
Contributor

@jkosh44 jkosh44 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

src/adapter/src/catalog.rs Outdated Show resolved Hide resolved
@ParkMyCar ParkMyCar enabled auto-merge (squash) April 1, 2024 20:20
Copy link
Member

@benesch benesch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Can we add a test for this? Maybe in test/limits? No need to block this PR on it. And perhaps @MaterializeInc/testing have some cycles to assist.

@ParkMyCar
Copy link
Member Author

Nice! Can we add a test for this? Maybe in test/limits? No need to block this PR on it. And perhaps @MaterializeInc/testing have some cycles to assist.

Will do! I'll follow up with the testing team about this

@ParkMyCar ParkMyCar merged commit 412ee9c into MaterializeInc:main Apr 1, 2024
122 of 183 checks passed
@nrainer-materialize
Copy link
Contributor

Will do! I'll follow up with the testing team about this

The test will be added with #26377.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants