Skip to content

Feat: Create snapshot tables on demand#2028

Merged
izeigerman merged 6 commits intomainfrom
on-demand-snapshot-table-creation
Jan 26, 2024
Merged

Feat: Create snapshot tables on demand#2028
izeigerman merged 6 commits intomainfrom
on-demand-snapshot-table-creation

Conversation

@izeigerman
Copy link
Collaborator

This update makes sure that physical tables for snapshots are created lazily when they are actually needed. This is particularly helpful when using the model selector, in which case we create tables for selected models only.

Additionally, this update has a nice side effect of adding self-healing properties to the plan / apply flow by automatically re-creating snapshot tables if they happen to be missing.

@izeigerman izeigerman requested a review from a team January 26, 2024 06:08
@izeigerman izeigerman force-pushed the on-demand-snapshot-table-creation branch from 7ee011a to 2c488a0 Compare January 26, 2024 19:21
@eakmanrq
Copy link
Collaborator

Since you added a public method usually these get a test added in test_integration. Specifically it would be good to test the object names functionality and possibly with the batching too. So it would be something like create table and view and make sure they show up and then filter and make sure the filter limits the number of results. If you want something quick you could just update the _get_data_objects call to instead use the new public method which would at least have us using that public path.

@izeigerman izeigerman force-pushed the on-demand-snapshot-table-creation branch from 28f4e83 to e68bbcb Compare January 26, 2024 21:11
@izeigerman
Copy link
Collaborator Author

@eakmanrq added additional tests

@izeigerman izeigerman merged commit 3fce5e5 into main Jan 26, 2024
@izeigerman izeigerman deleted the on-demand-snapshot-table-creation branch January 26, 2024 22:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants