Add Lakebase Autoscaling support for Databricks Asset Bundles #4423
+7,328
−6
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| bundle: | ||
| name: test-bundle-$UNIQUE_NAME | ||
|
|
||
| resources: | ||
| postgres_projects: | ||
| project: | ||
| project_id: test-pg-project-$UNIQUE_NAME | ||
| display_name: Test Postgres Project | ||
|
|
||
| postgres_branches: | ||
| foo: | ||
| parent: ${resources.postgres_projects.project.name} | ||
| branch_id: test-branch-$UNIQUE_NAME | ||
| no_expiry: true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| bundle: | ||
| name: test-bundle-$UNIQUE_NAME | ||
|
|
||
| resources: | ||
| postgres_projects: | ||
| project: | ||
| project_id: test-pg-project-$UNIQUE_NAME | ||
| display_name: Test Postgres Project | ||
|
|
||
| postgres_branches: | ||
| branch: | ||
| parent: ${resources.postgres_projects.project.name} | ||
| branch_id: test-branch-$UNIQUE_NAME | ||
| no_expiry: true | ||
|
|
||
| postgres_endpoints: | ||
| foo: | ||
| parent: ${resources.postgres_branches.branch.name} | ||
| endpoint_id: test-endpoint-$UNIQUE_NAME | ||
| endpoint_type: ENDPOINT_TYPE_READ_WRITE |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| bundle: | ||
| name: test-bundle-$UNIQUE_NAME | ||
|
|
||
| resources: | ||
| postgres_projects: | ||
| foo: | ||
| project_id: test-pg-project-$UNIQUE_NAME | ||
| display_name: Test Postgres Project |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3593,6 +3593,131 @@ resources.pipelines.*.permissions.permissions[*].group_name string ALL | |
| resources.pipelines.*.permissions.permissions[*].permission_level iam.PermissionLevel ALL | ||
| resources.pipelines.*.permissions.permissions[*].service_principal_name string ALL | ||
| resources.pipelines.*.permissions.permissions[*].user_name string ALL | ||
| resources.postgres_branches.*.branch_id string INPUT STATE | ||
| resources.postgres_branches.*.create_time *time.Time REMOTE | ||
| resources.postgres_branches.*.expire_time *time.Time INPUT STATE | ||
| resources.postgres_branches.*.id string INPUT | ||
| resources.postgres_branches.*.is_protected bool INPUT STATE | ||
| resources.postgres_branches.*.lifecycle resources.Lifecycle INPUT | ||
| resources.postgres_branches.*.lifecycle.prevent_destroy bool INPUT | ||
| resources.postgres_branches.*.modified_status string INPUT | ||
| resources.postgres_branches.*.name string REMOTE | ||
| resources.postgres_branches.*.no_expiry bool INPUT STATE | ||
| resources.postgres_branches.*.parent string ALL | ||
| resources.postgres_branches.*.source_branch string INPUT STATE | ||
| resources.postgres_branches.*.source_branch_lsn string INPUT STATE | ||
| resources.postgres_branches.*.source_branch_time *time.Time INPUT STATE | ||
| resources.postgres_branches.*.spec *postgres.BranchSpec REMOTE | ||
| resources.postgres_branches.*.spec.expire_time *time.Time REMOTE | ||
| resources.postgres_branches.*.spec.is_protected bool REMOTE | ||
| resources.postgres_branches.*.spec.no_expiry bool REMOTE | ||
| resources.postgres_branches.*.spec.source_branch string REMOTE | ||
| resources.postgres_branches.*.spec.source_branch_lsn string REMOTE | ||
| resources.postgres_branches.*.spec.source_branch_time *time.Time REMOTE | ||
| resources.postgres_branches.*.spec.ttl *duration.Duration REMOTE | ||
| resources.postgres_branches.*.status *postgres.BranchStatus REMOTE | ||
| resources.postgres_branches.*.status.current_state postgres.BranchStatusState REMOTE | ||
| resources.postgres_branches.*.status.default bool REMOTE | ||
| resources.postgres_branches.*.status.expire_time *time.Time REMOTE | ||
|
Contributor
There was a problem hiding this comment. should we move
Contributor
There was a problem hiding this comment. Note, we can only do it if status.expire_time tracks user setting without delay. If it's possible that field in status can lag behind field in input, this would create unnecessary drift.
Contributor
There was a problem hiding this comment. Some fields there should not drift, e.g. branch_id, because they are immutable. So safe to move. See this commit for full list of fields: 6f62615?new_files_changed=true#diff-7478ee1612d5f53124855c9fa144177bd5ea219b2ec22e416c690c59d99ae6ffR101 |
||
| resources.postgres_branches.*.status.is_protected bool REMOTE | ||
| resources.postgres_branches.*.status.logical_size_bytes int64 REMOTE | ||
| resources.postgres_branches.*.status.pending_state postgres.BranchStatusState REMOTE | ||
| resources.postgres_branches.*.status.source_branch string REMOTE | ||
| resources.postgres_branches.*.status.source_branch_lsn string REMOTE | ||
| resources.postgres_branches.*.status.source_branch_time *time.Time REMOTE | ||
| resources.postgres_branches.*.status.state_change_time *time.Time REMOTE | ||
| resources.postgres_branches.*.ttl *duration.Duration INPUT STATE | ||
| resources.postgres_branches.*.uid string REMOTE | ||
| resources.postgres_branches.*.update_time *time.Time REMOTE | ||
| resources.postgres_branches.*.url string INPUT | ||
| resources.postgres_endpoints.*.autoscaling_limit_max_cu float64 INPUT STATE | ||
| resources.postgres_endpoints.*.autoscaling_limit_min_cu float64 INPUT STATE | ||
| resources.postgres_endpoints.*.create_time *time.Time REMOTE | ||
| resources.postgres_endpoints.*.disabled bool INPUT STATE | ||
| resources.postgres_endpoints.*.endpoint_id string INPUT STATE | ||
| resources.postgres_endpoints.*.endpoint_type postgres.EndpointType INPUT STATE | ||
| resources.postgres_endpoints.*.id string INPUT | ||
| resources.postgres_endpoints.*.lifecycle resources.Lifecycle INPUT | ||
| resources.postgres_endpoints.*.lifecycle.prevent_destroy bool INPUT | ||
| resources.postgres_endpoints.*.modified_status string INPUT | ||
| resources.postgres_endpoints.*.name string REMOTE | ||
| resources.postgres_endpoints.*.no_suspension bool INPUT STATE | ||
| resources.postgres_endpoints.*.parent string ALL | ||
| resources.postgres_endpoints.*.settings *postgres.EndpointSettings INPUT STATE | ||
| resources.postgres_endpoints.*.settings.pg_settings map[string]string INPUT STATE | ||
| resources.postgres_endpoints.*.settings.pg_settings.* string INPUT STATE | ||
| resources.postgres_endpoints.*.spec *postgres.EndpointSpec REMOTE | ||
| resources.postgres_endpoints.*.spec.autoscaling_limit_max_cu float64 REMOTE | ||
| resources.postgres_endpoints.*.spec.autoscaling_limit_min_cu float64 REMOTE | ||
| resources.postgres_endpoints.*.spec.disabled bool REMOTE | ||
| resources.postgres_endpoints.*.spec.endpoint_type postgres.EndpointType REMOTE | ||
| resources.postgres_endpoints.*.spec.no_suspension bool REMOTE | ||
| resources.postgres_endpoints.*.spec.settings *postgres.EndpointSettings REMOTE | ||
| resources.postgres_endpoints.*.spec.settings.pg_settings map[string]string REMOTE | ||
| resources.postgres_endpoints.*.spec.settings.pg_settings.* string REMOTE | ||
| resources.postgres_endpoints.*.spec.suspend_timeout_duration *duration.Duration REMOTE | ||
| resources.postgres_endpoints.*.status *postgres.EndpointStatus REMOTE | ||
| resources.postgres_endpoints.*.status.autoscaling_limit_max_cu float64 REMOTE | ||
| resources.postgres_endpoints.*.status.autoscaling_limit_min_cu float64 REMOTE | ||
| resources.postgres_endpoints.*.status.current_state postgres.EndpointStatusState REMOTE | ||
| resources.postgres_endpoints.*.status.disabled bool REMOTE | ||
| resources.postgres_endpoints.*.status.endpoint_type postgres.EndpointType REMOTE | ||
| resources.postgres_endpoints.*.status.hosts *postgres.EndpointHosts REMOTE | ||
| resources.postgres_endpoints.*.status.hosts.host string REMOTE | ||
| resources.postgres_endpoints.*.status.pending_state postgres.EndpointStatusState REMOTE | ||
| resources.postgres_endpoints.*.status.settings *postgres.EndpointSettings REMOTE | ||
| resources.postgres_endpoints.*.status.settings.pg_settings map[string]string REMOTE | ||
| resources.postgres_endpoints.*.status.settings.pg_settings.* string REMOTE | ||
| resources.postgres_endpoints.*.status.suspend_timeout_duration *duration.Duration REMOTE | ||
| resources.postgres_endpoints.*.suspend_timeout_duration *duration.Duration INPUT STATE | ||
| resources.postgres_endpoints.*.uid string REMOTE | ||
| resources.postgres_endpoints.*.update_time *time.Time REMOTE | ||
| resources.postgres_endpoints.*.url string INPUT | ||
| resources.postgres_projects.*.create_time *time.Time REMOTE | ||
| resources.postgres_projects.*.default_endpoint_settings *postgres.ProjectDefaultEndpointSettings INPUT STATE | ||
| resources.postgres_projects.*.default_endpoint_settings.autoscaling_limit_max_cu float64 INPUT STATE | ||
| resources.postgres_projects.*.default_endpoint_settings.autoscaling_limit_min_cu float64 INPUT STATE | ||
| resources.postgres_projects.*.default_endpoint_settings.no_suspension bool INPUT STATE | ||
| resources.postgres_projects.*.default_endpoint_settings.pg_settings map[string]string INPUT STATE | ||
| resources.postgres_projects.*.default_endpoint_settings.pg_settings.* string INPUT STATE | ||
| resources.postgres_projects.*.default_endpoint_settings.suspend_timeout_duration *duration.Duration INPUT STATE | ||
| resources.postgres_projects.*.display_name string INPUT STATE | ||
| resources.postgres_projects.*.history_retention_duration *duration.Duration INPUT STATE | ||
| resources.postgres_projects.*.id string INPUT | ||
| resources.postgres_projects.*.lifecycle resources.Lifecycle INPUT | ||
| resources.postgres_projects.*.lifecycle.prevent_destroy bool INPUT | ||
| resources.postgres_projects.*.modified_status string INPUT | ||
| resources.postgres_projects.*.name string REMOTE | ||
| resources.postgres_projects.*.pg_version int INPUT STATE | ||
| resources.postgres_projects.*.project_id string INPUT STATE | ||
| resources.postgres_projects.*.spec *postgres.ProjectSpec REMOTE | ||
| resources.postgres_projects.*.spec.default_endpoint_settings *postgres.ProjectDefaultEndpointSettings REMOTE | ||
| resources.postgres_projects.*.spec.default_endpoint_settings.autoscaling_limit_max_cu float64 REMOTE | ||
| resources.postgres_projects.*.spec.default_endpoint_settings.autoscaling_limit_min_cu float64 REMOTE | ||
| resources.postgres_projects.*.spec.default_endpoint_settings.no_suspension bool REMOTE | ||
| resources.postgres_projects.*.spec.default_endpoint_settings.pg_settings map[string]string REMOTE | ||
| resources.postgres_projects.*.spec.default_endpoint_settings.pg_settings.* string REMOTE | ||
| resources.postgres_projects.*.spec.default_endpoint_settings.suspend_timeout_duration *duration.Duration REMOTE | ||
| resources.postgres_projects.*.spec.display_name string REMOTE | ||
| resources.postgres_projects.*.spec.history_retention_duration *duration.Duration REMOTE | ||
| resources.postgres_projects.*.spec.pg_version int REMOTE | ||
| resources.postgres_projects.*.status *postgres.ProjectStatus REMOTE | ||
| resources.postgres_projects.*.status.branch_logical_size_limit_bytes int64 REMOTE | ||
| resources.postgres_projects.*.status.default_endpoint_settings *postgres.ProjectDefaultEndpointSettings REMOTE | ||
| resources.postgres_projects.*.status.default_endpoint_settings.autoscaling_limit_max_cu float64 REMOTE | ||
| resources.postgres_projects.*.status.default_endpoint_settings.autoscaling_limit_min_cu float64 REMOTE | ||
| resources.postgres_projects.*.status.default_endpoint_settings.no_suspension bool REMOTE | ||
| resources.postgres_projects.*.status.default_endpoint_settings.pg_settings map[string]string REMOTE | ||
| resources.postgres_projects.*.status.default_endpoint_settings.pg_settings.* string REMOTE | ||
| resources.postgres_projects.*.status.default_endpoint_settings.suspend_timeout_duration *duration.Duration REMOTE | ||
| resources.postgres_projects.*.status.display_name string REMOTE | ||
| resources.postgres_projects.*.status.history_retention_duration *duration.Duration REMOTE | ||
| resources.postgres_projects.*.status.owner string REMOTE | ||
| resources.postgres_projects.*.status.pg_version int REMOTE | ||
| resources.postgres_projects.*.status.synthetic_storage_size_bytes int64 REMOTE | ||
| resources.postgres_projects.*.uid string REMOTE | ||
| resources.postgres_projects.*.update_time *time.Time REMOTE | ||
| resources.postgres_projects.*.url string INPUT | ||
| resources.quality_monitors.*.assets_dir string ALL | ||
| resources.quality_monitors.*.baseline_table_name string ALL | ||
| resources.quality_monitors.*.custom_metrics []catalog.MonitorMetric ALL | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| bundle: | ||
| name: deploy-postgres-branch-$UNIQUE_NAME | ||
|
|
||
| sync: | ||
| paths: [] | ||
|
|
||
| resources: | ||
| postgres_projects: | ||
| my_project: | ||
| project_id: test-pg-proj-$UNIQUE_NAME | ||
| display_name: "Test Project for Branch" | ||
| pg_version: 16 | ||
| history_retention_duration: "604800s" | ||
| default_endpoint_settings: | ||
| autoscaling_limit_min_cu: 0.5 | ||
| autoscaling_limit_max_cu: 4 | ||
| suspend_timeout_duration: "300s" | ||
|
|
||
| postgres_branches: | ||
| main: | ||
| parent: ${resources.postgres_projects.my_project.id} | ||
| branch_id: main | ||
| no_expiry: true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| { | ||
| "method": "POST", | ||
| "path": "/api/2.0/postgres/projects", | ||
| "q": { | ||
| "project_id": "test-pg-proj-[UNIQUE_NAME]" | ||
| }, | ||
| "body": { | ||
| "spec": { | ||
| "default_endpoint_settings": { | ||
| "autoscaling_limit_max_cu": 4, | ||
| "autoscaling_limit_min_cu": 0.5, | ||
| "suspend_timeout_duration": "300s" | ||
| }, | ||
| "display_name": "Test Project for Branch", | ||
| "history_retention_duration": "604800s", | ||
| "pg_version": 16 | ||
| } | ||
| } | ||
| } | ||
| { | ||
| "method": "POST", | ||
| "path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches", | ||
| "q": { | ||
| "branch_id": "main" | ||
| }, | ||
| "body": { | ||
| "spec": { | ||
| "no_expiry": true | ||
| } | ||
| } | ||
| } | ||
| { | ||
| "method": "GET", | ||
| "path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/main" | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| { | ||
| "method": "POST", | ||
| "path": "/api/2.0/postgres/projects", | ||
| "q": { | ||
| "project_id": "test-pg-proj-[UNIQUE_NAME]" | ||
| }, | ||
| "body": { | ||
| "spec": { | ||
| "default_endpoint_settings": { | ||
| "autoscaling_limit_max_cu": 4, | ||
| "autoscaling_limit_min_cu": 0.5, | ||
| "suspend_timeout_duration": "300s" | ||
| }, | ||
| "display_name": "Test Project for Branch", | ||
| "history_retention_duration": "604800s", | ||
| "pg_version": 16 | ||
| } | ||
| } | ||
| } | ||
| { | ||
| "method": "POST", | ||
| "path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches", | ||
| "q": { | ||
| "branch_id": "main" | ||
| }, | ||
| "body": { | ||
| "parent": "projects/test-pg-proj-[UNIQUE_NAME]", | ||
| "spec": { | ||
| "no_expiry": true | ||
| } | ||
| } | ||
| } | ||
| { | ||
| "method": "GET", | ||
| "path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/main" | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
replace_ids.py is another script that could use a similar change.