Exporter: don't emit all UC objects when handling dependencies #3556
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Right now, if we discover any UC dependency, we emit upstream objects as well, such as schemas, catalogs, etc., but in their `Import` operations we're doing all nested objects unnecessarily - so emit one init script from a UC Volume may lead to emitting of the whole UC Catalog with all schemas/tables/volumes... With this change, we list all nested objects (schemas in catalog, tables/models/volumes in a schema) only if these services are explicitly specified in the `-listing` option. Fixes #3555
hectorcast-db
approved these changes
May 8, 2024
alexott
added a commit
that referenced
this pull request
May 10, 2024
…to default listing The `uc-schemas`, `uc-models`, `uc-tables`, and `uc-volumes` services don't have explicit `List` operation, but we need to include them into the default listing so they will be exported in the default mode.
tanmay-db
added a commit
that referenced
this pull request
May 13, 2024
### New Features and Improvements * fix ([#3542](#3542)). * Update data_aws_crossaccount_policy to format resource string with account and region ([#3544](#3544)). * Robust retries for workspace get-status ([#3550](#3550)). * Fix error message typo in mounts.go ([#3552](#3552)). * State the options for run_if ([#3548](#3548)). * Exporter: don't emit all UC objects when handling dependencies ([#3556](#3556)). * Added state upgrader to remove `max_clusters_per_user` and ebs volume attributes set to zero ([#3551](#3551)). * Exporter: track removed objects during the `Emit` phase ([#3554](#3554)). * Jobs GoSDK schema migration ([#3532](#3532)). * Exporter: optimize generation of `databricks_group_member` resource ([#3559](#3559)). ### Documentation Changes ### Exporter ### Internal Changes
Merged
github-merge-queue bot
pushed a commit
that referenced
this pull request
May 13, 2024
…to default listing (#3565) The `uc-schemas`, `uc-models`, `uc-tables`, and `uc-volumes` services don't have explicit `List` operation, but we need to include them into the default listing so they will be exported in the default mode.
github-merge-queue bot
pushed a commit
that referenced
this pull request
May 14, 2024
* Release v1.43.0 ### New Features and Improvements * fix ([#3542](#3542)). * Update data_aws_crossaccount_policy to format resource string with account and region ([#3544](#3544)). * Robust retries for workspace get-status ([#3550](#3550)). * Fix error message typo in mounts.go ([#3552](#3552)). * State the options for run_if ([#3548](#3548)). * Exporter: don't emit all UC objects when handling dependencies ([#3556](#3556)). * Added state upgrader to remove `max_clusters_per_user` and ebs volume attributes set to zero ([#3551](#3551)). * Exporter: track removed objects during the `Emit` phase ([#3554](#3554)). * Jobs GoSDK schema migration ([#3532](#3532)). * Exporter: optimize generation of `databricks_group_member` resource ([#3559](#3559)). ### Documentation Changes ### Exporter ### Internal Changes * - * - * -
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Right now, if we discover any UC dependency, we emit upstream objects as well, such as schemas, catalogs, etc., but in their
Importoperations we're doing all nested objects unnecessarily - so emit one init script from a UC Volume may lead to emitting of the whole UC Catalog with all schemas/tables/volumes...With this change, we list all nested objects (schemas in catalog, tables/models/volumes in a schema) only if these services are explicitly specified in the
-listingoption.Fixes #3555
Tests
make testrun locallydocs/folderinternal/acceptance