Use local Terraform state only when lineage match #1588
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Integration tests are passing. |
shreyas-goenka
changed the title
pietern
changed the title
pietern
reviewed
Jul 15, 2024
pietern
reviewed
Jul 17, 2024
pietern
approved these changes
Jul 18, 2024
andrewnester
added a commit
that referenced
this pull request
Jul 18, 2024
CLI: * [Fix] Do not buffer files in memory when downloading ([#1599](#1599)). Bundles: * Allow artifacts (JARs, wheels) to be uploaded to UC Volumes ([#1591](#1591)). * Upgrade TF provider to 1.48.3 ([#1600](#1600)). * Fixed job name normalisation for bundle generate ([#1601](#1601)). Internal: * Add UUID to uniquely identify a deployment state ([#1595](#1595)). * Track multiple locations associated with a `dyn.Value` ([#1510](#1510)). * Attribute Terraform API requests the CLI ([#1598](#1598)). * Use local Terraform state only when lineage match ([#1588](#1588)). * Implement readahead cache for Workspace API calls ([#1582](#1582)). Dependency updates: * Bump github.com/databricks/databricks-sdk-go from 0.43.0 to 0.43.2 ([#1594](#1594)).
Closed
andrewnester
added a commit
that referenced
this pull request
Jul 18, 2024
CLI: * Do not buffer files in memory when downloading ([#1599](#1599)). Bundles: * Allow artifacts (JARs, wheels) to be uploaded to UC Volumes ([#1591](#1591)). * Upgrade TF provider to 1.48.3 ([#1600](#1600)). * Fixed job name normalisation for bundle generate ([#1601](#1601)). Internal: * Add UUID to uniquely identify a deployment state ([#1595](#1595)). * Track multiple locations associated with a `dyn.Value` ([#1510](#1510)). * Attribute Terraform API requests the CLI ([#1598](#1598)). * Implement readahead cache for Workspace API calls ([#1582](#1582)). * Use local Terraform state only when lineage match ([#1588](#1588)). Dependency updates: * Bump github.com/databricks/databricks-sdk-go from 0.43.0 to 0.43.2 ([#1594](#1594)).
Merged
github-merge-queue bot
pushed a commit
that referenced
this pull request
Jul 18, 2024
CLI: * Do not buffer files in memory when downloading ([#1599](#1599)). Bundles: * Allow artifacts (JARs, wheels) to be uploaded to UC Volumes ([#1591](#1591)). * Upgrade TF provider to 1.48.3 ([#1600](#1600)). * Fixed job name normalisation for bundle generate ([#1601](#1601)). Internal: * Add UUID to uniquely identify a deployment state ([#1595](#1595)). * Track multiple locations associated with a `dyn.Value` ([#1510](#1510)). * Attribute Terraform API requests the CLI ([#1598](#1598)). * Implement readahead cache for Workspace API calls ([#1582](#1582)). * Use local Terraform state only when lineage match ([#1588](#1588)). * Add read-only mode for extension aware workspace filer ([#1609](#1609)). Dependency updates: * Bump github.com/databricks/databricks-sdk-go from 0.43.0 to 0.43.2 ([#1594](#1594)).
witi83
pushed a commit
to witi83/databricks-cli
that referenced
this pull request
Jul 29, 2024
## Changes DABs deployments should be isolated if `root_path` and workspace host are different. This PR fixes a bug where local terraform state gets piggybacked if the same cwd is used to deploy two isolated deployments for the same bundle target. This can happen if: 1. A user switches to a different identity on the same machine. 2. The workspace host URL the bundle/target points to is changed. 3. A user changes the `root_path` while doing bundle development. To solve this problem we rely on the lineage field available in the terraform state, which is a uuid identifying unique terraform deployments. There's a 1:1 mapping between a terraform deployment and a bundle deployment. For more details on how lineage works in terraform, see: https://developer.hashicorp.com/terraform/language/state/backends#manual-state-pull-push ## Tests Manually verified that changing the identity no longer results in the incorrect terraform state being used. Also, new unit tests are added.
witi83
pushed a commit
to witi83/databricks-cli
that referenced
this pull request
Jul 29, 2024
CLI: * Do not buffer files in memory when downloading ([databricks#1599](databricks#1599)). Bundles: * Allow artifacts (JARs, wheels) to be uploaded to UC Volumes ([databricks#1591](databricks#1591)). * Upgrade TF provider to 1.48.3 ([databricks#1600](databricks#1600)). * Fixed job name normalisation for bundle generate ([databricks#1601](databricks#1601)). Internal: * Add UUID to uniquely identify a deployment state ([databricks#1595](databricks#1595)). * Track multiple locations associated with a `dyn.Value` ([databricks#1510](databricks#1510)). * Attribute Terraform API requests the CLI ([databricks#1598](databricks#1598)). * Implement readahead cache for Workspace API calls ([databricks#1582](databricks#1582)). * Use local Terraform state only when lineage match ([databricks#1588](databricks#1588)). * Add read-only mode for extension aware workspace filer ([databricks#1609](databricks#1609)). Dependency updates: * Bump github.com/databricks/databricks-sdk-go from 0.43.0 to 0.43.2 ([databricks#1594](databricks#1594)).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
DABs deployments should be isolated if
root_pathand workspace host are different. This PR fixes a bug where local terraform state gets piggybacked if the same cwd is used to deploy two isolated deployments for the same bundle target. This can happen if:root_pathwhile doing bundle development.To solve this problem we rely on the lineage field available in the terraform state, which is a uuid identifying unique terraform deployments. There's a 1:1 mapping between a terraform deployment and a bundle deployment.
For more details on how lineage works in terraform, see: https://developer.hashicorp.com/terraform/language/state/backends#manual-state-pull-push
Tests
Manually verified that changing the identity no longer results in the incorrect terraform state being used. Also, new unit tests are added.