Correctly overwrite local state if remote state is newer #1008
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A bug in the code that pulls the remote state could cause the local state to be empty instead of a copy of the remote state. This happened only if the local state was present and stale, compared to the remote version. We correctly checked for the state serial to see if the local state had to be replaced but didn't seek back on the remote state before writing it out. Because the staleness check would read the remote state in full, copying from the same reader would immediately yield an EOF. This change includes: * A fix for this problem. * Unit tests for state pull and push mutators that rely on a mocked filer. * An integration test that deploys the same bundle from multiple paths, triggering the staleness logic.
pietern
commented
Nov 23, 2023
| OverwriteIfExists WriteMode = iota | ||
| CreateParentDirectories = iota << 1 | ||
| OverwriteIfExists WriteMode = 1 << iota | ||
| CreateParentDirectories |
There was a problem hiding this comment.
These changes were necessary to make the mocks work; turns out they were incorrect to begin with.
pietern
commented
Nov 23, 2023
mgyucht
reviewed
Nov 24, 2023
There was a problem hiding this comment.
This seems alright to me, but I would defer to @andrewnester or @shreyas-goenka who probably have more context here.
andrewnester
approved these changes
Nov 24, 2023
Merged
pietern
added a commit
that referenced
this pull request
Nov 29, 2023
This release includes the new `databricks labs` command to install, manage, and run Databricks Labs projects. CLI: * Add `--debug` as shortcut for `--log-level debug` ([#964](#964)). * Improved usability of `databricks auth login ... --configure-cluster` ([#956](#956)). * Make `databricks configure` save only explicit fields ([#973](#973)). * Add `databricks labs` command group ([#914](#914)). * Tolerate missing .databrickscfg file during `databricks auth login` ([#1003](#1003)). * Add `--configure-cluster` flag to configure command ([#1005](#1005)). * Fix bug where the account or workspace client could be `nil` ([#1020](#1020)). Bundles: * Do not allow empty descriptions for bundle template inputs ([#967](#967)). * Added support for top-level permissions ([#928](#928)). * Allow jobs to be manually unpaused in development mode ([#885](#885)). * Fix template initialization from current working directory ([#976](#976)). * Add `--tag` and `--branch` options to bundle init command ([#975](#975)). * Work around DLT issue with `$PYTHONPATH` not being set correctly ([#999](#999)). * Enable `spark_jar_task` with local JAR libraries ([#993](#993)). * Pass `USERPROFILE` environment variable to Terraform ([#1001](#1001)). * Improve error message when path is not a bundle template ([#985](#985)). * Correctly overwrite local state if remote state is newer ([#1008](#1008)). * Add mlops-stacks to the default `databricks bundle init` prompt ([#988](#988)). * Do not add wheel content hash in uploaded Python wheel path ([#1015](#1015)). * Do not replace pipeline libraries if there are no matches for pattern ([#1021](#1021)). Internal: * Update CLI version in the VS Code extension during release ([#1014](#1014)). API Changes: * Changed `databricks functions create` command . New request type is . * Changed `databricks metastores create` command with new required argument order. * Removed `databricks metastores enable-optimization` command. * Removed `databricks account o-auth-enrollment` command group. * Removed `databricks apps delete` command. * Removed `databricks apps get` command. * Added `databricks apps delete-app` command. * Added `databricks apps get-app` command. * Added `databricks apps get-app-deployment-status` command. * Added `databricks apps get-apps` command. * Added `databricks apps get-events` command. * Added `databricks account network-connectivity` command group. OpenAPI commit 22f09783eb8a84d52026f856be3b2068f9498db3 (2023-11-23) Dependency updates: * Bump golang.org/x/term from 0.13.0 to 0.14.0 ([#981](#981)). * Bump github.com/hashicorp/terraform-json from 0.17.1 to 0.18.0 ([#979](#979)). * Bump golang.org/x/oauth2 from 0.13.0 to 0.14.0 ([#982](#982)). * Bump github.com/databricks/databricks-sdk-go from 0.24.0 to 0.25.0 ([#980](#980)). * Bump github.com/databricks/databricks-sdk-go from 0.25.0 to 0.26.0 ([#1019](#1019)).
github-merge-queue bot
pushed a commit
that referenced
this pull request
Nov 29, 2023
This release includes the new `databricks labs` command to install, manage, and run Databricks Labs projects. CLI: * Add `--debug` as shortcut for `--log-level debug` ([#964](#964)). * Improved usability of `databricks auth login ... --configure-cluster` ([#956](#956)). * Make `databricks configure` save only explicit fields ([#973](#973)). * Add `databricks labs` command group ([#914](#914)). * Tolerate missing .databrickscfg file during `databricks auth login` ([#1003](#1003)). * Add `--configure-cluster` flag to configure command ([#1005](#1005)). * Fix bug where the account or workspace client could be `nil` ([#1020](#1020)). Bundles: * Do not allow empty descriptions for bundle template inputs ([#967](#967)). * Added support for top-level permissions ([#928](#928)). * Allow jobs to be manually unpaused in development mode ([#885](#885)). * Fix template initialization from current working directory ([#976](#976)). * Add `--tag` and `--branch` options to bundle init command ([#975](#975)). * Work around DLT issue with `$PYTHONPATH` not being set correctly ([#999](#999)). * Enable `spark_jar_task` with local JAR libraries ([#993](#993)). * Pass `USERPROFILE` environment variable to Terraform ([#1001](#1001)). * Improve error message when path is not a bundle template ([#985](#985)). * Correctly overwrite local state if remote state is newer ([#1008](#1008)). * Add mlops-stacks to the default `databricks bundle init` prompt ([#988](#988)). * Do not add wheel content hash in uploaded Python wheel path ([#1015](#1015)). * Do not replace pipeline libraries if there are no matches for pattern ([#1021](#1021)). Internal: * Update CLI version in the VS Code extension during release ([#1014](#1014)). API Changes: * Changed `databricks functions create` command. * Changed `databricks metastores create` command with new required argument order. * Removed `databricks metastores enable-optimization` command. * Removed `databricks account o-auth-enrollment` command group. * Removed `databricks apps delete` command. * Removed `databricks apps get` command. * Added `databricks apps delete-app` command. * Added `databricks apps get-app` command. * Added `databricks apps get-app-deployment-status` command. * Added `databricks apps get-apps` command. * Added `databricks apps get-events` command. * Added `databricks account network-connectivity` command group. OpenAPI commit 22f09783eb8a84d52026f856be3b2068f9498db3 (2023-11-23) Dependency updates: * Bump golang.org/x/term from 0.13.0 to 0.14.0 ([#981](#981)). * Bump github.com/hashicorp/terraform-json from 0.17.1 to 0.18.0 ([#979](#979)). * Bump golang.org/x/oauth2 from 0.13.0 to 0.14.0 ([#982](#982)). * Bump github.com/databricks/databricks-sdk-go from 0.24.0 to 0.25.0 ([#980](#980)). * Bump github.com/databricks/databricks-sdk-go from 0.25.0 to 0.26.0 ([#1019](#1019)).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
A bug in the code that pulls the remote state could cause the local state to be empty instead of a copy of the remote state. This happened only if the local state was present and stale when compared to the remote version.
We correctly checked for the state serial to see if the local state had to be replaced but didn't seek back on the remote state before writing it out. Because the staleness check would read the remote state in full, copying from the same reader would immediately yield an EOF.
Tests
Both failed prior to the fix and now pass.