Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add account-level API support for Unity Catalog objects #2182

Merged
merged 73 commits into from Aug 24, 2023

Conversation

nkvuong
Copy link
Contributor

@nkvuong nkvuong commented Mar 31, 2023

Changes

  • Add account-level support for databricks_metastore, databricks_metastore_assignment, databricks_storage_credential and databricks_metastore_data_access_configuration
  • Refactor all resources to use go sdk
  • Refactor databricks_metastore_data_access_configuration to use storage-credentials endpoint

Tests

  • make test run locally
  • relevant change in docs/ folder
  • covered with integration tests in internal/acceptance
  • relevant acceptance tests are passing
  • using Go SDK

@nkvuong nkvuong changed the title Add account-level API support for UC resources Add account-level API support for databricks_metastore and databricks_metastore_assignment Mar 31, 2023
@nkvuong
Copy link
Contributor Author

nkvuong commented Mar 31, 2023

@nfx Need to change qa.ResourceFixture to be able to mock an AccountClient as well

nkvuong and others added 7 commits April 5, 2023 16:56
…lter (#2185)

* fix `databricks_service_principals` data source issue with empty filter

* fix acc tests
…by only changing `comment` field (#2114)

Tested manually for the following cases

Without this PR the provider recreates the entire workspace on a token update
With changes in this PR only the token is refreshed
When both token and storage_configuration_id are changed then the entire workspace is recreated
Additional unit tests also added that allow checks that patch workspace calls are not made when only token is changed

Also added an integration test to check tokens are successfully updated
…im service (#2181)

* Exclude roles in scim API list calls

* more test fixes
* Update SDK to v0.6.0

* go mod tidy
@codecov-commenter
Copy link

codecov-commenter commented Apr 6, 2023

Codecov Report

Merging #2182 (5b28e6e) into master (754aad4) will decrease coverage by 1.44%.
Report is 48 commits behind head on master.
The diff coverage is 75.44%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2182      +/-   ##
==========================================
- Coverage   88.21%   86.77%   -1.44%     
==========================================
  Files         145      146       +1     
  Lines       11989    12655     +666     
==========================================
+ Hits        10576    10982     +406     
- Misses        930     1108     +178     
- Partials      483      565      +82     
Files Changed Coverage Δ
catalog/resource_catalog.go 71.76% <ø> (ø)
catalog/resource_table.go 100.00% <ø> (ø)
common/client.go 28.57% <0.00%> (-2.47%) ⬇️
common/version.go 80.00% <ø> (ø)
exporter/model.go 91.66% <ø> (ø)
pipelines/resource_pipeline.go 92.37% <ø> (ø)
pools/resource_instance_pool.go 98.90% <ø> (ø)
sql/api/query.go 75.40% <ø> (ø)
workspace/resource_notebook.go 88.88% <ø> (ø)
catalog/resource_storage_credential.go 55.55% <48.23%> (-44.45%) ⬇️
... and 30 more

... and 1 file with indirect coverage changes

@allebacco
Copy link

Any news on this feature? It will be very useful for me.

Thanks

@MitchVL
Copy link

MitchVL commented May 5, 2023

Our infrastructure setup is currently on hold, waiting for this merge.
Any news would be greatly appreciated :)

@alexandremoyrand
Copy link

Hello, any news on this PR ?

@vsluc
Copy link

vsluc commented Jun 11, 2023

This is a very important feature for rolling out our stack and a significant blocker for our Databricks adoption. Could you please prioritize this PR?

@nkvuong nkvuong changed the title Add account-level API support for databricks_metastore and databricks_metastore_assignment Add account-level API support for databricks_metastore, databricks_metastore_data_access and databricks_metastore_assignment Jul 4, 2023
@nkvuong nkvuong requested review from a team July 4, 2023 18:51
@nkvuong nkvuong changed the title Add account-level API support for databricks_metastore, databricks_metastore_data_access and databricks_metastore_assignment [WIP] Add account-level API support for databricks_metastore, databricks_metastore_data_access and databricks_metastore_assignment Jul 4, 2023
@nkvuong
Copy link
Contributor Author

nkvuong commented Jul 4, 2023

This is still WIP because account-level API is not fully working, so acceptance tests cannot be run

@nkvuong nkvuong changed the title [WIP] Add account-level API support for databricks_metastore, databricks_metastore_data_access and databricks_metastore_assignment [WIP] Add account-level API support for Unity Catalog objects Jul 5, 2023
@nkvuong
Copy link
Contributor Author

nkvuong commented Aug 23, 2023

@mgyucht addressed all feedback :)

Copy link
Contributor

@mgyucht mgyucht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two nits, otherwise LGTM provided our integration tests pass.

func TestUcAccMetastore(t *testing.T) {
cloudEnv := os.Getenv("CLOUD_ENV")
switch cloudEnv {
case "ucacct":
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, you're completely right. I didn't realize there was another set of names here... we need to standardize this a bit but otherwise I think it is good.

DefaultDacID string `json:"default_data_access_config_id,omitempty" tf:"suppress_diff"`
Owner string `json:"owner,omitempty" tf:"computed"`
MetastoreID string `json:"metastore_id,omitempty" tf:"computed"`
WorkspaceIDs []int64 `json:"workspace_ids,omitempty" tf:"computed"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This field is not in the new structure?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could keep this for compatibility, but it does not exist in the SDK anymore

common/client.go Outdated
@@ -81,6 +81,22 @@ func (c *DatabricksClient) AccountClient() (*databricks.AccountClient, error) {
return acc, nil
}

func (c *DatabricksClient) WorkspaceOrAccountRequest(accCallback func(*databricks.AccountClient) error, wsCallback func(*databricks.WorkspaceClient) error) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Micronit: can we name it AccountOrWorkspaceRequest (since the account callback comes first, and the workspace callback comes second)?

},
Method: "DELETE",
Resource: "/api/2.1/unity-catalog/metastores/abc",
ExpectedRequest: map[string]interface{}{"-": "abc", "force": true},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yuck! The next release of the Go SDK will fix this request to not include the - key.

@@ -25,8 +26,15 @@ func TestCreateMetastore(t *testing.T) {
},
},
{
Method: "GET",
Method: "PATCH",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like this is introduced because in the old updateFunctionFactory, we checked if there were any fields that needed to be updated. Now, we always call update, even when it is a no-op. I think this should be safe (it just adds one extra round trip that doesn't change the underlying resource).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes - we could try to be more precise and check for presence of fields that are only updatable, but it feels like too much extra logic to maintain

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think something should be possible with reflection (i.e. check whether there is any field in the update struct that is not the zero value for its type), but I would rather scope that out of this PR.

@nkvuong
Copy link
Contributor Author

nkvuong commented Aug 24, 2023

integration tests passed

@nfx nfx added this pull request to the merge queue Aug 24, 2023
Merged via the queue into master with commit c521a1a Aug 24, 2023
4 checks passed
@nfx nfx deleted the feature/uc-account-api branch August 24, 2023 19:44
@tanmay-db tanmay-db mentioned this pull request Aug 25, 2023
5 tasks
@barracudarin
Copy link

thanks for the work on this!

is there work planned to hit the remaining Unity Catalog objects that still need to be managed with workspace providers? for my use-case, the databricks_external_location and databricks_grants on Unity Catalog objects still require a workspace-configured provider.

@ConnorBrereton
Copy link

Thank you for your work on this. Super impactful for a few of my customers!

nkvuong added a commit that referenced this pull request Sep 7, 2023
* first draft

* account client check

* account client check

* Fixed `databricks_service_principals` data source issue with empty filter (#2185)

* fix `databricks_service_principals` data source issue with empty filter

* fix acc tests

* Allow rotating `token` block in `databricks_mws_workspaces` resource by only changing `comment` field (#2114)

Tested manually for the following cases

Without this PR the provider recreates the entire workspace on a token update
With changes in this PR only the token is refreshed
When both token and storage_configuration_id are changed then the entire workspace is recreated
Additional unit tests also added that allow checks that patch workspace calls are not made when only token is changed

Also added an integration test to check tokens are successfully updated

* Excludes roles in scim API list calls to reduce load on databricks scim service (#2181)

* Exclude roles in scim API list calls

* more test fixes

* Update SDK to v0.6.0 (#2186)

* Update SDK to v0.6.0

* go mod tidy

* update sdk to 0.7.0

* add integration tests

* fix acceptance tests

* fix tests

* add account-level API support for `metastore_data_access`

* add account API support for `databricks_storage_credential`

* address feedback

* refactor to `WorkspaceOrAccountRequest`

* fix acceptance tests

* Release v1.21.0 (#2471)

Release v1.21.0 of the Terraform Provider for Databricks.

 * Added condition_task to the [`databricks_job`](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) resource (private preview) ([#2459](#2459)).
 * Added `AccountData`, `AccountClient` and define generic databricks data utilites for defining workspace and account-level data sources ([#2429](#2429)).
 * Added documentation link to existing Databricks Terraform modules ([#2439](#2439)).
 * Added experimental compute field to [databricks_job](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) resource ([#2401](#2401)).
 * Added import example to doc for [databricks_group_member](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/group_member) resource ([#2453](#2453)).
 * Added support for subscriptions in dashboards & alert SQL tasks in [databricks_job](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) ([#2447](#2447)).
 * Fixed model serving integration test ([#2460](#2460), [#2461](#2461)).
 * Fixed [databricks_job](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) resource file arrival trigger parameter name ([#2438](#2438)).
 * Fixed catalog_workspace_binding_test ([#2463](#2463), [#2451](#2451)).

No breaking changes in this release.

* Install mlflow cluster using  in model serving test if the cluster is already running (#2470)

* Bump golang.org/x/mod from 0.11.0 to 0.12.0 (#2462)

Bumps [golang.org/x/mod](https://github.com/golang/mod) from 0.11.0 to 0.12.0.
- [Commits](golang/mod@v0.11.0...v0.12.0)

---
updated-dependencies:
- dependency-name: golang.org/x/mod
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Exporter: make resource names more unique to avoid duplicate resources errors (#2452)

This includes following changes:

* Add user ID to the `databricks_user` resource name to avoid clashes on names like, `user+1` and `user_1`
* Add user/sp/group ID to the name of the `databricks_group_member` resource
* Remove too aggressive name normalization pattern that also leads to the generation of
  the duplicate resource names for different resources

* Add documentation notes about legacy cluster type & data access (#2437)

* Add documentation notes about legacy cluster type & data access

* Update docs/resources/cluster.md

Co-authored-by: Miles Yucht <miles@databricks.com>

* Update docs/resources/mount.md

Co-authored-by: Miles Yucht <miles@databricks.com>

---------

Co-authored-by: Miles Yucht <miles@databricks.com>

* Use random catalog name in SQL table integration tests (#2473)

The fixed value prevented concurrent integration test runs.

* Link model serving docs to top level README (#2474)

* Add one more item to the troubleshooting guide (#2477)

It's related to use OAuth for authentication but not providing `account_id` in the
provider configuration.

* Added `databricks_access_control_rule_set` resource for managing account-level access (#2371)

* Added `acl_principal_id` attribute to `databricks_user`, `databricks_group` & `databricks_service_principal` for easier use with `databricks_access_control_rule_set` (#2485)

It should simplify specification of principals in the `databricks_access_control_rule_set`
so instead of this (string with placeholders):

```
   grant_rules {
     principals = ["groups/${databricks_group.ds.display_name}"]
     role       = "roles/servicePrincipal.user"
   }
```

it will be simpler to refer like this:

```
   grant_rules {
     principals = [databricks_group.ds.acl_principal_id]
     role       = "roles/servicePrincipal.user"
   }
```

* Added support for Unity Catalog `databricks_metastores` data source  (#2017)

* add documentation for databricks_metastores data source

* add API endpoint for listing metastores

* add metastores data resource

* add test for metastores data source

* add metastores datasource to resource mapping

* fix reference to wrong resource docs

* add a Metastores struct for the response of the API, use this in the datSource

* update terraform specific object attributes

* add new data test

* remove slice_set property from MetastoreData

* use databricks-go-sdk for data_metastore.go

* removed listMetastores endpoint since it's unused

* make sure tests also use the unitycatalog.MetastoreInfo from the sdk

* remove redundant resource

* test -dev

* fix

* fmt

* cleanup

* Added AccountClient to DatabricksClient and AccountData

* upd

* cleanup

* accountLevel

* upd

* add example

* list

* cleanup

* docs

* remove dead code

* wip

* use maps

* upd

* cleanup

* comments

* -

* remove redundant test

---------

Co-authored-by: Tanmay Rustagi <tanmay.rustagi@databricks.com>
Co-authored-by: vuong-nguyen <44292934+nkvuong@users.noreply.github.com>

* Added support for Unity Catalog `databricks_metastore` data source (#2492)

Enable fetching account level metastore information through id for a single metastore.

* Supported new Delve binary name format (#2497)

https://github.com/go-delve/delve/blob/master/CHANGELOG.md#1210-2023-06-23 changes the naming of the delve debug binary. This PR changes isInDebug to accommodate old and new versions of Delve.

* Add code owners for Terraform (#2498)

* Removed unused dlvLoadConfig configuration from settings.json (#2499)

* Fix provider after updating SDK to 0.13 (#2494)

* Fix provider after updating SDK to 0.13

* add unit test

* split test

* Added `control_run_state` flag to the `databricks_job` resource for continuous jobs (#2466)

This PR introduces a new flag, control_run_state, to replace the always_running flag. This flag only applies to continuous jobs. Its behavior is described below:

For jobs with pause_status = PAUSED, it is a no-op on create and stops the active job run on update (if applicable).
For jobs with pause_status = UNPAUSED, it starts a job run on create and stops the active job run on update (if applicable).
The job does not need to be started, as that is handled by the Jobs service itself.

This fixes #2130.

* Added exporter for `databricks_workspace_file` resource (#2493)

* Preliminary changes to make workspace files implementation

- make `NotebooksAPI.List` to return directories as well when called in the recursive
  mode (same as non-recursive behavior)
- Because of that, remove the separate `ListDirectories`
- Extend `workspace.ObjectStatus` with additional fields (will be required for
  incremental notebooks export)
- Cache listing of all workspace objects, and then use it for all operations - list
  notebooks, list directories, list workspace files

* Added exporting of workspace files

---------

Co-authored-by: Miles Yucht <miles@databricks.com>

* Supported boolean values in `databricks_sql_alert` alerts (#2506)

* Added more common issues for troubleshooting (#2486)

* add troubleshooting

* fix doc category

---------

Co-authored-by: Miles Yucht <miles@databricks.com>

* Fixed handling of comments in `databricks_sql_table` resource (#2472)

* column comments and single quote escape

* Delimiter collision avoidance table comment

* compatible with user single quote escape

* unit tests for parseComment

* corrected fmt

---------

Co-authored-by: Miles Yucht <miles@databricks.com>

* Added clarification that `databricks_schema` and `databricks_sql_table` should be imported by their full name, not just by name (#2491)

Co-authored-by: Miles Yucht <miles@databricks.com>

* Updated `databricks_user` with `force = true` to check for error message prefix (#2510)

This fixes #2500

* fix force delete

* remove orphaned code

* fix acceptance tests

* upgrade go sdk

* fix metastoreinfo struct

* docs update

* fix acceptance tests

* fix tests

* updated docs

* fix tests

* rename test

* update tests

* fix tests

* fix test

* add state upgrader

* fix struct

* fix tests

* feedback

* feedback

* fix acc test

* fix test

* fix test

* fix test

* feedback

* fix acc tests

* feedback

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>
Co-authored-by: Serge Smertin <259697+nfx@users.noreply.github.com>
Co-authored-by: Miles Yucht <miles@databricks.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alex Ott <alexey.ott@databricks.com>
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
Co-authored-by: Gautham Sunjay <gauthamsunjay17@gmail.com>
Co-authored-by: guillesd <74136033+guillesd@users.noreply.github.com>
Co-authored-by: Tanmay Rustagi <tanmay.rustagi@databricks.com>
Co-authored-by: Tanmay Rustagi <88379306+tanmay-db@users.noreply.github.com>
Co-authored-by: Fabian Jakobs <fabian.jakobs@databricks.com>
Co-authored-by: klus <lus.karol@gmail.com>
nkvuong added a commit that referenced this pull request Oct 3, 2023
* first draft

* account client check

* account client check

* Fixed `databricks_service_principals` data source issue with empty filter (#2185)

* fix `databricks_service_principals` data source issue with empty filter

* fix acc tests

* Allow rotating `token` block in `databricks_mws_workspaces` resource by only changing `comment` field (#2114)

Tested manually for the following cases

Without this PR the provider recreates the entire workspace on a token update
With changes in this PR only the token is refreshed
When both token and storage_configuration_id are changed then the entire workspace is recreated
Additional unit tests also added that allow checks that patch workspace calls are not made when only token is changed

Also added an integration test to check tokens are successfully updated

* Excludes roles in scim API list calls to reduce load on databricks scim service (#2181)

* Exclude roles in scim API list calls

* more test fixes

* Update SDK to v0.6.0 (#2186)

* Update SDK to v0.6.0

* go mod tidy

* update sdk to 0.7.0

* add integration tests

* fix acceptance tests

* fix tests

* add account-level API support for `metastore_data_access`

* add account API support for `databricks_storage_credential`

* address feedback

* refactor to `WorkspaceOrAccountRequest`

* fix acceptance tests

* Release v1.21.0 (#2471)

Release v1.21.0 of the Terraform Provider for Databricks.

## Changes
 * Added condition_task to the [`databricks_job`](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) resource (private preview) ([#2459](#2459)).
 * Added `AccountData`, `AccountClient` and define generic databricks data utilites for defining workspace and account-level data sources ([#2429](#2429)).
 * Added documentation link to existing Databricks Terraform modules ([#2439](#2439)).
 * Added experimental compute field to [databricks_job](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) resource ([#2401](#2401)).
 * Added import example to doc for [databricks_group_member](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/group_member) resource ([#2453](#2453)).
 * Added support for subscriptions in dashboards & alert SQL tasks in [databricks_job](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) ([#2447](#2447)).
 * Fixed model serving integration test ([#2460](#2460), [#2461](#2461)).
 * Fixed [databricks_job](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) resource file arrival trigger parameter name ([#2438](#2438)).
 * Fixed catalog_workspace_binding_test ([#2463](#2463), [#2451](#2451)).

No breaking changes in this release.

* Install mlflow cluster using  in model serving test if the cluster is already running (#2470)

* Bump golang.org/x/mod from 0.11.0 to 0.12.0 (#2462)

Bumps [golang.org/x/mod](https://github.com/golang/mod) from 0.11.0 to 0.12.0.
- [Commits](golang/mod@v0.11.0...v0.12.0)

---
updated-dependencies:
- dependency-name: golang.org/x/mod
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Exporter: make resource names more unique to avoid duplicate resources errors (#2452)

This includes following changes:

* Add user ID to the `databricks_user` resource name to avoid clashes on names like, `user+1` and `user_1`
* Add user/sp/group ID to the name of the `databricks_group_member` resource
* Remove too aggressive name normalization pattern that also leads to the generation of
  the duplicate resource names for different resources

* Add documentation notes about legacy cluster type & data access (#2437)

* Add documentation notes about legacy cluster type & data access

* Update docs/resources/cluster.md

Co-authored-by: Miles Yucht <miles@databricks.com>

* Update docs/resources/mount.md

Co-authored-by: Miles Yucht <miles@databricks.com>

---------

Co-authored-by: Miles Yucht <miles@databricks.com>

* Use random catalog name in SQL table integration tests (#2473)

The fixed value prevented concurrent integration test runs.

* Link model serving docs to top level README (#2474)

* Add one more item to the troubleshooting guide (#2477)

It's related to use OAuth for authentication but not providing `account_id` in the
provider configuration.

* Added `databricks_access_control_rule_set` resource for managing account-level access (#2371)

* Added `acl_principal_id` attribute to `databricks_user`, `databricks_group` & `databricks_service_principal` for easier use with `databricks_access_control_rule_set` (#2485)

It should simplify specification of principals in the `databricks_access_control_rule_set`
so instead of this (string with placeholders):

```
   grant_rules {
     principals = ["groups/${databricks_group.ds.display_name}"]
     role       = "roles/servicePrincipal.user"
   }
```

it will be simpler to refer like this:

```
   grant_rules {
     principals = [databricks_group.ds.acl_principal_id]
     role       = "roles/servicePrincipal.user"
   }
```

* Added support for Unity Catalog `databricks_metastores` data source  (#2017)

* add documentation for databricks_metastores data source

* add API endpoint for listing metastores

* add metastores data resource

* add test for metastores data source

* add metastores datasource to resource mapping

* fix reference to wrong resource docs

* add a Metastores struct for the response of the API, use this in the datSource

* update terraform specific object attributes

* add new data test

* remove slice_set property from MetastoreData

* use databricks-go-sdk for data_metastore.go

* removed listMetastores endpoint since it's unused

* make sure tests also use the unitycatalog.MetastoreInfo from the sdk

* remove redundant resource

* test -dev

* fix

* fmt

* cleanup

* Added AccountClient to DatabricksClient and AccountData

* upd

* cleanup

* accountLevel

* upd

* add example

* list

* cleanup

* docs

* remove dead code

* wip

* use maps

* upd

* cleanup

* comments

* -

* remove redundant test

---------

Co-authored-by: Tanmay Rustagi <tanmay.rustagi@databricks.com>
Co-authored-by: vuong-nguyen <44292934+nkvuong@users.noreply.github.com>

* Added support for Unity Catalog `databricks_metastore` data source (#2492)

Enable fetching account level metastore information through id for a single metastore.

* Supported new Delve binary name format (#2497)

https://github.com/go-delve/delve/blob/master/CHANGELOG.md#1210-2023-06-23 changes the naming of the delve debug binary. This PR changes isInDebug to accommodate old and new versions of Delve.

* Add code owners for Terraform (#2498)

* Removed unused dlvLoadConfig configuration from settings.json (#2499)

* Fix provider after updating SDK to 0.13 (#2494)

* Fix provider after updating SDK to 0.13

* add unit test

* split test

* Added `control_run_state` flag to the `databricks_job` resource for continuous jobs (#2466)

This PR introduces a new flag, control_run_state, to replace the always_running flag. This flag only applies to continuous jobs. Its behavior is described below:

For jobs with pause_status = PAUSED, it is a no-op on create and stops the active job run on update (if applicable).
For jobs with pause_status = UNPAUSED, it starts a job run on create and stops the active job run on update (if applicable).
The job does not need to be started, as that is handled by the Jobs service itself.

This fixes #2130.

* Added exporter for `databricks_workspace_file` resource (#2493)

* Preliminary changes to make workspace files implementation

- make `NotebooksAPI.List` to return directories as well when called in the recursive
  mode (same as non-recursive behavior)
- Because of that, remove the separate `ListDirectories`
- Extend `workspace.ObjectStatus` with additional fields (will be required for
  incremental notebooks export)
- Cache listing of all workspace objects, and then use it for all operations - list
  notebooks, list directories, list workspace files

* Added exporting of workspace files

---------

Co-authored-by: Miles Yucht <miles@databricks.com>

* Supported boolean values in `databricks_sql_alert` alerts (#2506)

* Added more common issues for troubleshooting (#2486)

* add troubleshooting

* fix doc category

---------

Co-authored-by: Miles Yucht <miles@databricks.com>

* Fixed handling of comments in `databricks_sql_table` resource (#2472)

* column comments and single quote escape

* Delimiter collision avoidance table comment

* compatible with user single quote escape

* unit tests for parseComment

* corrected fmt

---------

Co-authored-by: Miles Yucht <miles@databricks.com>

* Added clarification that `databricks_schema` and `databricks_sql_table` should be imported by their full name, not just by name (#2491)

Co-authored-by: Miles Yucht <miles@databricks.com>

* Updated `databricks_user` with `force = true` to check for error message prefix (#2510)

This fixes #2500

* fix force delete

* remove orphaned code

* fix acceptance tests

* upgrade go sdk

* fix metastoreinfo struct

* docs update

* fix acceptance tests

* fix tests

* updated docs

* fix tests

* rename test

* update tests

* fix tests

* fix test

* add state upgrader

* fix struct

* fix tests

* feedback

* feedback

* fix acc test

* fix test

* fix test

* fix test

* feedback

* fix acc tests

* feedback

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>
Co-authored-by: Serge Smertin <259697+nfx@users.noreply.github.com>
Co-authored-by: Miles Yucht <miles@databricks.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alex Ott <alexey.ott@databricks.com>
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
Co-authored-by: Gautham Sunjay <gauthamsunjay17@gmail.com>
Co-authored-by: guillesd <74136033+guillesd@users.noreply.github.com>
Co-authored-by: Tanmay Rustagi <tanmay.rustagi@databricks.com>
Co-authored-by: Tanmay Rustagi <88379306+tanmay-db@users.noreply.github.com>
Co-authored-by: Fabian Jakobs <fabian.jakobs@databricks.com>
Co-authored-by: klus <lus.karol@gmail.com>
github-merge-queue bot pushed a commit that referenced this pull request Dec 18, 2023
* refactor to support account-level ip acl

* add doc

* add acceptance tests

* Refactor `databricks_schema` to Go SDK (#2572)

* refactor `databricks_schema` to Go SDK

* clean up

* Refactor `databricks_external_location` to Go SDK (#2546)

* refactor external location to Go SDK

* keep force_new

* switch struct to sdk

* Refresh `databricks_grants` with latest permissible grants (#2567)

* refresh latest grants

* fix test

* fix typo

* add test

* Fixed databricks_access_control_rule_set integration test in Azure (#2591)

* Update Go SDK to v0.17.0 (#2599)

* Update Go SDK to 0.17.0

* go mod tidy

* RunJobTask job_id type fix (#2588)

* fix run job id bug

* update tests

* update exporter tests

* Add `databricks_connection` resource to support Lakehouse Federation (#2528)

* first draft

* add foreign catalog

* update doc

* Fixed `databricks_job` resource to clear instance-specific attributes when `instance_pool_id` is specified (#2507)

NodeTypeID cannot be set in jobsAPI.Update() if InstancePoolID is specified.
If both are specified, assume InstancePoolID takes precedence and NodeTypeID is only computed.

Closes #2502.
Closes #2141.

* Added `full_refresh` attribute to the `pipeline_task` in `databricks_job` (#2444)

This allows to force full refresh of the pipeline from the job.

This fixes #2362

* Configured merge queue for the provider (#2533)

* misc doc updates (#2516)

* Bump github.com/databricks/databricks-sdk-go from 0.13.0 to 0.14.1 (#2523)

Bumps [github.com/databricks/databricks-sdk-go](https://github.com/databricks/databricks-sdk-go) from 0.13.0 to 0.14.1.
- [Release notes](https://github.com/databricks/databricks-sdk-go/releases)
- [Changelog](https://github.com/databricks/databricks-sdk-go/blob/main/CHANGELOG.md)
- [Commits](databricks/databricks-sdk-go@v0.13.0...v0.14.1)

---
updated-dependencies:
- dependency-name: github.com/databricks/databricks-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Miles Yucht <miles@databricks.com>

* Fix IP ACL read (#2515)

* Add support for `USE_MARKETPLACE_ASSETS` privilege to metastore (#2505)

* Update docs to include USE_MARKETPLACE_ASSETS privilege

* Add USE_MARKETPLACE_ASSETS to metastore privileges

* Add git job_source to job resource (#2538)

* Add git job_source to job resource

* lint

* fix test

* Use go sdk type

* Allow search SQL Warehouses by name in `databricks_sql_warehouse` data source (#2458)

* Allow search SQL Warehouses by name in `databricks_sql_warehouse` data source

Right now it's possible to search only by the warehouse ID, but it's not always convenient
although it's possible by using `databricks_sql_warehouses` data source + explicit
filtering.  This PR adds a capability to search by either SQL warehouse name or ID.

This fixes #2443

* Update docs/data-sources/sql_warehouse.md

Co-authored-by: Miles Yucht <miles@databricks.com>

* Address review comments

also change documentation a bit to better match the data source - it was copied from the
resource as-is.

* More fixes from review

* code review comments

---------

Co-authored-by: Miles Yucht <miles@databricks.com>

* Late jobs support (aka health conditions) in `databricks_job` resource (#2496)

* Late jobs support (aka health conditions) in `databricks_job` resource

Added support for `health` block that is used to detect late jobs.  Also, this PR includes
following changes:

* Added `on_duration_warning_threshold_exceeded` attribute to email & webhook notifications (needed for late jobs support)
* Added `notification_settings` on a task level & use jobs & task notification structs from Go SDK
* Reorganized documentation for task block as it's getting more & more attributes

* Update docs/resources/job.md

Co-authored-by: Gabor Ratky <gabor.ratky@databricks.com>

* Update docs/resources/job.md

Co-authored-by: Gabor Ratky <gabor.ratky@databricks.com>

* Update docs/resources/job.md

Co-authored-by: Gabor Ratky <gabor.ratky@databricks.com>

* Update docs/resources/job.md

Co-authored-by: Gabor Ratky <gabor.ratky@databricks.com>

* Update docs/resources/job.md

Co-authored-by: Gabor Ratky <gabor.ratky@databricks.com>

* Update docs/resources/job.md

Co-authored-by: Gabor Ratky <gabor.ratky@databricks.com>

* Update docs/resources/job.md

Co-authored-by: Gabor Ratky <gabor.ratky@databricks.com>

* Update docs/resources/job.md

Co-authored-by: Gabor Ratky <gabor.ratky@databricks.com>

* Update docs/resources/job.md

Co-authored-by: Gabor Ratky <gabor.ratky@databricks.com>

* Update docs/resources/job.md

Co-authored-by: Gabor Ratky <gabor.ratky@databricks.com>

* address review comments

* add list of tasks

* more review chanes

---------

Co-authored-by: Gabor Ratky <gabor.ratky@databricks.com>
Co-authored-by: Miles Yucht <miles@databricks.com>

* feedback

* update struct

* add suppress diff

* fix suppress diff

* fix acceptance tests

* test feedback

* make id a pair

* better sensitive options handling

* reorder id pair

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: marekbrysa <53767523+marekbrysa@users.noreply.github.com>
Co-authored-by: Alex Ott <alexey.ott@databricks.com>
Co-authored-by: Miles Yucht <miles@databricks.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: bvdboom <bvdboom@users.noreply.github.com>
Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>
Co-authored-by: Gabor Ratky <gabor.ratky@databricks.com>

* Documentation changes (#2576)

* add troubleshooting guide for grants/permissions config drifts

* update authentication in index

* feedback

* Exporter: Incremental export of notebooks, SQL objects and some other resources (#2563)

* First pass on incremental export of notebooks/files & SQL objects

* use optional modified_at field

* Add `-incremental` & `-update-since` command-line options

* Incremental generation of `import.sh` and `vars.tf`

* Incremental export of the TF resources themselves

* Added incremental support for the rest of the objects

also updated documentation & compatibility matrix

* Add tests for incremental generation

* Add clarification about periodic full export

* Store last run timestamp in the file on disk and use with `-incremental`

* Address initial review comments

* Incremental export of MLflow Webhooks, expanded tests to cover merge of variables

* fix test

* Fix reflection method marshallJSON for CMK in mws workspace (#2605)

* Fix reflection method marshallJSON for CMK in mws workspace

* Add UT

* Add missing documentation for CMK support on GCP (#2604)

* add CMK on GCP to docs

* feedback

* Add `owner` parameter to `databricks_share` resource (#2594)

* add `owner` parameter to `databricks_share`

* suppress diff

* Exporter: command-line option to control output format for notebooks (#2569)

New command-line option `-notebooksFormat` allows to export notebooks in DBC and IPython formats.

This fixes #2568

* Fix creation of views with comments using `databricks_sql_table` (#2589)

* mark column type as omitempty

* add acc test

* escape names for sql

* add test & suppress_diff

* fix tests

* fix acc test

* update doc

* fix acc tests

* fix doc

* Add account-level API support for Unity Catalog objects (#2182)

* first draft

* account client check

* account client check

* Fixed `databricks_service_principals` data source issue with empty filter (#2185)

* fix `databricks_service_principals` data source issue with empty filter

* fix acc tests

* Allow rotating `token` block in `databricks_mws_workspaces` resource by only changing `comment` field (#2114)

Tested manually for the following cases

Without this PR the provider recreates the entire workspace on a token update
With changes in this PR only the token is refreshed
When both token and storage_configuration_id are changed then the entire workspace is recreated
Additional unit tests also added that allow checks that patch workspace calls are not made when only token is changed

Also added an integration test to check tokens are successfully updated

* Excludes roles in scim API list calls to reduce load on databricks scim service (#2181)

* Exclude roles in scim API list calls

* more test fixes

* Update SDK to v0.6.0 (#2186)

* Update SDK to v0.6.0

* go mod tidy

* update sdk to 0.7.0

* add integration tests

* fix acceptance tests

* fix tests

* add account-level API support for `metastore_data_access`

* add account API support for `databricks_storage_credential`

* address feedback

* refactor to `WorkspaceOrAccountRequest`

* fix acceptance tests

* Release v1.21.0 (#2471)

Release v1.21.0 of the Terraform Provider for Databricks.

 * Added condition_task to the [`databricks_job`](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) resource (private preview) ([#2459](#2459)).
 * Added `AccountData`, `AccountClient` and define generic databricks data utilites for defining workspace and account-level data sources ([#2429](#2429)).
 * Added documentation link to existing Databricks Terraform modules ([#2439](#2439)).
 * Added experimental compute field to [databricks_job](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) resource ([#2401](#2401)).
 * Added import example to doc for [databricks_group_member](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/group_member) resource ([#2453](#2453)).
 * Added support for subscriptions in dashboards & alert SQL tasks in [databricks_job](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) ([#2447](#2447)).
 * Fixed model serving integration test ([#2460](#2460), [#2461](#2461)).
 * Fixed [databricks_job](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) resource file arrival trigger parameter name ([#2438](#2438)).
 * Fixed catalog_workspace_binding_test ([#2463](#2463), [#2451](#2451)).

No breaking changes in this release.

* Install mlflow cluster using  in model serving test if the cluster is already running (#2470)

* Bump golang.org/x/mod from 0.11.0 to 0.12.0 (#2462)

Bumps [golang.org/x/mod](https://github.com/golang/mod) from 0.11.0 to 0.12.0.
- [Commits](golang/mod@v0.11.0...v0.12.0)

---
updated-dependencies:
- dependency-name: golang.org/x/mod
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Exporter: make resource names more unique to avoid duplicate resources errors (#2452)

This includes following changes:

* Add user ID to the `databricks_user` resource name to avoid clashes on names like, `user+1` and `user_1`
* Add user/sp/group ID to the name of the `databricks_group_member` resource
* Remove too aggressive name normalization pattern that also leads to the generation of
  the duplicate resource names for different resources

* Add documentation notes about legacy cluster type & data access (#2437)

* Add documentation notes about legacy cluster type & data access

* Update docs/resources/cluster.md

Co-authored-by: Miles Yucht <miles@databricks.com>

* Update docs/resources/mount.md

Co-authored-by: Miles Yucht <miles@databricks.com>

---------

Co-authored-by: Miles Yucht <miles@databricks.com>

* Use random catalog name in SQL table integration tests (#2473)

The fixed value prevented concurrent integration test runs.

* Link model serving docs to top level README (#2474)

* Add one more item to the troubleshooting guide (#2477)

It's related to use OAuth for authentication but not providing `account_id` in the
provider configuration.

* Added `databricks_access_control_rule_set` resource for managing account-level access (#2371)

* Added `acl_principal_id` attribute to `databricks_user`, `databricks_group` & `databricks_service_principal` for easier use with `databricks_access_control_rule_set` (#2485)

It should simplify specification of principals in the `databricks_access_control_rule_set`
so instead of this (string with placeholders):

```
   grant_rules {
     principals = ["groups/${databricks_group.ds.display_name}"]
     role       = "roles/servicePrincipal.user"
   }
```

it will be simpler to refer like this:

```
   grant_rules {
     principals = [databricks_group.ds.acl_principal_id]
     role       = "roles/servicePrincipal.user"
   }
```

* Added support for Unity Catalog `databricks_metastores` data source  (#2017)

* add documentation for databricks_metastores data source

* add API endpoint for listing metastores

* add metastores data resource

* add test for metastores data source

* add metastores datasource to resource mapping

* fix reference to wrong resource docs

* add a Metastores struct for the response of the API, use this in the datSource

* update terraform specific object attributes

* add new data test

* remove slice_set property from MetastoreData

* use databricks-go-sdk for data_metastore.go

* removed listMetastores endpoint since it's unused

* make sure tests also use the unitycatalog.MetastoreInfo from the sdk

* remove redundant resource

* test -dev

* fix

* fmt

* cleanup

* Added AccountClient to DatabricksClient and AccountData

* upd

* cleanup

* accountLevel

* upd

* add example

* list

* cleanup

* docs

* remove dead code

* wip

* use maps

* upd

* cleanup

* comments

* -

* remove redundant test

---------

Co-authored-by: Tanmay Rustagi <tanmay.rustagi@databricks.com>
Co-authored-by: vuong-nguyen <44292934+nkvuong@users.noreply.github.com>

* Added support for Unity Catalog `databricks_metastore` data source (#2492)

Enable fetching account level metastore information through id for a single metastore.

* Supported new Delve binary name format (#2497)

https://github.com/go-delve/delve/blob/master/CHANGELOG.md#1210-2023-06-23 changes the naming of the delve debug binary. This PR changes isInDebug to accommodate old and new versions of Delve.

* Add code owners for Terraform (#2498)

* Removed unused dlvLoadConfig configuration from settings.json (#2499)

* Fix provider after updating SDK to 0.13 (#2494)

* Fix provider after updating SDK to 0.13

* add unit test

* split test

* Added `control_run_state` flag to the `databricks_job` resource for continuous jobs (#2466)

This PR introduces a new flag, control_run_state, to replace the always_running flag. This flag only applies to continuous jobs. Its behavior is described below:

For jobs with pause_status = PAUSED, it is a no-op on create and stops the active job run on update (if applicable).
For jobs with pause_status = UNPAUSED, it starts a job run on create and stops the active job run on update (if applicable).
The job does not need to be started, as that is handled by the Jobs service itself.

This fixes #2130.

* Added exporter for `databricks_workspace_file` resource (#2493)

* Preliminary changes to make workspace files implementation

- make `NotebooksAPI.List` to return directories as well when called in the recursive
  mode (same as non-recursive behavior)
- Because of that, remove the separate `ListDirectories`
- Extend `workspace.ObjectStatus` with additional fields (will be required for
  incremental notebooks export)
- Cache listing of all workspace objects, and then use it for all operations - list
  notebooks, list directories, list workspace files

* Added exporting of workspace files

---------

Co-authored-by: Miles Yucht <miles@databricks.com>

* Supported boolean values in `databricks_sql_alert` alerts (#2506)

* Added more common issues for troubleshooting (#2486)

* add troubleshooting

* fix doc category

---------

Co-authored-by: Miles Yucht <miles@databricks.com>

* Fixed handling of comments in `databricks_sql_table` resource (#2472)

* column comments and single quote escape

* Delimiter collision avoidance table comment

* compatible with user single quote escape

* unit tests for parseComment

* corrected fmt

---------

Co-authored-by: Miles Yucht <miles@databricks.com>

* Added clarification that `databricks_schema` and `databricks_sql_table` should be imported by their full name, not just by name (#2491)

Co-authored-by: Miles Yucht <miles@databricks.com>

* Updated `databricks_user` with `force = true` to check for error message prefix (#2510)

This fixes #2500

* fix force delete

* remove orphaned code

* fix acceptance tests

* upgrade go sdk

* fix metastoreinfo struct

* docs update

* fix acceptance tests

* fix tests

* updated docs

* fix tests

* rename test

* update tests

* fix tests

* fix test

* add state upgrader

* fix struct

* fix tests

* feedback

* feedback

* fix acc test

* fix test

* fix test

* fix test

* feedback

* fix acc tests

* feedback

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>
Co-authored-by: Serge Smertin <259697+nfx@users.noreply.github.com>
Co-authored-by: Miles Yucht <miles@databricks.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alex Ott <alexey.ott@databricks.com>
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
Co-authored-by: Gautham Sunjay <gauthamsunjay17@gmail.com>
Co-authored-by: guillesd <74136033+guillesd@users.noreply.github.com>
Co-authored-by: Tanmay Rustagi <tanmay.rustagi@databricks.com>
Co-authored-by: Tanmay Rustagi <88379306+tanmay-db@users.noreply.github.com>
Co-authored-by: Fabian Jakobs <fabian.jakobs@databricks.com>
Co-authored-by: klus <lus.karol@gmail.com>

* Fix UC acceptance test (#2613)

* fix acc test

* remove deprecated field from sdk

* Release v1.24.0 (#2614)

* Release v1.24.0

* Release v1.24.0

* Release v1.24.0

* Release v1.24.0

* Fixed verification of workspace reachability by using scim/me which is always available  (#2618)

* add flag to skip verification

* cleanup

* cleanup

* -

* test

* test

* test

* test

* -

* -

* Release v1.24.1 (#2625)

* Release v1.24.1

* go upd

* new line

* Add doc strings for ResourceFixtures (#2633)

* Add doc strings for ResourceFixtures

* fmt

* Update qa/testing.go

Co-authored-by: Miles Yucht <miles@databricks.com>

* Update qa/testing.go

Co-authored-by: Miles Yucht <miles@databricks.com>

---------

Co-authored-by: Miles Yucht <miles@databricks.com>

* Bump github.com/hashicorp/hcl/v2 from 2.17.0 to 2.18.0 (#2636)

Bumps [github.com/hashicorp/hcl/v2](https://github.com/hashicorp/hcl) from 2.17.0 to 2.18.0.
- [Release notes](https://github.com/hashicorp/hcl/releases)
- [Changelog](https://github.com/hashicorp/hcl/blob/main/CHANGELOG.md)
- [Commits](hashicorp/hcl@v2.17.0...v2.18.0)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/hcl/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* terrafmt; updated share and share recipient docs (#2641)

* update documentation (#2644)

* fix response

* remove preview path

* rename test

* fix create call

* add wait for acceptance tests

* fix test

* feedback

* use golang struct

* add wait time for acc test

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Miles Yucht <miles@databricks.com>
Co-authored-by: Krishna Swaroop K <krishna.swaroop@databricks.com>
Co-authored-by: marekbrysa <53767523+marekbrysa@users.noreply.github.com>
Co-authored-by: Alex Ott <alexey.ott@databricks.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: bvdboom <bvdboom@users.noreply.github.com>
Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>
Co-authored-by: Gabor Ratky <gabor.ratky@databricks.com>
Co-authored-by: Rohan Kabra <rohan.kabra@databricks.com>
Co-authored-by: Serge Smertin <259697+nfx@users.noreply.github.com>
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
Co-authored-by: Gautham Sunjay <gauthamsunjay17@gmail.com>
Co-authored-by: guillesd <74136033+guillesd@users.noreply.github.com>
Co-authored-by: Tanmay Rustagi <tanmay.rustagi@databricks.com>
Co-authored-by: Tanmay Rustagi <88379306+tanmay-db@users.noreply.github.com>
Co-authored-by: Fabian Jakobs <fabian.jakobs@databricks.com>
Co-authored-by: klus <lus.karol@gmail.com>
Co-authored-by: Oleh Mykolaishyn <owlleg6@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet