- Added configurable provisioning timeout for
databricks_mws_workspaces, so that local DNS cache issues would be more tolerated. - Added [databricks_current_user] to simplify applying the same Terraform configuration by different users in the shared workspace for testing purposes.
- Added client-side rate limiting to release the pressure on backend APIs and prevent client blocking (#465)
- Member usernames, group names and instance profile names in
databricks_groupdata source are now sorted and providing consistent behavior between runs (#449) - Fixed redundant multiple mounting clusters (#445)
- Added optional parameter azure_environment to provider config which defaults to public (#437).
- Added databricks_service_principal resource.
skip_validationfromdatabricks_instance_profilewas removed and is always set totrue.- Added propagation of terraform version to
User-Agentheader, along with type of resource used. databricks_notebook&databricks_dbfs_filegot newsourcefield to specify location of a local file.databricks_notebookcan havelanguagefield optional, as long assourceis set to a file with.py,.scala,.sql, or.rextension.databricks_medata source was added to representuser_name,home&idof the caller user (or service principal).- Added validation for secret scope name in
databricks_secret,databricks_secret_scopeanddatabricks_secret_acl. Non-compliant names may cause errors. - Added databricks_spark_version data source.
- Fixed support for single node clusters support by allowing
num_workersto be0. - Fixed bug in destruction of IP access lists (#426).
- All resource imports are now making call to corresponding Databricks API by default (#471).
Behavior changes
- Removed deprecated
library_jar,library_egg,library_whl,library_pypi,library_cran, andlibrary_mavenfromdatabricks_clusteranddatabricks_jobin favor of more API-transparent library configuration block. - Removed deprecated
notebook_pathandnotebook_base_parametersfromdatabricks_jobin favor of notebook_task configuration block. - Removed deprecated
jar_uri,jar_main_class_name, andjar_parametersfromdatabricks_jobin favor of spark_jar_task configuration block. - Removed deprecated
python_fileandpython_parametersfromdatabricks_jobin favor of spark_python_task configuration block. - Removed deprecated
spark_submit_parametersfromdatabricks_jobin favor of spark_submit_task configuration block. - Removed deprecated
databricks_scim_userresource in favor of databricks_user. - Removed deprecated
databricks_scim_groupresource in favor of databricks_group. - Removed deprecated
databricks_default_user_rolesdata source in favor of databricks_group data source. - Removed deprecated
basic_authandazure_authprovider configuration blocks in favor of documented authentication methods. format,overwrite, andmkdirswere removed fromdatabricks_notebook. To follow expected behavior of Terraform, notebooks are always overwritten.skip_validationfromdatabricks_instance_profilewas removed and is always set totruefor subsequent requests.databricks_mws_workspacegotverify_workspace_runnningremoved and now validates all every deployment. In case deployment failed, it removes workspace that failed and returns error message with explanation.default_tagswere removed fromdatabricks_instance_pool.disk_specgot new attributedisk_type, that containsazure_disk_volume_typeandebs_volume_type. This change is made to closer reflect API structure.databricks_notebook&databricks_dbfs_filegotcontentattribute renamed tocontent_base64and now share the same logic to work with local files.
- Fixed documentation issues.
- Added missing resource importers and test to cover it.
- Migrated build from TravisCI to GitHub Actions.
- Fixed custom
config_fileissue configuration handling (#420).
Deprecations
databricks_notebookhas gotoverwrite,mkdirsandformatparameters, that always have to be set to certain values in order to follow expected behavior of terraform. These fields would be removed in 0.3 and always set to proper values.databricks_notebook&databricks_dbfs_filefieldcontentis deprecated and would be renamed tocontent_base64to further increase clarity.databricks_dbfs_filehas gotcontent,content_b64_md5,overwrite,mkdirs,validate_remote_filefields deprecated and they will be removed in the next version, where critical code path will be shared withdatabricks_notebook.network_error_messagesandverify_workspace_runnningfromdatabricks_mws_workspacesis deprecated and going to be removed in 0.3.error_messagesfromdatabricks_mws_networksare deprecated and would be removed in 0.3.ebs_volume_typeandazure_disk_volume_typefromdatabricks_instance_poolis going to be moved todisk_typesub-block in 0.3, which means you'll slightly have to modify configuration while migrating to 0.3. Computed fielddefault_tagsis going to be removed from resource. This is done to further increase maintainability of provider in the future.
Updated dependency versions:
- github.com/aws/aws-sdk-go 35.36
- github.com/hashicorp/go-retryablehttp 0.6.8
- github.com/Azure/go-autorest/autorest 0.11.12
Behavior changes
min_idle_instancesfordatabricks_instance_poolis now optional.skip_validationfordatabricks_instance_profileis going to be removed in 0.3.
- Added databricks_workspace_conf resource.
- Added databricks_mws_log_delivery resource for billable usage & audit logs consumption.
- Added databricks_node_type data source for simpler selection of node types across AWS & Azure.
- Added Azure Key Vault support for
databricks_secret_scopefor Azure CLI authenticated users. - Added is_pinned support for
databricks_clusterresource. - Fixed restarting cluster on changes in cluster configuration aren't related to the cluster configuration (issue #379)
- Fixed issue #383 by cleaning up clusters that fail to start.
- Fixed issue #382 by ignoring any incoming changes to deployment name of
databricks_mws_workspaces, as well as propagating the right error messages. - Internal: API for retrieval of the cluster events.
- Increased code coverage to 71%.
Updated dependency versions:
- github.com/Azure/go-autorest/autorest v0.11.9
- github.com/Azure/go-autorest/autorest/adal v0.9.5
- github.com/Azure/go-autorest/autorest/azure/auth v0.5.3
- github.com/Azure/go-autorest/autorest/azure/cli v0.4.2
- gopkg.in/ini.v1 1.62.0
Deprecations
network_error_messagesfromdatabricks_mws_workspacesis deprecated and going to be removed in 0.3
- Small fixes
- Added support for customer managed keys for Accounts API.
- Added
databricks_userresource. - Added
databricks_user_instance_profileresource. - Added
databricks_groupdata source.
Updated dependency versions:
- github.com/Azure/go-autorest/autorest v0.11.6
- github.com/Azure/go-autorest/autorest/adal v0.9.4
- github.com/Azure/go-autorest/autorest/azure/auth v0.5.2
- github.com/Azure/go-autorest/autorest/azure/cli v0.4.1
- gopkg.in/ini.v1 v1.61.0
Deprecations
databricks_scim_useris no longer receiving fixes and will be removed in0.3, please rewrite using thedatabricks_userresource, which has more consistent semantics withdatabricks_groupand works better with identity provider SCIM sync.databricks_scim_groupis no longer receiving fixes and will be removed in0.3. Please rewrite using thedatabricks_groupresource.databricks_default_user_rolesis no longer receiving fixes and will be removed in0.3, please rewrite usingdatabricks_user&databricks_groupresources.
Behavior changes
- State changes to legacy
spark.databricks.delta.preview.enabledconfig option are now ignored bydatabricks_job&databricks_cluster - Libraries, which are installed on all clusters and are not part of cluster resource definition, won't be waited for INSTALLED status
- Fixed "Secret scope ACL is MANAGE for all users by default" (issue 322). If you were relying on setting
MANAGEpermission to all users by default, you need to addinitial_manage_principal = "users"to yourresource "databricks_secret_scope"declaration.
- Added support for local disk encryption
- Added more reliable indication about Azure environment and fixed azure auth issue for Terraform 0.13
- Updated databricks_aws_crossaccount_policy to latest rules
- Fixed missing importers for databricks_scim_* resources
- Updated Terraform Plugin SDK to latest version along with transitive dependencies.
- Added support disclaimers
- Increased code coverage to 65%
- Added Azure CLI authentication to bridge the gap of local development workflows and let more people use the provider.
- All authentication is completely lazy-initialized, which makes it provider overall more stable.
- Significantly increased unit test coverage, which runs before every merge of a pull request.
- Introduced constantly running integration test suite environments: azsp, azcli & awsmt
- Numerous stability improvements for clusters, mounts, libraries, notebooks, files, authentication and TLS connectivity.
- Added ability to mount storage without explicitly defining a cluster, though it will still launch auto-terminating
terraform-mountcluster to perform the mount. databricks_cluster&databricks_jobnow share significant portion of configuration wiring code, therefore increasing the stability of codebase.- Added support for Terraform 0.13 local builds for those who develop or cannot wait for next release.
- Added AWS IAM Policy data helpers to simplify new deployments.
- Migrated all documentation to Terraform Registry format, therefore having a single always-accurate place for end-user guides.
- Internally, codebase has been split into multiple packages, which should make further contributions simpler.
Updated dependency versions:
- github.com/Azure/go-autorest/autorest v0.11.4
- github.com/Azure/go-autorest/autorest/adal v0.9.2
- github.com/Azure/go-autorest/autorest/azure/auth v0.5.1
- github.com/aws/aws-sdk-go v1.34.13
- gopkg.in/ini.v1 v1.60.2
Deprecations
library_*is no longer receiving fixes and will be removed in0.3, please rewrite cluster & job resources to uselibraryconfiguration block.basic_authprovider block is no longer receiving fixesand will be removed in0.3, please useusernameandpasswordoptionsazure_authprovider block is no longer receiving fixesand will be removed in0.3, please useazure_*options
Behavior changes
- Previously, mounts code paths were different functions. This release unifies them to be a single testable codebase with different configuration options & re-use of the critical code paths. For maintainability reasons, there's no longer check performed on container & storage account names, but rather on high-level mount source uri.