Intel Cloud Optimization Modules for Terraform

© Copyright 2022, Intel Corporation

AWS Databricks

The module can deploy an Intel Optimized AWS Databricks Workspace.

Learn more about optimizations :

Performance Data

Faster insights With Databricks Photon Using AWS i4i Instances With the Latest Intel Ice Lake Scalable Processors

5.3x relative speed up of i4i Photon against the i3 DBR

Accelerating Azure Databricks Runtime for Machine Learning

Usage

See examples folder for code ./examples/databricks-workspace/main.tf

All the examples in example folder shows how to create a databricks workspace using this module. Additionally, some of the examples display how to create a databricks cluster with the workspace using this module.

Usage Considerations

• Prerequisites:

  1. Create a databricks account.. Remember the email and password used to create an Databricks account.

  2. After logging in the account, in the top right corner you can find your Databricks Account ID

  3. Follow the steps here to create VPC with subnets and security groups in your AWS Console.

  4. Create a terraform.tfvars file and fill in the details.

dbx_account_id       = <""> 
dbx_account_password = <"">
dbx_account_username = <"">
vpc_id               = <"">
vpc_subnet_ids       = <["subnet-XXXX", "subnet-XXXXX"]>
security_group_ids   = <["sg-XXXX"]>

Run Terraform

terraform init  
terraform plan
terraform apply 

Considerations

More Information regarding deploying Databricks Workspace Databricks

Requirements

Name	Version
aws	~> 4.15.0
databricks	~> 1.14.2
random	~> 3.4.3
time	~> 0.9.1

Providers

Name	Version
aws	~> 4.15.0
databricks	~> 1.14.2
databricks.workspace	~> 1.14.2
random	~> 3.4.3
time	~> 0.9.1

Modules

No modules.

Resources

Name	Type
aws_iam_policy.iap	resource
aws_iam_policy_attachment.ipa	resource
aws_iam_role.ir	resource
aws_s3_bucket.rsb	resource
aws_s3_bucket_policy.root_bucket_policy	resource
aws_s3_bucket_versioning.sbv	resource
databricks_global_init_script.intel_optimized_script	resource
databricks_mws_credentials.cr	resource
databricks_mws_networks.nw	resource
databricks_mws_storage_configurations.sc	resource
databricks_mws_workspaces.ws	resource
random_string.naming	resource
time_sleep.wait	resource
databricks_aws_assume_role_policy.rp	data source
databricks_aws_bucket_policy.bp	data source
databricks_aws_crossaccount_policy.cap	data source

Inputs

Name	Description	Type	Default	Required
aws_cross_account_arn	ARN that will be used for databricks cross account IAM role.	string	""	no
aws_cross_account_role_name	Flag that determines if a cross account role will be created in the AWS account provided.	string	"dbx_module_account_role"	no
aws_iam_policy_attachment_name	Name that will be used when attaching the policy to the arn that is created or provided	string	"dbx_module_attatchment_policy"	no
aws_iam_policy_name	Name of the IAM policy that will be created and associated with the aws_account_role_name.	string	"dbx_module_account_role"	no
bucket_name	Name of the existing S3 bucket that Databricks will consume.	string	"dbx-root-bucket"	no
create_aws_account_role	Flag that determines if a cross account role will be created in the AWS account provided.	bool	true	no
create_bucket	Boolean that when true will create a root S3 bucket for Databricks to consume.	bool	true	no
dbx_account_id	Account ID Number for the Databricks Account	string	n/a	yes
dbx_account_password	Account Login Password for the Databricks Account	string	n/a	yes
dbx_account_username	Account Login Username/Email for the Databricks Account	string	n/a	yes
dbx_credential_name	Name that will be associated with the credential configuration in Databricks.	string	"dbx_module_credential"	no
dbx_network_name	Name that will be associated with the network configuration in Databricks.	string	"dbx_module_network"	no
dbx_storage_name	Name that will be associated with the storage configuration that will be created in Databricks.	string	"dbx_module_storage"	no
dbx_workspace_name	Name that will be associated with the workspace that will be created in Databricks.	string	"dbx_module_workspace"	no
enable_intel_tags	If true adds additional Intel tags to resources	bool	true	no
intel_tags	Intel Tags	map(string)	{ "intel-module": "terraform-intel-aws-databricks-workspace", "intel-registry": "https://registry.terraform.io/namespaces/intel" }	no
prefix	Prefix that will be added to all resources that are created	string	null	no
region	AWS Region that will be used as a part of the deployment	string	"us-east-2"	no
s3_bucket_acl	ACL that will be attached to the S3 bucket (if created) during the provisioning process.	string	"private"	no
s3_bucket_force_destroy	Flag that determines if a bucket will destroy all contents when a Terraform destroy takes place.	bool	true	no
s3_bucket_versioning	Flag that enables/disables versioning of the S3 root storage bucket	string	"Disabled"	no
security_group_ids	List of security group IDs that will be utilized by Databricks.	set(string)	n/a	yes
tags	A map of key / values to apply to the resources that will be created.	map(string)	{}	no
vpc_id	ID for the VPC that Databricks will be attaching to.	string	n/a	yes
vpc_subnet_ids	List of subnet IDs that will be utilized by Databricks.	set(string)	n/a	yes

Outputs

Name	Description
dbx_account_id	Account ID for the Databricks Account
dbx_bucket_name	Name of the existing S3 bucket that Databricks will consume.
dbx_create_bucket	Flag to create AWS S3 bucket or not.
dbx_create_role	Flag to create AWS IAM Role or not.
dbx_credentials_name	Name that will be associated with the credential configuration in Databricks.
dbx_host	URL of the Databricks workspace
dbx_id	ID of the Databricks workspace
dbx_network_name	Name that will be associated with the network configuration in Databricks.
dbx_role_arn	ARN that will be used for databricks cross account IAM role.
dbx_security_group_ids	List of security group IDs that will be utilized by Databricks.
dbx_storage_configuration_name	Name of the existing S3 bucket that Databricks will consume.
dbx_vpc_id	ID for the VPC that Databricks will be attaching to.
dbx_vpc_subnet_ids	List of subnet IDs that will be utilized by Databricks.