S3 Mount via cli or API

[morkot]

Hey, guys!

We need to have a programmatic (described in code) configuration for s3 bucket mounts in clusters.

Are there any plans for API/CLI that will allow mounting S3 buckets? And what is the possible workaround? How to call dbutils.fs.mount externally?

[mireille-raad]

I was looking for the same thing and found this https://docs.databricks.com/spark/latest/data-sources/aws/amazon-s3.html#mount-an-s3-bucket

ACCESS_KEY = "<aws-access-key>"
SECRET_KEY = "<aws-secret-key>"
ENCODED_SECRET_KEY = SECRET_KEY.replace("/", "%2F")
AWS_BUCKET_NAME = "<aws-bucket-name>"
MOUNT_NAME = "<mount-name>"

dbutils.fs.mount("s3a://%s:%s@%s" % (ACCESS_KEY, ENCODED_SECRET_KEY, AWS_BUCKET_NAME), "/mnt/%s" % MOUNT_NAME)
display(dbutils.fs.ls("/mnt/%s" % MOUNT_NAME))

@morkot
fixes your problem?

[morkot]

@DC-coder this is what we do on our clusters. Also, we have automated parameterized job to do set of mounts automatically. However, we aren't quite happy with this approach. What I was looking for is API/CLI call outside of cluster to make mounts. E.g. running a command like that on my laptop:

databricks fs mount my-s3-bucket-name/bucket-prefix/ /my/cluster/mount/folder

[mireille-raad]

ahhh... I hope this feature request makes it. It is really useful.

[colemanja91]

I also would be happy to see this - would make managing DBFS configs via Ansible super easy

[dougbateman]

If you have a cluster already up and running, this could be done by executing the script via the databricks-cli's job's API.

[nfx]

Currently, there's no API for this. please use databricks_mount from terraform integration - https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/mount - which is the only supported way of doing mounts from CI/CD tooling for the foreseeable future.