DATABRICKS_CONFIG_FILE isn't used by many commands

[rebot]

Hi,

I'm storing my .databrickscfg file within my repo (FYI; it's ignored in .git) and like to run databricks commands against it, choosing the right profile for the job. However, a simple DATABRICKS_CONFIG_FILE=./.databrickscfg databricks users list --profile staging doesn't work. I removed the ~/.databrickscfg out of interest and to me, it seems like it's still trying to look into the default file location.

I'm running:

Databricks CLI v0.208.0

Thank you

[andrewnester]

Hi @rebot ! Could you please try to run the same command with --log-level=DEBUG flag set and post the output here? Thank you!

[rebot]

@andrewnester This is the result after adding --log-level=DEBUG

~/Development/Python/ControlMicroservice feature/#27205-build-databricks-on-main-and-feature-branch*
auge-control-framework-yg4VaEo2-py3.10 ❯ DATABRICKS_CONFIG_FILE=./.databrickscfg databricks users list --profile user-uat --log-level=DEBUG
13:30  INFO start version=0.208.0 args=databricks, users, list, --profile, user-uat, --log-level=DEBUG
13:30 DEBUG Loading bundle configuration from: /Users/gillestrenson/Development/Python/ControlMicroservice/databricks.yml
time=2023-10-19T13:30:54.011+02:00 level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq
time=2023-10-19T13:30:54.011+02:00 level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=scripts.preinit
time=2023-10-19T13:30:54.011+02:00 level=DEBUG source="scripts.go 37}" msg="No script defined for preinit, skipping" mutator=seq mutator=scripts.preinit
time=2023-10-19T13:30:54.011+02:00 level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=ProcessRootIncludes
time=2023-10-19T13:30:54.011+02:00 level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=ProcessRootIncludes mutator=seq
time=2023-10-19T13:30:54.011+02:00 level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(databricks/bundle.resources.yml)
time=2023-10-19T13:30:54.012+02:00 level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(databricks/bundle.targets.yml)
time=2023-10-19T13:30:54.013+02:00 level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=DefineDefaultTarget(default)
time=2023-10-19T13:30:54.013+02:00 level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=LoadGitDetails
time=2023-10-19T13:30:54.013+02:00 level=DEBUG source="mutator.go 22}" msg=Apply mutator=SelectDefaultTarget
panic: cannot parse config file: open /Users/gillestrenson/.databrickscfg: no such file or directory

goroutine 1 [running]:
github.com/databricks/cli/bundle.(*Bundle).WorkspaceClient.func1()
	github.com/databricks/cli/bundle/bundle.go:119 +0x7c
sync.(*Once).doSlow(0x1d8?, 0x140004abfb0?)
	sync/once.go:74 +0x100
sync.(*Once).Do(...)
	sync/once.go:65
github.com/databricks/cli/bundle.(*Bundle).WorkspaceClient(0x140003b82c0)
	github.com/databricks/cli/bundle/bundle.go:115 +0x50
github.com/databricks/cli/cmd/root.MustWorkspaceClient(0x1400049e000, {0x140004abe60, 0x0, 0x3})
	github.com/databricks/cli/cmd/root/auth.go:174 +0xe8
github.com/spf13/cobra.(*Command).execute(0x1400049e000, {0x140004abe00, 0x3, 0x3})
	github.com/spf13/cobra@v1.7.0/command.go:925 +0x600
github.com/spf13/cobra.(*Command).ExecuteC(0x140003d0000)
	github.com/spf13/cobra@v1.7.0/command.go:1068 +0x320
github.com/spf13/cobra.(*Command).ExecuteContextC(...)
	github.com/spf13/cobra@v1.7.0/command.go:1001
github.com/databricks/cli/cmd/root.Execute(0x1016716e0?)
	github.com/databricks/cli/cmd/root/root.go:99 +0x58
main.main()
	github.com/databricks/cli/main.go:11 +0x30

[rebot]

@andrewnester , in this output, I saw it also tried to load some bundle configuration. However, I didn't invoke the databricks bundle command here. So what I did to avoid this, was changing my directory to the parent directory and I tried it again. Now it worked! So because their was a databricks.yml file next to it, it failed.. My databricks.yml is pretty standard.

~/Development/Python
❯ DATABRICKS_CONFIG_FILE=ControlMicroservice/.databrickscfg databricks users list --profile user-uat --log-level=DEBUG
13:32  INFO start version=0.208.0 args=databricks, users, list, --profile, user-uat, --log-level=DEBUG
13:32 DEBUG Loading user-uat profile from ControlMicroservice/.databrickscfg sdk=true
13:32 DEBUG GET /api/2.0/preview/scim/v2/Users?attributes=id,userName,groups,active
< HTTP/2.0 200 OK
< {
<   "Resources": [
<     {
<       "active": true,

databricks.yml:

# yaml-language-server: $schema=databricks.schema.json

bundle: 
  name: auge-control-databricks

variables:
  job_prefix: 
    description: A unifying prefix for this bundle's job and task names
    default: CICD
  runtime:
    description: The clusters Databricks Runtime
    default: 11.3.x-scala2.12
  worker_type:
    description: The Databricks Worker Type
    default: Standard_D16s_v3
  docker_image:
    description: The Docker image used for the compute

include: 
  - databricks/bundle*.yml

sync:
  include:
    - .config # We'll sync the config file that is being populated during the release process
    - databricks/notebooks/*.py
    - databricks/notebooks/*.ipynb

[pietern]

@rebot Thanks for providing the related information and logs.

Our intent is for any CLI command that is executed in the bundle context (working directory equal to the bundle root, or any nested path) to pick up the authentication information from the bundle. That way you generally don't need to specify a profile or other authentication options when they are already implied.

That said, the bug is legit. What happens is that we don't pick up the configuration env vars when validating that the specified (or inferred) profile is compatible with the host specified in the bundle configuration.

We'll work on a fix and post back here when it is fixed.