databricks.nvim

Experimental Databricks connectivity for Neovim notebook workflows.

The plugin is intentionally a thin adapter. Databricks session bootstrap, target-specific kernelspec generation, health checks, and Workspace notebook access belong here. Cell editing, navigation, Jupytext conversion, kernel transport, and output rendering remain the responsibility of the user's notebook configuration and Molten.

Current Slice

The Python package exposes:

from databricks_nvim import install

install()

install() creates a Databricks Connect session from the kernelspec environment and injects spark, sql, udf, dbutils, display, displayHTML, and bounded display_spark helpers into the active IPython namespace. The package includes ipykernel because it is installed into the target-specific kernel environment.

Development

The development environment currently targets DBR 16.4 and therefore requires Python 3.12 and Neovim 0.10 or newer:

uv sync --dev
uv run pytest

Databricks Connect must remain pinned to the target DBR major and minor version. The kernelspec generator records that target and the health check enforces the relationship. Version 0.1 accepts DBR 16.4 or newer releases within the DBR 16.x line.

Authentication

Create or refresh a local Databricks CLI profile before generating or using a kernel:

databricks auth login --profile PROFILE
databricks current-user me --profile PROFILE

The CLI normally opens a browser for OAuth authentication. databricks.nvim stores only the selected profile name and compute ID in the kernelspec. It delegates credential lookup and refresh to Databricks unified authentication and never copies tokens into Neovim configuration, command arguments, or notebook buffers.

Use placeholders in documentation, fixtures, screenshots, and bug reports. Profile names, workspace URLs, compute IDs, and compute names are not credentials, but they may reveal workplace information.

Kernelspec Generation

Create one managed kernel per Databricks profile and compute target:

uv run databricks-nvim-kernelspec \
  --profile PROFILE \
  --compute-id CLUSTER_ID \
  --compute-name "FRIENDLY NAME" \
  --runtime-version MAJOR.MINOR \
  --python /absolute/path/to/project/.venv/bin/python

The generator writes a Jupyter kernelspec plus an isolated IPYTHONDIR startup hook. Kernel names are stable across display-name changes because they derive from the profile and compute ID. Existing kernels are replaced only when their metadata identifies them as managed by databricks.nvim.

Target Picker

Run :DatabricksTarget to select an installed managed target. The picker shows the friendly compute name, profile, and DBR version. It uses Telescope when available and otherwise falls back to vim.ui.select.

If the selected kernel is already running in another buffer, the plugin attaches it with Molten's shared-kernel mode. If the current buffer already has a different Molten kernel, the plugin refuses to replace it implicitly; run :MoltenDeinit first when you intend to terminate that kernel and switch targets.

The target command is discovered from PATH or the repository development environment. It can also be configured explicitly:

vim.g.databricks_nvim_targets_command = {
  "/absolute/path/to/project/.venv/bin/python",
  "-m",
  "databricks_nvim.targets",
}

vim.keymap.set("n", "<leader>dk", "<cmd>DatabricksTarget<cr>", {
  desc = "Select Databricks target",
})

Health Check

Make the Python health command available to Neovim. When the target environment is not on PATH, point the plugin at its module explicitly:

vim.g.databricks_nvim_health_command = {
  "/absolute/path/to/project/.venv/bin/python",
  "-m",
  "databricks_nvim.health",
}

-- Optional fail-safe; defaults to five minutes for multiple kernels.
vim.g.databricks_nvim_health_timeout_ms = 300000

Then run :checkhealth databricks.

The check discovers managed kernelspecs and validates:

• Databricks CLI availability and profile authentication
• Compute existence, state, and compatibility with the supported DBR runtime range
• Kernelspec JSON, target metadata, Python executable, IPYTHONDIR, and startup hook
• Bootstrap importability and the databricks-connect/DBR version match
• UDF-specific Python minor compatibility
• Absence of a conflicting standalone pyspark distribution

Health output never includes tokens, authorization headers, command stderr, environment dumps, or Databricks configuration contents. The same structured report is available outside Neovim:

uv run databricks-nvim-health
uv run databricks-nvim-health --json

Manual, target-independent regression notebooks live under tests/manual/. Private workspace paths, compute identifiers, exploratory datasets, and executed outputs belong under ignored scratch/ storage.