New Container: DatabricksContainer for Databricks SQL Connector

[adendek]

What is the new container you'd like to have?

I propose the addition of a DatabricksContainer module to facilitate testing applications that use the databricks-sql-connector. This would allow developers to run integration and end-to-end tests against a mock Databricks SQL API in an isolated and reproducible manner.

Unlike many services, Databricks does not provide an official local emulator in a Docker container. Therefore, this module would be designed to work with a user-provided Docker image that runs a mock server emulating the Databricks SQL API.

The benefits of having this dedicated container module include:

• Isolated Testing: Enables hermetic tests without relying on shared, live Databricks workspaces.
• CI/CD Integration: Simplifies running automated tests in CI/CD pipelines without complex credential management or network configurations.
• Developer Experience: Provides a simple, Pythonic interface consistent with other testcontainers modules like AzuriteContainer or PostgresContainer.
• Reliability: Eliminates test flakiness caused by network issues or changes in shared development environments.

Why not just use a generic container for this?

While it is possible to use a generic DockerContainer("my-databricks-mock:latest"), a dedicated DatabricksContainer module would abstract away significant complexity related to configuration and readiness checks.

1. Complicated Setup and Configuration:

The databricks-sql-connector requires specific connection parameters: server_hostname, http_path, and access_token. A user of DockerContainer would have to manually:

• Get the container's dynamic IP address and port.
• Correctly format the server_hostname and http_path.
• Know which token the mock server expects.
• This process is cumbersome and error-prone.

Generic DockerContainer approach :

from databricks import sql
from testcontainers.core.container import DockerContainer

with DockerContainer("my-databricks-mock:latest").with_exposed_ports(8080) as mock_container:
    host = mock_container.get_container_host_ip()
    port = mock_container.get_exposed_port(8080)
    
    # User must manually construct connection parameters
    connection = sql.connect(
        server_hostname=host,
        http_path=f"/sql/1.0/warehouses/{port}", # Path might be complex and mock-specific
        access_token="dummy-token"
    )

A dedicated DatabricksContainer would provide helper methods to abstract this away, offering a much cleaner interface.

Proposed DatabricksContainer approach:

from databricks import sql
# from testcontainers.databricks import DatabricksContainer

with DatabricksContainer as databricks_container:
    # Clean, abstracted methods
    connection = sql.connect(
        server_hostname=databricks_container.get_server_hostname(),
        http_path=databricks_container.get_http_path(),
        access_token=databricks_container.get_token()
    )

[WPrendota]

Hello, I’d like to work on this issue if nobody else is currently working on it.

[alexanderankin]

I'm not sure this would get accepted as a python module since it doesnt have a maintained image to use with it

[adendek]

Hello @alexanderankin,

Thank you for your comment. That's a very valid point, and I agree that an officially maintained image is the ideal standard for a testcontainers module.

However, given that Databricks does not currently provide one, developers are left with a significant gap in their testing capabilities. My proposal is to create a DatabricksContainer module designed to work with a lightweight mock server that emulates the Databricks SQL API. Even if it only supports basic SELECT and INSERT operations initially, it would provide immense value.

---

To justify the need, here is a simplified example of an E2E test from my current project. This test needs to verify a full pipeline that reads from Databricks, processes data, and writes a report back to Databricks, all within an isolated CI environment:

# Simplified E2E Test Example
def test_e2e_pipeline_with_real_integrations():
    """
    Tests the full data pipeline, verifying integration with both
    Azure Storage and the Databricks SQL API.
    """
    with AzuriteContainer() as azurite, DatabricksContainer() as databricks:
        # Configure the application to use the test containers
        env_vars = {
            "AZURE_CONNECTION_STRING": azurite.get_connection_string(),
            "DBX_HOST": databricks.get_server_hostname(),
            "DBX_HTTP_PATH": databricks.get_http_path(),
            "DBX_TOKEN": databricks.get_token(),
        }
        with patch.dict(os.environ, env_vars):
            # 1. Trigger the application's main pipeline
            response = test_client.post("/pipelines/model_train", ...)
            assert response.status_code == 200

            # 2. Verify the final report was written correctly to the mock Databricks
            report = read_report_from_databricks()
            assert "success" in report

Without a DatabricksContainer, this kind of critical E2E test is nearly impossible to automate reliably.

I plan to implement this for my project regardless. By contributing it, we could provide an immediate solution for the many developers facing this issue. Furthermore, a successful community module could serve as a strong starting point to advocate for an official image from Databricks. I am willing to help initiate that discussion through my company's channels, though I expect it would take time.

I believe this pragmatic approach would be a valuable addition to the library.