Databricks Connector Fails

[P2W2]

Describe the bug
Hello,
I am in the process of trying out cube.js. I wanted to use the Databricks JDBC connector with Docker.
I tried to follow this Documentation.
The Container Starts up but if I try to open the schema (http://localhost:4000/#/schema) I got following failure:

Error: ENOENT: no such file or directory, open 'package.json'
    at DatabricksDriver.testConnection (/cube/node_modules/@cubejs-backend/jdbc-driver/src/JDBCDriver.ts:188:13)
    at CubejsServerCore.getDriver (/cube/node_modules/@cubejs-backend/server-core/src/core/server.ts:782:7)
    at /cube/node_modules/@cubejs-backend/server-core/src/core/DevServer.ts:108:22
    at /cube/node_modules/@cubejs-backend/server-core/src/core/DevServer.ts:78:9

To Reproduce

1. Follow the Documentation to build the Docker Image
2. docker run -p 4000:4000 -p 3000:3000 --env-file=.env -v ${PWD}:/cube/conf -e CUBEJS_DEV_MODE=true cubejs-jdk:latest

Warning. There is no cube.js file. Continue with environment variables
🔥 Cube Store (0.31.64) is assigned to 3030 port.
Warning. Option apiSecret is required in dev mode. Cube.js has generated it as 5bc38414df47015c0432991a0b8f2cbe
🔓 Authentication checks are disabled in developer mode. Please use NODE_ENV=production to enable it.
🦅 Dev environment available at http://localhost:4000
🚀 Cube API server (0.31.64) is listening on 4000
2023-02-23 11:26:41,803 INFO  [cubestored] <pid:29> Cube Store version 0.31.64
2023-02-23 11:26:41,890 INFO  [cubestore::http::status] <pid:29> Serving status probes at 0.0.0.0:3031
2023-02-23 11:26:41,902 INFO  [cubestore::metastore::rocks_fs] <pid:29> Using existing metastore in /cube/conf/.cubestore/data/metastore
2023-02-23 11:26:42,315 INFO  [cubestore::mysql] <pid:29> MySQL port open on 0.0.0.0:13306
2023-02-23 11:26:42,321 INFO  [cubestore::http] <pid:29> Http Server is listening on 0.0.0.0:3030
You accepted Terms & Conditions for JDBC driver from DataBricks by CUBEJS_DB_DATABRICKS_ACCEPT_POLICY
Downloading DatabricksJDBC42-2.6.29.1051
Release notes: https://databricks-bi-artifacts.s3.us-east-2.amazonaws.com/simbaspark-drivers/jdbc/2.6.29/docs/release-notes.txt

3. Open the Playground schema page on localhost:4000

.env file

CUBEJS_DB_TYPE=databricks-jdbc
# You can find this inside the cluster's configuration
CUBEJS_DB_DATABRICKS_URL=jdbc:spark://***.databricks.com:443/default;transportMode=http;ssl=1;httpPath=sql/protocolv1/o/*******/****-******-******;AuthMech=3;UID=token
# You can specify the personal access token separately from `CUBEJS_DB_DATABRICKS_URL` by doing this:
CUBEJS_DB_DATABRICKS_TOKEN=************************
# This accepts the Databricks usage policy and must be set to `true` to use the Databricks JDBC driver
CUBEJS_DB_DATABRICKS_ACCEPT_POLICY=true

Additional context
I use the dev mode otherwise I get following error message:

Error: apiSecret is required option(s)
    at OptsHandler.initializeCoreOptions (/cube/node_modules/@cubejs-backend/server-core/src/core/OptsHandler.ts:533:15)
    at new OptsHandler (/cube/node_modules/@cubejs-backend/server-core/src/core/OptsHandler.ts:49:36)
    at new CubejsServerCore (/cube/node_modules/@cubejs-backend/server-core/src/core/server.ts:170:24)
    at CubejsServer.createCoreInstance (/cube/node_modules/@cubejs-backend/server/src/server.ts:84:12)
    at new CubejsServer (/cube/node_modules/@cubejs-backend/server/src/server.ts:79:22)
    at ServerContainer.createServer (/cube/node_modules/@cubejs-backend/server/src/server/container.ts:254:12)
    at ServerContainer.runServerInstance (/cube/node_modules/@cubejs-backend/server/src/server/container.ts:235:25)
    at makeInstance (/cube/node_modules/@cubejs-backend/server/src/server/container.ts:347:33)
    at ServerContainer.start (/cube/node_modules/@cubejs-backend/server/src/server/container.ts:360:20)
    at Server.run (/cube/node_modules/@cubejs-backend/server/src/command/server.ts:25:5)
    at Command.<anonymous> (/cube/node_modules/cubejs-cli/src/command/proxy-command.ts:51:13)

Need some help? -------------------------------------

Thanks for the help.

[jelstongreen]

+1

[caiobelfort]

Same problem here!

[carlagouveia]

Having the same issue. Could this be a problem with the base image?

cube_example  | You accepted Terms & Conditions for JDBC driver from DataBricks by CUBEJS_DB_DATABRICKS_ACCEPT_POLICY
cube_example  | Downloading DatabricksJDBC42-2.6.29.1051
cube_example  | Release notes: https://databricks-bi-artifacts.s3.us-east-2.amazonaws.com/simbaspark-drivers/jdbc/2.6.29/docs/release-notes.txt
cube_example  | Error: ENOENT: no such file or directory, open 'package.json'
cube_example  |     at DatabricksDriver.testConnection (/cube/node_modules/@cubejs-backend/jdbc-driver/src/JDBCDriver.ts:205:13)
cube_example  |     at CubejsServerCore.getDriver (/cube/node_modules/@cubejs-backend/server-core/src/core/server.ts:792:7)
cube_example  |     at /cube/node_modules/@cubejs-backend/server-core/src/core/DevServer.ts:108:22
cube_example  |     at /cube/node_modules/@cubejs-backend/server-core/src/core/DevServer.ts:78:9

@paveltiunov any other place where we should file this as a regression?

[carlagouveia]

The issue seems to be related to this line: https://github.com/cube-js/cube/blob/40195c97d5b0d2caf89b04ed4f7767a1ff767f01/packages/cubejs-databricks-jdbc-driver/src/DatabricksDriver.ts#LL30C14-L30C14

Which is used here:

cube/packages/cubejs-databricks-jdbc-driver/src/DatabricksDriver.ts

Line 213 in 40195c9

UserAgentEntry: `CubeDev+Cube/${version} (Databricks)`,

I made a PR to fix this issue. In my case there was an exception while connecting to Databricks, but it was hidden with the error related to not finding package.json. This is my actual error, after my fix is applied:

cube_example  | Error: Error running static method
cube_example  | java.sql.SQLException: [Databricks][JDSI](20280) Required setting UID is not present in connection settings.
cube_example  |  at com.databricks.client.dsi.core.impl.DSIConnection.getRequiredSetting(Unknown Source)
cube_example  |  at com.databricks.client.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source)
cube_example  |  at com.databricks.client.spark.core.SparkJDBCConnection.establishConnection(Unknown Source)
cube_example  |  at com.databricks.client.jdbc.core.LoginTimeoutConnection.connect(Unknown Source)
cube_example  |  at com.databricks.client.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
cube_example  |  at com.databricks.client.jdbc.common.AbstractDriver.connect(Unknown Source)
cube_example  |  at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
cube_example  |  at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:189)
cube_example  |  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
cube_example  |  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
cube_example  |  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
cube_example  |  at java.base/java.lang.reflect.Method.invoke(Method.java:566)
cube_example  | Caused by: com.databricks.client.dsi.exceptions.BadAuthException: [Databricks][JDSI](20280) Required setting UID is not present in connection settings.

PR: #6521

[jjunior84]

I had the same problem, the tutorial says to copy the JDBC URL from you Databricks SQL Warehouse to use as connection, but the connection string is missing ;UID=token.

When I added it in the end of the connection, it worked fine!

I also was tested with Unity Catalog and I had to specify the catalog name from my database to work too.
CUBEJS_DB_DATABRICKS_CATALOG=<catalog_name>

[paveltiunov]

Fixed by #6521

[asanchezdelc]

I'm still getting the same error after pulling the latest image.

docker run -p 4000:4000 -p 3000:3000 --env-file=.env -v ${PWD}:/cube/conf -e CUBEJS_DEV_MODE=true cubejs-jdk:latest
Warning. There is no cube.js file. Continue with environment variables
🔥 Cube Store (0.33.12) is assigned to 3030 port.
Warning. Option apiSecret is required in dev mode. Cube has generated it as 93c98045d89c258a6d77d5210824904e
🔓 Authentication checks are disabled in developer mode. Please use NODE_ENV=production to enable it.
🦅 Dev environment available at http://localhost:4000
🔗 Cube SQL (pg) is listening on 0.0.0.0:15432
🚀 Cube API server (0.33.12) is listening on 4000
2023-05-24 17:09:04,326 INFO  [cubestored] <pid:19> Cube Store version 0.33.11
2023-05-24 17:09:04,330 INFO  [cubestore::http::status] <pid:19> Serving status probes at 0.0.0.0:3031
2023-05-24 17:09:04,332 INFO  [cubestore::metastore::rocks_fs] <pid:19> Using existing metastore in /cube/conf/.cubestore/data/metastore
2023-05-24 17:09:04,428 INFO  [cubestore::mysql] <pid:19> MySQL port open on 0.0.0.0:13306
2023-05-24 17:09:04,428 INFO  [cubestore::http] <pid:19> Http Server is listening on 0.0.0.0:3030
You accepted Terms & Conditions for JDBC driver from DataBricks by CUBEJS_DB_DATABRICKS_ACCEPT_POLICY
Downloading DatabricksJDBC42-2.6.29.1051
Release notes: https://databricks-bi-artifacts.s3.us-east-2.amazonaws.com/simbaspark-drivers/jdbc/2.6.29/docs/release-notes.txt
Error: ENOENT: no such file or directory, open 'package.json'
    at DatabricksDriver.testConnection (/cube/node_modules/@cubejs-backend/jdbc-driver/src/JDBCDriver.ts:205:13)
    at CubejsServerCore.getDriver (/cube/node_modules/@cubejs-backend/server-core/src/core/server.ts:792:7)
    at /cube/node_modules/@cubejs-backend/server-core/src/core/DevServer.ts:108:22
    at /cube/node_modules/@cubejs-backend/server-core/src/core/DevServer.ts:78:9

[carlagouveia]

@asanchezdelc Are you using the image cubejs/cube:jdk?

[caiobelfort]

Hi, tried again and still getting the same error. Using last image of tag jdk uploaded 2 days ago. Em qua., 24 de mai. de 2023 15:45, Carla Gouveia ***@***.***> escreveu:

…

@asanchezdelc <https://github.com/asanchezdelc> Are you using the image cubejs/cube:jdk? — Reply to this email directly, view it on GitHub <#6197 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHSS7QZDRCB3LYGPUJPD73XHZJOHANCNFSM6AAAAAAVFZ33F4> . You are receiving this because you commented.Message ID: ***@***.***>

[asanchezdelc]

@asanchezdelc Are you using the image cubejs/cube:jdk?

Yea, I'm using the latest tag. I also checked this #6651 code is in the container when running it.

[caiobelfort]

Ok, I managed to make it work. Apparently when we create docker compose and we don't provide the 'package.json' file in the root folder it is not possible to make the connection. I suggest a revision on documentation about this process.

My package.json is based on cube cloud development environment

{
  "name": "cubejs-project-template",
  "version": "0.0.1",
  "private": true,
  "scripts": {
    "dev": "cubejs-server"
  },
  "template": "docker",
  "templateVersion":   "0.33.11"    --> cube cloud set this as 0.28.19
}