Databricks jobs create JSON accepted by API is rejected by CLI.

[Christopher-Lawford]

Describe the issue

Using the databricks cli to create a job doesn't work when you specify non string values in objects but the same value works when calling the API directly.
The object in question:

spark_conf
object
An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

The example failure configuration:

"spark_conf": {
  "spark.sql.autoBroadcastJoinThreshold": -1,
  "spark.sql.legacy.setCommandRejectsSparkCoreConfs": false
}

example passing configuration:

"spark_conf": {
  "spark.sql.autoBroadcastJoinThreshold": "-1",
  "spark.sql.legacy.setCommandRejectsSparkCoreConfs": "false"
}

Steps to reproduce the behavior

1. Create a simple databricks 2.1 job definition like below and save it to a location . e.g. C:/temp/Untitled1.json

{
  "tasks": [
    {
      "task_key": "Example",
      "new_cluster": {
        "spark_version": "11.3.x-scala2.12",
        "node_type_id": "Standard_E8d_v4",
        "num_workers": 1,
        "spark_conf": {
          "spark.sql.autoBroadcastJoinThreshold": -1,
          "spark.sql.legacy.setCommandRejectsSparkCoreConfs": false
        }
      },
      "libraries": [
        {
          "jar": "dbfs:/FileStore/example.jar"
        }
      ],
      "spark_jar_task": {
        "main_class_name": "uk.os.ngd.specregionenricher.Main",
        "parameters": ["dev"]
      }
    }
  ]
}

3. Run databricks jobs create --json '@C:/temp/Untitled1.json'

Expected Behavior

{
"job_id":1082401429085992
}

Actual Behavior

Error: invalid character '\r' in string literal

OS and CLI version

OS : Windows
CLI: Databricks CLI v0.209.1

Is this a regression?

Yes,
CLI: Version 0.17.0 was tried with a similar command
databricks jobs create --json-file 'C:/temp/Untitled1.json' --version 2.1
And this works as expected and creates the job.

Debug Logs

Slightly different paths in my debug run due to having both databricks cli's installed.
PS C:\Users\clawford> C:/Windows/Databricks.exe jobs create --json '@C:/temp/temp/Untitled1.json' --log-level Trace
time=2023-11-16T11:36:03.621Z level=INFO source="root.go 55}" msg=start pid=44856 version=0.209.1 args="C:\Windows\Databricks.exe, jobs, create, --json, @C:/temp/temp/Untitled1.json, --log-level, Trace"
time=2023-11-16T11:36:03.623Z level=TRACE source="config.go 193}" msg="Loading config via environment" pid=44856 sdk=true
time=2023-11-16T11:36:03.623Z level=TRACE source="config.go 193}" msg="Loading config via config-file" pid=44856 sdk=true
time=2023-11-16T11:36:03.623Z level=DEBUG source="config_file.go 100}" msg="Loading DEFAULT profile from C:\Users\clawford/.databrickscfg" pid=44856 sdk=true
time=2023-11-16T11:36:03.624Z level=TRACE source="auth_default.go 52}" msg="Attempting to configure auth: pat" pid=44856 sdk=true
Error: invalid character '\r' in string literal
time=2023-11-16T11:36:03.625Z level=ERROR source="root.go 114}" msg="failed execution" pid=44856 exit_code=1 error="invalid character '\r' in string literal"

[andrewnester]

@Christopher-Lawford it looks like it fails even eralier, on configuring auth. Does databricks current-user me command work?
If not, please make sure that you don't have any unnecessary new lines and etc. in your PAT token

[Christopher-Lawford]

Hello Andrew,
Yes that command works. I am already using the same cfg file when running commands like databricks jobs list, etc.

[andrewnester]

@Christopher-Lawford invalid character '\r' in string literal error usually indicates that there is some unexpected new line or so in configuration as for example here: https://stackoverflow.com/questions/70055650/terraform-invalid-character-r-in-string-literal

Could you try to run your JSON payload through some formatter like here and try formatted output?
https://jsonformatter.curiousconcept.com/

[Christopher-Lawford]

@andrewnester thanks for the suggestion. I had looked into that previously and used a different json validation service (https://jsonlint.com/) but both show the file being valid. You are free to try this yourself using the config from step 1 of the "Steps to reproduce the behavior" process.
It was through checking that this file worked on the old CLI and then slowly removing aspects of the json until it worked on the new CLI that I uncovered this issue.

[andrewnester]

@Christopher-Lawford this is a bit unclear but the values of spark_conf at the moment have to be strings as well, so your spark_conf should look like this

        "spark_conf": {
          "spark.sql.autoBroadcastJoinThreshold": "-1",
          "spark.sql.legacy.setCommandRejectsSparkCoreConfs": "false"
        }

This is a limitation on CLI side which we are actively working on. (cc @pietern )

[pietern]

@Christopher-Lawford Could you attach the JSON file itself that triggers the error for you? I suspect there is a newline hiding somewhere that we can't see in the pasted contents. This would allow us to reproduce the issue.

[pietern]

@Christopher-Lawford Nevermind the above, @andrewnester brought me up to speed and it's that the error is totally unclear and doesn't seem to be related to the type mismatch of the spark confs.

[andrewnester]

Closing as there's a workaround for this issue and we're working on long-term improvement to help with these sort of problems