[BUG] Databricks notebook setContext NoSuchMethodException

[iburquijo]

Description
This bugs appears when trying to set the notebook context from the getContext method.

Reproduction

import com.databricks.sdk.scala.dbutils.DBUtils
val dbutils2 = DBUtils.getDBUtils()
val ctx2 = dbutils2.notebook.getContext()
dbutils2.notebook.setContext(ctx2)

The error is the following:

NoSuchMethodException: com.databricks.backend.common.rpc.CommandContext.(scala.Option, scala.Option, java.lang.String, scala.collection.immutable.Map, scala.collection.immutable.Map)
Expected behavior
A clear and concise description of what you expected to happen.

This is due to the class com.databricks.backend.common.rpc.CommandContext constructor is com.databricks.sdk.scala.dbutils.CommandContext(scala.Option,scala.Option,scala.Option,scala.collection.immutable.Map,scala.collection.immutable.Map)

In this api (file: https://github.com/databricks/databricks-dbutils-scala/blob/main/databricks-dbutils-scala/src/main/scala/com/databricks/sdk/scala/dbutils/ProxyDBUtilsImpl.scala , line 171) is calling a field as String instead Option[String] (the third argument)

// This class is defined in DBR.
val commandContextClass = Class.forName("com.databricks.backend.common.rpc.CommandContext")
commandContextClass
.getConstructor(
classOf[Option[AnyRef]],
classOf[Option[AnyRef]],
classOf[String], //should be change to classOf[Option[String]]
classOf[Map[String, String]],
classOf[Map[String, String]])
.newInstance(
rootRunIdOpt,
currentRunIdOpt,
commandContext.jobGroup,
commandContext.tags,
commandContext.extraContext)
.asInstanceOf[AnyRef]

Debug Logs
N/A

Other Information
Running inside Databricks Notebooks.