Config Value spark.sql.mapKeyDedupPolicy not Supported by Databricks SQL Warehouse

[ArtnerC]

Getting the error spark.sql.mapKeyDedupPolicy is not supported by Databricks SQL Warehouses when using ibis pyspark with a Databricks SQL Warehouse Cluster.

See: https://community.databricks.com/t5/data-engineering/spark-settings-in-sql-warehouse/td-p/7959

Set in do_connnect:
https://github.com/ibis-project/ibis/blame/e425ad57899f8ebbea29b57bb53cedb40ebd7193/ibis/backends/pyspark/__init__.py#L180

self._session.conf.set("spark.sql.mapKeyDedupPolicy", "LAST_WIN")

Workaround could be as simple as:

try:
    spark.conf.set("spark.sql.mapKeyDedupPolicy", "LAST_WIN")
except Exception as e:
    if "not available" in str(e):
        print("Likely running in a SQL Warehouse")
    else:
        raise e  # Re-raise other exceptions

but I'm not sure what other approaches there might be.

[jcrist]

Thanks for opening this @ArtnerC! I think we'd happily accept a PR with the fix you suggest if you're interested in submitting one. I think we can skip checking for a specific exception and just ignore any exceptions on that line:

try:
    spark.conf.set("spark.sql.mapKeyDedupPolicy", "LAST_WIN")
except Exception:  # is there a specific exception class we could catch instead of just `Exception`?
    pass

Re: general databricks support, we don't have any immediate plans to set up a databricks testing environment (or a databricks-specific backend if needed), but if it's possible to make things work with just our existing pyspark backend, we'd happily continue to accept bugfixes towards making that work.