[ADAP 619][Bug] Unable to materialize table in anything else than delta

[irsath]

Is this a new bug in dbt-spark?

• I believe this is a new bug in dbt-spark
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

The file_format config is not taken into account.
Table materialization is hard coded to be in delta.

dbt-spark/dbt/include/spark/macros/materializations/table.sql

Line 101 in e741034

df.write.mode("overwrite").format("delta").option("overwriteSchema", "true").saveAsTable("{{ target_relation }}")

My use case is to have AWS Glue as my global metastore.
I have my databricks clusters linked to my glue metastore and I can successfully read / write tables in it.
Unfortunately I'm not able to write anything else than delta table (as it is hardcoded)

Expected Behavior

The table is materialized in the format specified in the file_format config.

Steps To Reproduce

Execute this model:

from pyspark.sql import SparkSession

def model(dbt, session: SparkSession):
    dbt.config(
        file_format = "iceberg"
    )
    df =  .....my spark df....
    return df

Relevant log output

No response

Environment

- OS: Mac Os
- Python: Python 3.10.3
- dbt-core: 1.5.1
- dbt-spark: 1.5.0

Additional Context

No response

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.