[CT-622] TempTableAlreadyExistsException: Temporary view already exists

[phillem15]

I have been having some issues in dbt cloud using the Databricks connection with delta merge as the incremental strategy for one of my models. The issue in the logs is "Error running query: org.apache.spark.sql.catalyst.analysis.TempTableAlreadyExistsException: Temporary view '<model_name>__dbt_tmp' already exists'.

I believe this may be caused by the following line of code saying 'create temporary view' rather than 'create or replace temporary view'. I have only been able to re-create this issue when running a different job on the same model. For example, we run a separate job instance in dbt cloud for hotfixes, rather than a full job run.

dbt-spark/dbt/include/spark/macros/adapters.sql

Line 126 in f03d217

create temporary view {{ relation.include(schema=false) }} as

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[InbarShirizly]

I have been having some issues in dbt cloud using the Databricks connection with delta merge as the incremental strategy for one of my models. The issue in the logs is "Error running query: org.apache.spark.sql.catalyst.analysis.TempTableAlreadyExistsException: Temporary view '<model_name>__dbt_tmp' already exists'.

I believe this may be caused by the following line of code saying 'create temporary view' rather than 'create or replace temporary view'. I have only been able to re-create this issue when running a different job on the same model. For example, we run a separate job instance in dbt cloud for hotfixes, rather than a full job run.

dbt-spark/dbt/include/spark/macros/adapters.sql

Line 126 in f03d217

create temporary view {{ relation.include(schema=false) }} as

something similar is happening to me, did you resolve it? @phillem15

[Fleid]

Hi @phillem15, @InbarShirizly - if you are targeting Databricks on dbt Cloud, the best is to use the dbt-databricks connection released since last November. A migration guide is available.

Which means that this should be solved not on dbt-spark but dbt-databricks (issues).

I'm closing this for now (until we hear from Spark users).

[data-blade]

hi @Fleid, spark users here, since still not fixed on dbt-spark. can we please re-open this? is it as easy as adding or replace to that line?

[shabbirk-db]

We really should fix this issue in dbt-spark because dbt-databricks continues to have dependencies to dbt-spark.
Macros in dbt-databricks like this one:
https://github.com/databricks/dbt-databricks/blob/0bedfd5be0355d504845a55e10dc849cbc6cc2e6/dbt/include/databricks/macros/adapters.sql#L69

They call the create temporary view macro themselves and so the same issue continues to present itself even for dbt-databricks users.

[data-blade]

pinging @dataders for help

[data-blade]

also @colin-rogers-dbt. apologies for pinging 3 maintainers.

[colin-rogers-dbt]

Re-opening as this does seem applicable to dbt-spark