dbt-sqlserver 1.7.2 is_incremental not compiling and overwrites the incremental load table #481
Replies: 9 comments 25 replies
-
Does it work as expected on the previous version? |
Beta Was this translation helpful? Give feedback.
-
I tried the following and couldn't recreate your issue. Removed existing log fileif a log file is in log/ then removed it. CREATE TABLEScreate table dbo.tst (
ID BIGINT,
DT DATETIME2
)
GO
INSERT INTO dbo.tst (ID, DT)
VALUES
(1, '2020-01-01'),
(2, '2023-01-01'),
(3, GETDATE());
GO Create DBT model{{
config(
materialized='incremental'
)
}}
select
*
from dbo.tst
{% if is_incremental() %}
-- this filter will only be applied on an incremental run
-- (uses >= to include records whose timestamp occurred since the last run of this model)
where DT >= (select max(DT) from {{ this }})
{% endif %} Ran dbt
expected and got 3 rows. Added new rowINSERT INTO dbo.tst (ID, DT)
VALUES
(9, GETDATE());
GO Ran dbt again
Expected result would be 5 rows. Max row from before + new additional row. Run full refresh
Expected 4 rows. 3 original + 1 new. Can you do the above and send the log file here. feel free to scrub your pii as needed. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
@schlich any chance you could try the above in your project and see if it passes? |
Beta Was this translation helpful? Give feedback.
-
https://github.com/cody-scott/incremental_test Here is a fresh project that also runs as expected. you can run the sql files to create the db + add the data. I'm using docker + sql server 2022 but it also works in 2019. I added the log files for you to compare. Without seeing more of the structure (how the configs are setup, dbt_project, etc.) i'm at a bit of a loss to understand more here unfortunately. |
Beta Was this translation helpful? Give feedback.
-
full pip export but the install was |
Beta Was this translation helpful? Give feedback.
-
Is this something that was fixed in 1.7.3 or 1.7.4? |
Beta Was this translation helpful? Give feedback.
-
Something doesn't look right with your model. You have a |
Beta Was this translation helpful? Give feedback.
-
I've recently tried to upgrade from 1.4.9 --> 1.7.4 and saw a similar behaviour. I ran SQL Profiler on my server and it seems that when DBT tries to get the objects in each database, the connection string remains as the one specified in profiles.yml Using the trace from @mark-manalo above, DBT creates 4 connections:
However, when its then opens the connections, they all point to "00_raw". (see log lines starting with "using connection string.." When DBT tries to run the incremental model, it does a "USE [01_dl]" before executing the table model... which fails as the model already exists.
|
Beta Was this translation helpful? Give feedback.
-
Hi Everyone,
I am having difficulties implementing incremental materialized model with the dbt-sqlserver adapter 1.7.2 version.
have the following code:
Target table exists.
But when I run the model with the newest data newest reference_dt it just overwrite the target table with the newest referenced_dt and all the other data is gone.
I look at the compiled code and the if is_incremental() is not evaluated:
I need some help please.
Beta Was this translation helpful? Give feedback.
All reactions