Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-3129] Ability to test the incremental behavior of models using incremental strategy that requires partitioning #8664

Open
Tracked by #8283
gshank opened this issue Sep 18, 2023 · 1 comment
Milestone

Comments

@gshank
Copy link
Contributor

gshank commented Sep 18, 2023

Description

Ability to test the incremental behavior of models using an incremental strategy that requires partitioning (insert_overwrite or delete+insert). See #8422.

Screenshot 2024-02-15 at 3 15 47 PM

If I have an incremental model that uses a strategy that uses "partitioning":

{{
  config(
    materialized='incremental',
    unique_key='date_day',
    incremental_strategy='insert_overwrite',
    ...
  )
}}

select
    date_trunc('day', event_at) as date_day,
    count(distinct user_id) as daily_active_users

from {{ ref('events') }} 

{% if is_incremental() %}

  -- this filter will only be applied on an incremental run
  -- (uses >= to include records arriving later on the same day as the last run of this model)
  where date_day >= (select max(date_day) from {{ this }})

{% endif %}

group by 1

I should be able to add a unit test to this model and have it work as expected.


unit_tests:
  - name: my_favorite_unit_test
    model: my_incremental_model
    given:
      - input: ref('events')
        format: csv
        fixture: my_events_input_fixture
      - input: this
        format: csv
        fixture: my_incremental_model_this_fixture
    expect:
      format: csv
      fixture: my_incremental_model_output_fixture
    overrides: 
      macros:
        is_incremental: true 

Acceptance Criteria

Unit testing of incremental models with insert_overwrite or delete+insert strategy.

Impact to Other Teams

None

Will backports be required?

No

Context

@github-actions github-actions bot changed the title Ability to test the incremental behavior of models using incremental strategy that requires partitioning [CT-3129] Ability to test the incremental behavior of models using incremental strategy that requires partitioning Sep 18, 2023
@graciegoheen graciegoheen added this to the v1.9 milestone Feb 15, 2024
@ChenyuLInx
Copy link
Contributor

blocked on #8499

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants