Skip to content

Commit

Permalink
Redshift unit test limitation (#5508)
Browse files Browse the repository at this point in the history
## What are you changing in this pull request and why?

Closes: #5430

## Checklist
<!--
Uncomment when publishing docs for a prerelease version of dbt:
- [ ] Add versioning components, as described in [Versioning
Docs](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-entire-pages)
- [ ] Add a note to the prerelease version [Migration
Guide](https://github.com/dbt-labs/docs.getdbt.com/tree/current/website/docs/docs/dbt-versions/core-upgrade)
-->
- [ ] Review the [Content style
guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md)
so my content adheres to these guidelines.
- [ ] For [docs
versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#about-versioning),
review how to [version a whole
page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version)
and [version a block of
content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content).
- [ ] Add a checklist item for anything that needs to happen before this
PR is merged, such as "needs technical review" or "change base branch."

Adding or removing pages (delete if not applicable):
- [ ] Add/remove page in `website/sidebars.js`
- [ ] Provide a unique filename for new pages
- [ ] Add an entry for deleted pages in `website/vercel.json`
- [ ] Run link testing locally with `npm run build` to update the links
that point to deleted pages
  • Loading branch information
matthewshaver committed May 15, 2024
2 parents dc0f8c5 + d01db25 commit b390a97
Show file tree
Hide file tree
Showing 2 changed files with 88 additions and 1 deletion.
2 changes: 1 addition & 1 deletion website/docs/docs/build/unit-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ With dbt Core v1.8 and dbt Cloud environments that opt to "Keep on latest versio
- You must specify all fields in a BigQuery STRUCT in a unit test. You cannot use only a subset of fields in a STRUCT.
- If your model has multiple versions, by default the unit test will run on *all* versions of your model. Read [unit testing versioned models](#unit-testing-versioned-models) for more information.
- Unit tests must be defined in a YML file in your `models/` directory.
- Available to dbt Cloud customers who have selected ["Keep on latest version"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) and dbt Core v1.8.0 or later.
- Table names must be [aliased](/docs/build/custom-aliases) in order to unit test `join` logic.
- Redshift customers need to be aware of a [limitation when building unit tests](/reference/resource-configs/redshift-configs#unit-test-limitations) that requires a workaround.

Read the [reference doc](/reference/resource-properties/unit-tests) for more details about formatting your unit tests.

Expand Down
87 changes: 87 additions & 0 deletions website/docs/reference/resource-configs/redshift-configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -257,3 +257,90 @@ The workaround is to execute `DROP MATERIALIZED VIEW my_mv CASCADE` on the data
</VersionBlock>

</VersionBlock>

<VersionBlock firstVersion="1.8">

## Unit test limitations

Redshift doesn't support Unit tests when the SQL in the common table expression (CTE) contains functions such as `LISTAGG`, `MEDIAN`, `PERCENTILE_CONT`, etc. These functions must be executed against a user-created table. dbt combines given rows to be part of the CTE, which Redshift does not support. For unit tests to function properly in this scenario, creating temporary tables for the unit tests to reference is a good workaround.

The following query illustrates the limitation:

```sql

create temporary table "test_tmpxxxxx" as (
with test_fixture as (
select
cast(1000 as integer) as id,
cast('menu1' as character varying(500)) as name,
cast( 1 as integer) as quantity
union all
select
cast(1001 as integer) as id,
cast('menu2' as character varying(500)) as name,
cast( 1 as integer) as quantity
union all
select
cast(1003 as integer) as id,
cast('menu1' as character varying(500)) as name,
cast( 1 as integer) as quantity
),
agg as (
SELECT
LISTAGG(name || ' x ' || quantity, ',') AS option_name_list,
id
FROM test_fixture
GROUP BY id
)
select * from agg
);

```
This query results in the error:

```bash

[XX000] ERROR: One or more of the used functions must be applied on at least one user created tables. Examples of user table only functions are LISTAGG, MEDIAN, PERCENTILE_CONT, etc

```

However, the following query works as expected:

```sql

create temporary table "test_tmp1234" as (
SELECT
cast(1000 as integer) as id,
cast('menu1' as character varying(500)) as name,
cast( 1 as integer) as quantity
union all
select
cast(1001 as integer) as id,
cast('menu2' as character varying(500)) as name,
cast( 1 as integer) as quantity
union all
select
cast(1000 as integer) as id,
cast('menu1' as character varying(500)) as name,
cast( 1 as integer) as quantity
);

with agg as (
SELECT
LISTAGG(name || ' x ' || quantity, ',') AS option_name_list,
id
FROM test_tmp1234
GROUP BY id
)
select * from agg;

```

When all given rows are created as a temporary table first, then running the test by referring to the temporary tables results in a successful run.

In short, separate the unit tests into two steps:
1. Prepare test fixtures by creating temporary tables.
2. Run unit test query by referring to the temporary tables.

</VersionBlock>

0 comments on commit b390a97

Please sign in to comment.