Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't generate change stream via dbt #5

Closed
gunnarmorling opened this issue Mar 24, 2023 · 2 comments · Fixed by #10
Closed

Can't generate change stream via dbt #5

gunnarmorling opened this issue Mar 24, 2023 · 2 comments · Fixed by #10

Comments

@gunnarmorling
Copy link

gunnarmorling commented Mar 24, 2023

When adding the following pipeline via dbt:

SELECT p.category, sum(o.price)
FROM gunnar_orders o
    LEFT JOIN gunnar_products p ON o.product_id = p.id
    LEFT JOIN gunnar_shipments s ON s.order_id = o.order_id
GROUP BY p.category

I'm getting this error:

(dbt-venv) ➜  example git:(main) ✗ dbt run
12:01:14  Running with dbt=1.3.3
12:01:14  Found 1 model, 0 tests, 0 snapshots, 0 analyses, 273 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics
12:01:14
12:01:16  Concurrency: 1 threads (target='dev')
12:01:16
12:01:16  1 of 1 START sql table model gunnar_orders_aggregated .......................... [RUN]
12:01:19  1 of 1 ERROR creating sql table model gunnar_orders_aggregated ................. [ERROR in 2.71s]
12:01:19
12:01:19  Finished running 1 table model in 0 hours 0 minutes and 5.28 seconds (5.28s).
12:01:19
12:01:19  Completed with 1 error and 0 warnings:
12:01:19
12:01:19  Compilation Error in model gunnar_orders_aggregated (models/example/gunnar_orders_aggregated.sql)
12:01:19    Error checking changes to the 'gunnar_orders_aggregated' stream: Compilation Error
12:01:19      Type 'STRING PRIMARY KEY' not recognized
12:01:19
12:01:19    > in macro materialization_table_decodable (macros/materializations/table/table.sql)
12:01:19    > called by model gunnar_orders_aggregated (models/example/gunnar_orders_aggregated.sql)
12:01:19
12:01:19  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
@rmetzger
Copy link
Contributor

@nicoweidner
Copy link
Contributor

I think this highlights a general issue: When executing a query that requires a change stream as target (e.g. a GROUP BY query), there is no generic way to automatically assign a primary key in the target stream. When doing the same in the UI, the user has to select a primary key for the target stream, and dbt-decodable would need to do the same, either via query-specific configuration or a prompt for user input (though the latter feels like it goes against the purpose of dbt-decodable)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants