Fusion diaries: Redshift, clone, stack traces (2025-08-06)

[dataders]

This has been a great week for the yin and yang of “release big new features” and “polish the experience of the existing features”. The below list is impressive to have shipped in just one week! It’s really nice to see some long-running projects pay off in big ways:

• 30+ issues closed across the public and internal repos
• 5 new betas released
• A new adapter: Redshift!
• A new command: clone!
• New features: CTRL+C to cancel, --empty and --sample
• More precise stack traces in packages’ error messages
• 1 squashed bug that was a personal "white whale" of mine for months

🆕 What’s been released this week

dbt Fusion engine releases

While we’re still in beta, you should always first run dbt system update to make sure you’re on the latest and greatest!

Here’s an abridged version of what we’ve shipped over the past week

• 2.0.0-beta.60
  • unblocks dbt-external-tables users by discovering unqualified macro names: dbt-fusion#441
• 2.0.0-beta.58 reverted a change where SAO skips rebuilding views
• 2.0.0-beta.57
  • clone dbt-fusion#22
  • cancel runs with CTRL+C dbt-fusion#14
  • further refine when introspection queries get labelled as unsafe dbt-fusion#453
  • BigQuery fix for getting the table schema of info_schema views dbt-fusion#498
  • my personal white whale of a bug dbt-fusion#504
• 2.0.0-beta.56
  • Redshift dbt-fusion#6
  • anchors are supported! dbt-fusion#116
  • --sample and --empty dbt-fusion#34
• 2.0.0-beta.55
  • make stack traces more helpful dbt-fusion#492

Big Rocks

As hinted at above, here’s some big things from this week that are especially worth your attention:

Redshift!

Redshift is now in beta with support for user+password authentication with support for more authentication methods coming later this year.

clone!

clone was shipped this week — try it out to create zero-copy clones (or pointer views as a fallback) of production objects in your development schema.

This is especially important as a workaround for situations where state deferral isn't working as intended.

Better stack traces!

Stack traces are the part of the error message that point you right to where an error happens. Fusion’s improved Jinja handling included precise line-level error messaging, but sometimes it didn’t work across package boundaries. This is a huge DX unlock, especially since you can click a warning in your IDE and go to the exact line (not to mention the critical context they provide to LLMs).

longer explanation of the improvements

To illustrate, here's an error I made up just now

• in models/orders.sql, I employ a dbt_utils.get_column_values() to get the values from the stg_payments model’s payment_method column.
• but I misspell the column name and add an extra z.

Now I want to show you how this error is surfaced when dbt compile is run. In each example, consider how helpful the error message is.

dbt Core

The big pros are that it tells me

• in which model the problem lives: orders.sql
• the column name that is invalid

However:

• what does line 2 at position 16 have to do with anything? in my model the error is on line 17?
• what dbt or SQL function failed inside the model? I have no clue

Runtime Error
  Database Error in model orders (models/orders.sql)
    000904 (42000): SQL compilation error: error line 2 at position 16
    invalid identifier 'PAYMENT_METHODZ'

dbt Fusion (last week)

This error and stacktrace was arguably worse than Core's as we don't know which dbt model causes the error. While indicating get_column_values() is novel information and correct, there isn't much actionable. Where should I go to solve this problem?

error: dbt1501: Failed to render SQL invalid operation: AdapterError: ADBC error: [Snowflake] 000904 (42000): SQL compilation error: error line 2 at position 16
invalid identifier 'PAYMENT_METHODZ' (SQLSTATE: 42000, Vendor code: 904)
(in dbt_packages/dbt_utils/macros/sql/get_column_values.sql:10)
  --> dbt_packages/dbt_utils/macros/sql/get_column_values.sql:10:1

dbt Fusion (today)

Wow this is much better! Those trained in stack traces know that the last line is the most important. Crucially, I can command+click on that path and it brings me to the exact offending line 17 column 14 (where get_column_values() is used in orders.sql).

The preceding lines in the call stack are helpful as they show how dbt macros depend on one another: in this case that get_column_values is effectively a wrapper around a statement block.

error: dbt1501: Failed to render SQL invalid operation: AdapterError: ADBC error: [Snowflake] 000904 (42000): SQL compilation error: error line 2 at position 16
invalid identifier 'PAYMENT_METHODZ' (SQLSTATE: 42000, Vendor code: 904)
(in models/orders.sql:17:43)
(in dbt_packages/dbt_utils/macros/sql/get_column_values.sql:2:31)
(in dbt_packages/dbt_utils/macros/sql/get_column_values.sql:17:14)
(in dbt_internal_packages/dbt-adapters/macros/etc/statement.sql:14:35)
  --> models/orders.sql:17:14

VS Code Extension: Refresh source schemas

Earlier this week @Simonot asked:

is there a way to only refresh sources? I changed the schema of a source (it used to have a saturday_delivery bool attribute, and it now has a business_day json attribute). Now when I run compile my project, I get the following error:

error: dbt0209: Failed to resolve function json_extract_scalar: No column business_days found. Available are source.created_at, source.updated_at, source.use_promise, source.carrier_code, source.enabled, source.id, source.company_id, source.promise, source.sunday_delivery, source.saturday_delivery, source.default_carrier_service_id
  --> models/staging/monolith_db/stg_monolith_db__carrier_services.sql:8:29 (target/compiled/models/staging/monolith_db/stg_monolith_db__carrier_services.sql:9:29)

Have I got news for you @Simonot! Check out this fancy new "Refresh source schemas" button!

VS Code Extension: Lateral references in the tool tip

If you hover over a column alias, it's easy to show the origin of an alias, as long as it's a single column. But what happens if the column expression has two columns from two tables?

In beta.57, we shipped an update that displays multiple origins.

The gnarliest bug I've seen so far

Here’s the setup for what we discovered in dbt-fusion#504

1. a package has a schema defined for a model in the package’s dbt_project.yml
2. the user project has a different schema configured for that model
3. the package model in question sits in a folder in the package that has more than 40 other models

If the above were true, Fusion was somehow ignoring the user’s schema config! Shoutout to Chenyu for finding the root cause in what turned out to be a one-line change in Rust.

I’m 95% certain that @paulavidela and I observed this bug months ago. I’m so excited to know it’s gone!

🚧 Work in progress

There's a lot of work in progress but here are a few areas that are more immediately relevant.

Finalizing the new dbt authoring layer

We've been working on a long-term authoring layer that's shared by dbt Core and the new dbt Fusion engine. What does this work mean to you, dear user? It enables you to a smoother YAML editing experience, by both giving errors as you edit, as well as point you directly to errors that the language server surfaces.

There's a light at the end of the tunnel we can see now beyond which there are no more changes to the YAML spec (except for semantic models and metrics). This is incredible news as it means that users can confidently upgrade to the new authoring layer and not have to make tweaks on an ongoing basis. It also sets us up to durably solve the below challenge for the dbt package ecosystem.

In the meantime, we foresee the following as the last pieces of work to be done for standardizing the YAML authoring experience (excluding the semantic layer constructs): dbt-fusion#511, dbt-fusion#512, and dbt-autofix#114

dbt package parse conformance

you may have been seeing many warnings coming that point to your packages. this is to be expected! the solution is two-fold:

• make ‘em less noisy: (dbt-fusion#490)
• make ‘em go away: coming soon, an internal initiative to team up with our amazing package maintainers across the ecosystem to get new releases out that are compatible with the new authoring layer.

caching & “introspective” macros

We continue to refine the intersection on how state deferral, introspective macros, and static analysis intersect.

Beyond a few long-tail bugs and behavior discrepancies with Core, the real work in flight is work we've previously mentioned around implementing a cache for introspective queries (dbt-fusion#203) and also upcoming "incremental compilation". This will have an incredible impact on developer experience. More on that in a future diary!

🐉 Here be dragons

In the previous two diary entries, we’ve mentioned that the new authoring layer requires test arguments to be nested within an arguments: config.

This change has landed in Core, Fusion, and dbt-autofix!

There are two caveats with autofix to share (shoutout to @VDFaller and @coopawhite for flagging).

1. autofix today does not touch built-in tests like accepted_values, which naturally also come with arguments:. (see: dbt-autofix#114)
2. autofix's --include-packages cannot resolve test arguments in packages (fix in progress! dbt-autofix#113)

🤔 Looking for feedback

Last week, we asked for feedback on UDFs and the UX of stack traces and logs. Both of those are still valid topics to rant and rave to us about. I’d add to that list any feedback you have on new features we shipped this week (sample mode, clone , Redshift)

Please don’t hesitate! You make our day when you reach out — scout’s honor.

👓 Stuff you should read

The agenda is live for Coalesce in October! Check it out and please consider attending in person.

🏁 Made it to the meme

Any K-Pop Demon Hunter fans out there? Shoutout again to @gwenwindflower for being the supreme queen of the data meme