write to Agents Schema directly from dbt

[graciegoheen]

What is Agents Schema?

Agents Schema is an open-source standard for agentic context. Its a standard that defines how metadata should be stored inside your warehouse, for agents to easily look up the context they need for conversational analytics.

Many of you are probably familiar with the INFORMATION_SCHEMA — the standard set of views that provide information about all of the tables, views, columns, and procedures in a database (e.g. information_schema.tables). The Agents Schema is the standard set of views that provide information about all of the context your agents need (e.g. agents.dbt_model). While the INFORMATION_SCHEMA can be derived entirely from the metadata within your warehouse, the Agents Schema needs to get metadata from many sources.

Let’s say I’m an owner of a shop that sells magical wands. I ask an AI agent, “what does the distribution of sales look like across my different wand types over the past month?”. To answer this question, it first needs to understand the data landscape: which tables to query, how to join those tables appropriately, how to ensure the data is accurate, and so on.

Today, that context is scatted across a variety of tools — dbt, your data warehouse, your BI tool(s) of choice, a semantic layer. Your agent has to comb through all of these to even begin to answer your simple business question.

If instead, all of these tools write their metadata to a standard schema in your warehouse, your agent would instead have a single place to look — your context would be stored right alongside your data. Your agent could answer your questions faster and more reliably.

That standard can’t be overly optimized for a single tool, or gated within a single platform. Enter: Agents Schema, a vendor-neutral, open-source standard for agentic context.

Agents Schema was released on June 1 as a part of dbt + Fivetran’s vision for Open Data Infrastructure. To get started, you set up workflows for each source (dbt, Looker, OSI, etc.) that populate AGENTS.* tables in your warehouse. When you then ask an AI agent about your data, it can easily find the context it needs, without leaving the query interface (peep the agents-schema-analyst skill).

The existing dbt integration involves setting up a GitHub Workflow to scrape manifest.json. What if instead, dbt could write a small set of human-oriented metadata tables directly into Agents Schema as part of compile / build?

The idea of “write metadata about my dbt project to my warehouse” isn’t new: persist_docs, dbt-artifacts, dbt-project-evaluator, and elementary all accomplish this today — in different ways, optimized for different usecases. v2 of dbt Core ships with a new "index" layer (parquets to disk) to power features like our revamp of dbt-docs.

But dbt has always been an opinionated framework. It’s time for dbt to have an opinion about context: what’s the right amount, where to store it, and how.

Open Questions

How should users configure their dbt projects to write metadata to the Agents Schema?

Currently, users have to setup a Github Workflow and parse the manifest.json.

1. What is the right way for users to configure this?
   1. A new global flag (e.g. --write-agents-schema)
   2. A new node type (each table within Agents Schema is auto-injected into the DAG)
   3. A top-level yaml key
   4. Something else?
2. Is this just an “on/off” configuration? Or do folks want to configure additional things?
   1. target schema, database, alias name?
   2. which nodes of a project are included? (vs. just “all of them”)
3. Would people want to write to dev / CI schemas? Or just prod?
4. Should the Agents Schema tables only be written if command completes successfully?
5. Is a full-refresh on each write performant enough? Or do we need to do merge statements / track state? How do we ensure Agents Schema doesn’t get out of sync?
6. Would users (also? instead of?) want this metadata written to an Iceberg catalog?

Which tables and columns should the Agents Schema contain for dbt metadata?

Currently, Agents Schema contains 3 dbt-specific views: dbt_model, dbt_column, and dbt_dependency.

1. What should the table names and schemas be to optimize for agentic performance and human readability?
2. What is the appropriate “minimal, curated subset” to balance giving agents enough context without bloat?
   1. Should we only write that subset to the warehouse? And leave everything else in the parquets?
   2. Or should we write all of the dbt metadata to the warehouse, and then add tools/skills to retrieve just the subset?

[karenderer]

Are you planning to add the dbt semantic layer to this? I think right now the recommendation seems to be to convert the dbt semantic definitions to the OSI spec and then use a Github workflow to push the converted files to the agents schema, it would be much better if it just worked out of the box.

1. What is the right way for users to configure this?

I'd want to be able to choose when I want to write to the schema, so I don't think I'd want a top level yaml key. A flag seems best. I am imagining that I'd create a task to run after our daily pipeline that would execute something like a dbt compile --write-agents-schema and maybe in deployments dbt build -s state:modified+1 --write-agents-schema if it were possible to incrementally update context.

2. Is this just an “on/off” configuration? Or do folks want to configure additional things?

The option to exclude selectively is helpful, my agents don't need to know about specific tables or schemas.

3. Would people want to write to dev / CI schemas? Or just prod?

We use developer DBs (not schemas), so I imagine I would run it in dev for occasional testing.

4. Should the Agents Schema tables only be written if command completes successfully?

I'd hate to not be able to update my context if it is not completing successfully for 1 row/some edge case.

5. Is a full-refresh on each write performant enough? Or do we need to do merge statements / track state? How do we ensure Agents Schema doesn’t get out of sync?

I think incremental would be nice (so that it could be used in deployments) if the agents schema doesn't get out of sync.

1. What should the table names and schemas be to optimize for agentic performance and human readability?

I think it would a miss to exclude the meta config. The dbt-artifacts package makes that column available too. We use that column to store model ownership and I think it would be nice to have it available.

2. What is the appropriate “minimal, curated subset” to balance giving agents enough context without bloat?

I think giving the option to control this by schema and on an additive basis (maybe tag based method) would be helpful, e.g. give me the ability to add everything in 3 schemas + 4 tables from another schema that has a bunch of non-agent relevant tables in it, but has 4 useful tables.