write to Agents Schema directly from dbt

[graciegoheen]

What is Agents Schema?

Agents Schema is an open-source standard for agentic context. Its a standard that defines how metadata should be stored inside your warehouse, for agents to easily look up the context they need for conversational analytics.

Many of you are probably familiar with the INFORMATION_SCHEMA — the standard set of views that provide information about all of the tables, views, columns, and procedures in a database (e.g. information_schema.tables). The Agents Schema is the standard set of views that provide information about all of the context your agents need (e.g. agents.dbt_model). While the INFORMATION_SCHEMA can be derived entirely from the metadata within your warehouse, the Agents Schema needs to get metadata from many sources.

Let’s say I’m an owner of a shop that sells magical wands. I ask an AI agent, “what does the distribution of sales look like across my different wand types over the past month?”. To answer this question, it first needs to understand the data landscape: which tables to query, how to join those tables appropriately, how to ensure the data is accurate, and so on.

Today, that context is scatted across a variety of tools — dbt, your data warehouse, your BI tool(s) of choice, a semantic layer. Your agent has to comb through all of these to even begin to answer your simple business question.

If instead, all of these tools write their metadata to a standard schema in your warehouse, your agent would instead have a single place to look — your context would be stored right alongside your data. Your agent could answer your questions faster and more reliably.

That standard can’t be overly optimized for a single tool, or gated within a single platform. Enter: Agents Schema, a vendor-neutral, open-source standard for agentic context.

Agents Schema was released on June 1 as a part of dbt + Fivetran’s vision for Open Data Infrastructure. To get started, you set up workflows for each source (dbt, Looker, OSI, etc.) that populate AGENTS.* tables in your warehouse. When you then ask an AI agent about your data, it can easily find the context it needs, without leaving the query interface (peep the agents-schema-analyst skill).

The existing dbt integration involves setting up a GitHub Workflow to scrape manifest.json. What if instead, dbt could write a small set of human-oriented metadata tables directly into Agents Schema as part of compile / build?

The idea of “write metadata about my dbt project to my warehouse” isn’t new: persist_docs, dbt-artifacts, dbt-project-evaluator, and elementary all accomplish this today — in different ways, optimized for different usecases. v2 of dbt Core ships with a new "index" layer (parquets to disk) to power features like our revamp of dbt-docs.

But dbt has always been an opinionated framework. It’s time for dbt to have an opinion about context: what’s the right amount, where to store it, and how.

Open Questions

How should users configure their dbt projects to write metadata to the Agents Schema?

Currently, users have to setup a Github Workflow and parse the manifest.json.

1. What is the right way for users to configure this?
   1. A new global flag (e.g. --write-agents-schema)
   2. A new node type (each table within Agents Schema is auto-injected into the DAG)
   3. A top-level yaml key
   4. Something else?
2. Is this just an “on/off” configuration? Or do folks want to configure additional things?
   1. target schema, database, alias name?
   2. which nodes of a project are included? (vs. just “all of them”)
3. Would people want to write to dev / CI schemas? Or just prod?
4. Should the Agents Schema tables only be written if command completes successfully?
5. Is a full-refresh on each write performant enough? Or do we need to do merge statements / track state? How do we ensure Agents Schema doesn’t get out of sync?
6. Would users (also? instead of?) want this metadata written to an Iceberg catalog?

Which tables and columns should the Agents Schema contain for dbt metadata?

Currently, Agents Schema contains 3 dbt-specific views: dbt_model, dbt_column, and dbt_dependency.

1. What should the table names and schemas be to optimize for agentic performance and human readability?
2. What is the appropriate “minimal, curated subset” to balance giving agents enough context without bloat?
   1. Should we only write that subset to the warehouse? And leave everything else in the parquets?
   2. Or should we write all of the dbt metadata to the warehouse, and then add tools/skills to retrieve just the subset?

[karenderer]

Are you planning to add the dbt semantic layer to this? I think right now the recommendation seems to be to convert the dbt semantic definitions to the OSI spec and then use a Github workflow to push the converted files to the agents schema, it would be much better if it just worked out of the box.

1. What is the right way for users to configure this?

I'd want to be able to choose when I want to write to the schema, so I don't think I'd want a top level yaml key. A flag seems best. I am imagining that I'd create a task to run after our daily pipeline that would execute something like a dbt compile --write-agents-schema and maybe in deployments dbt build -s state:modified+1 --write-agents-schema if it were possible to incrementally update context.

2. Is this just an “on/off” configuration? Or do folks want to configure additional things?

The option to exclude selectively is helpful, my agents don't need to know about specific tables or schemas.

3. Would people want to write to dev / CI schemas? Or just prod?

We use developer DBs (not schemas), so I imagine I would run it in dev for occasional testing.

4. Should the Agents Schema tables only be written if command completes successfully?

I'd hate to not be able to update my context if it is not completing successfully for 1 row/some edge case.

5. Is a full-refresh on each write performant enough? Or do we need to do merge statements / track state? How do we ensure Agents Schema doesn’t get out of sync?

I think incremental would be nice (so that it could be used in deployments) if the agents schema doesn't get out of sync.

1. What should the table names and schemas be to optimize for agentic performance and human readability?

I think it would a miss to exclude the meta config. The dbt-artifacts package makes that column available too. We use that column to store model ownership and I think it would be nice to have it available.

2. What is the appropriate “minimal, curated subset” to balance giving agents enough context without bloat?

I think giving the option to control this by schema and on an additive basis (maybe tag based method) would be helpful, e.g. give me the ability to add everything in 3 schemas + 4 tables from another schema that has a bunch of non-agent relevant tables in it, but has 4 useful tables.

[Adarsh-Hota]

I'd lean towards a top-level YAML configuration. Feels more like project configuration.

I could also see value in configuring the target schema/database and which nodes publish metadata to AgentsSchema.

About dev environments... if I'm building an internal AI tool against my dev environment, I'd want it to understand the models I'm actively working on, not just what's in production. Other teams may only want production, so imo this could be a project configuration choice. Then I can simply change it in my dev branch and run dbt build.

For publishing, I would prefer to have metadata for the state the command reaches, while having the option to override that behavior with a CLI flag if I only want to publish after a successful build.

[pempey]

Open Questions

I would lean to a file based config, something that can be put into change management, over a run command flag. The more configuration the better, there are so many ways that dbt projects are used so whatever the default is someone will find a different configuration more useful.

I would want to be able to write everting about each invocation, including model and analysis failures. If there is metadata about executed command then the incomplete executions may be mor valuable than the completed ones.

Every invocation the models could have changed so there should be some kind of awareness of that or be completely blind to it and keep enough information to make the data unique.

The use case for the information, having dbt model metadata in something like the agents schema, and that make it hard to identify what informing would be useful to have there.

Opinion

I am not fully sold on the idea of the agents schema. I see the agents space fall into two broad interactions with the dbt related efforts: asking questions about models (metadata context) and asking question about the data the models produce (action context). I think that the metadata context about models is most useful during development and that context being distracting when asking questions about the data and looking for action context. And if it is most useful for development I do not what to have to query my data warehouse for that information.

I would think that a product database that could be brought back into a dbt project in as a source, just like one might with a tools like Airflow, would be more helpful. Then this metadata context can be built into tables and views, using all of our current development knowledge and have action context applied to them so that the information can be more easily used.

We have found that creating action context by digesting the needed business context, formatting it for AI consumption like a skill, and adding it to the object description in the database thought the dbt docs provides the context needed to query and interpret the query results. This treats the action context as part of the transformation, like instructions for IKEA furniture, like we have long propertied the documentation to be. I do not think we need a new way to store the documentation for the models, we just have to write it for a new audience.