Add PRD for data abstraction layer #71

calebmcquaid · 2026-07-06T15:11:16Z

Can we talk about how Coop would consume the DAL? My understanding of Partial Items is that its responses are validated against the Item Type schema and downstream code depends on that shap. Would the DAL be similar? How would the Item Type map to a DAL object?

this is a great question! i laid out three options here: https://github.com/roostorg/community/pull/71/changes#r3539117701

i kind of prefer the strict version, but that makes the DAL a prerequisite for deploying Coop and makes the lift of building this larger.

juanmrad · 2026-07-06T14:57:51Z

Is "full API access per token" an intentional v1 design decision, or just a simplification?

Since the DAL may expose sensitive organizational data, it would be useful to call out that the authentication model is intentionally coarse-grained for v1 and designed to evolve toward field/object-level authorization later. Otherwise readers may interpret this as a long-term architectural choice.

An intentional design decision! Field-level authorization is mentioned in the Future Work section but lmk if I can make that more clear!

julietshen · 2026-07-06T16:09:47Z

if we punt this question, does that change the requirements for the DAL?

I think it's impossible to punt fully. Coop needs to be able to communicate with the DAL (otherwise the DAL will be pretty useless) and this decision is crucial for how the API will work.

The weak version would be that we put the onus on the user to implement an ontology that matches what they have in Coop. But this is fragile and error-prone.

The medium version is that Coop somehow checks if the DAL's ontology is compatible with its defined Item Types, and errors if not.

The strict version is that the ontology lives entirely in the DAL and Coop just consumes from that. I.e. the DAL becomes the new source of truth for Item Types.

julietshen · 2026-07-06T16:12:25Z

this is my biggest concern - are we adding more friction and work for adopters and needing them to write more custom code?

if the adopters don't use the partial items API -- no, if we keep the DAL as an optional bolt-on, they don't need to do anything.

if the adopters do want to use the partial items API -- not really. i'd say that the DAL is a more generalised version of implementing the same idea. of course, if we do deprecate the partial items API, any existing adopters would need to migrate eventually per some deprecation timeline. (i would not recommend maintaining both pathways indefinitely).

though since the DAL will be deployed as a new service, there is some operational overhead to deploying it. we can work to minimize this.

julietshen · 2026-07-06T16:13:21Z

and if it's optional while we deprecate Partial Items, are we making Coop worse out of the box?

i'd say no -- there's still a way to achieve the same thing (i.e. via the DAL), and though the tech stack might be more opinionated (e.g. requiring graphQL) it is fundamentally not very different.

julietshen · 2026-07-06T16:14:51Z

i'd like to see an aggregated list as part of the Risks of what new problems/pain points might we be introducing to adopters

writing custom code as resolvers for the DAL

is overloading source systems an existing risk or a new one we'd introduce

requiring APIs that provide broad access to their data

(as in - a lot of that is here already, but call it out specifically as potential challenges for adopters)

i think these risks are basically all potential problems or pain points for adopters. but some also apply to the Partial Items API (i.e. they are pre-existing!).

the new problems that the DAL, as written here, could introduce are:

graphQL may be unfamiliar dev experience/tooling

the overhead of deploying a new service

it's easy to create N+1s in graphQL

the ones that also apply to the partial items API:

there is eng lift involved

source systems may be slow to respond with data

source systems may get overloaded

the API allows broad access to data

juanmrad · 2026-07-06T14:59:49Z

I wonder if it's worth calling this out earlier in the document as part of the design philosophy.

Something like:

The DAL is intentionally read-only, which allows implementations and clients to safely leverage aggressive caching where appropriate.

I think that helps explain why caching is a natural fit for the architecture, rather than introducing it later as an optimization.

I added a note in the Approach section!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add PRD for data abstraction layer #71

Uh oh!

Diff view

Diff view

There are no files selected for viewing

calebmcquaid Jul 6, 2026

Uh oh!

taobojlen Jul 7, 2026

Uh oh!

juanmrad Jul 6, 2026

Uh oh!

taobojlen Jul 7, 2026

Uh oh!

julietshen Jul 6, 2026

Uh oh!

taobojlen Jul 7, 2026

Uh oh!

julietshen Jul 6, 2026

Uh oh!

taobojlen Jul 7, 2026

Uh oh!

julietshen Jul 6, 2026

Uh oh!

taobojlen Jul 7, 2026

Uh oh!

julietshen Jul 6, 2026

Uh oh!

julietshen Jul 6, 2026

Uh oh!

taobojlen Jul 7, 2026

Uh oh!

juanmrad Jul 6, 2026

Uh oh!

taobojlen Jul 7, 2026

Uh oh!

-Original file line number
+Diff line change
@@ -0,0 +1,197 @@
+    # PRD 001: Data Abstraction Layer
+    ## Problem statement
+    In content moderation, more context can lead to better decisions, whether those decisions are taken by a human or an AI agent. Today, a Coop reviewer can only see limited data (whatever is pushed in). Accessing anything beyond that -- like an account's history or other posts -- has to be looked up in other tools, or populated using the Partial Items endpoint (which cannot populate information about relationships). If an org decides that some new field would be useful for moderation, then it requires engineering lift to get that data into each T&S tool.
+    In other words, getting richer context requires N custom integrations, because nothing abstracts over an org's objects, relationships, and signals.
+    **Success would mean**: an org implements that integration *once* -- declaring its ontology/object graph and exposing a query/enrichment API -- and every consumer reuses it, whether it's a UI for humans in Coop, agentic review, Osprey, or whatever comes next.
+    ## Background
+    The current [roadmap](https://github.com/roostorg/community/blob/main/roadmap.md#data-abstraction-layer) describes two components needed to prepare ROOST's projects for AI-powered features. One of these is a data abstraction layer (DAL) that lets both Coop and Osprey understand org-specific data models. This PRD focuses on the DAL and a Coop integration. There are a few more initial notes on this [here](https://github.com/orgs/roostorg/discussions/61#discussioncomment-16495095).
+    Today, Coop lets orgs configure its data model via Item Types. There are three kinds of Item:
+    - Content (any piece of user-generated content)
+    - Thread (an ordered collection of Content)
+    - User (an account or profile on the org's platform)
+    This is flexible, but has limitations. While it supports foreign-key relationships between Item Types, users cannot easily model many-to-many relationships such as users following each other.
+    Coop users push Items into Coop via our API, and items may then be shown in the Manual Review Tool (MRT). Items can be enriched at review time via the Partial Items feature -- the user can implement an API that returns further details about Items. This means that the latest version of an Item can be fetched on-demand.
+    ## Goals
+    A unified API to query for more information benefits both human, automated, and AI-driven review. We want to solidify & generalize the good ideas in Coop's Item Types datamodel and Partial Items API.
+    The concrete story we are initially targeting is:
+. An org implements the graph API, deployed as a new DAL service.
+        - We can write an agent skill to make this process smoother.
+. In Coop's Manual Review Tool, if this DAL is configured, the Partial Items API is replaced with a query to the DAL.
+    This new DAL service will not be required to run Coop today, though that may change in the future. But for now, we will deprecate the Partial Items API and replace it with the DAL. Any org that uses Partial Items will have to migrate.
+    ## Non-goals
+    - We are not building a graph database.
+    - We are also not building a new GraphQL library -- we will re-use an existing one.
+    - The DAL will not support unbounded multi-hop graph traversal, or advanced graph queries like community detection.
+    - The DAL does not persist data long-term (beyond, perhaps, caching). Data continues to live in the org's source systems.
+    - The Safety Decision Taxonomy from ROOST's roadmap is not included in this PRD -- though the DAL may be a sensible place to centralize this, too.
+    ## Approach
+    An org running Coop decides that they want to show more data in the reviewer UI. This is the flow I envision:
+. A developer clones a Roost template repo, e.g. `roostorg/data-abstraction-layer-template`.
+. Inside this template, they first define their ontology (what object types exist in their org, what properties do they have, and how do they relate to each other). The ontology is defined in some markup language like YAML or JSON.
+. Then, they write e.g. Python code to implement query resolvers that serve this ontology as a GraphQL API.
+        - This might only involve reading from a single database, or it could mean calling out to various data warehouses, APIs, and more. The goal is to unify everything into one API.
+        - The template provides the scaffolding to do this with a great DevX, providing the library, test setup, etc. to make the process easy.
+    For steps 2. and 3., an agent skill provides an easy walkthrough that does a lot of the heavy lifting. The developer's job then primarily becomes thinking through how to map the org's source system(s) together into a unified API. Note that this is a read-only API; the developer does *not* need to implement any mutations.
+    An example ontology for a simple blogging platform might look like:
+    > Object types
+    > - User
+    >     - ID (UUID)
+    >     - email (string)
+    >     - username (string)
+    >     - created_at (datetime)
+    > - BlogPost
+    >     - ID (UUID)
+    >     - content (string)
+    >     - created_at (datetime)
+    > - Comment
+    >     - ID (UUID)
+    >     - content (string)
+    >     - created_at (datetime)
+    >
+    > Relationships
+    > - (User)-[:follows]->(User)
+    > - (User)-[:author_of]->(BlogPost)
+    > - (User)-[:author_of]->(Comment)
+    > - (Comment)-[:reply_to]->(BlogPost)
+    An example query to fetch a user, their number of followers, and their 10 most recent blog posts might look like:
+    ```gql
+    query {
+      user(id: "019ef42a-6531-7039-8a8b-096d1ee2db64") {
+        email
+        username
+        authorOf {
+          blogPosts(orderBy: [{ created_at: DESC }], first: 10) {
+            edges {
+              node {
+                content
+                created_at
+              }
+            }
+          }
+        }
+        # The schema auto-generates reverse relationships, too
+        reverseFollows {
+          user {
+            totalCount
+          }
+        }
+      }
+    }
+    ```
+    In brief: a new module, that Coop (and eventually perhaps also Osprey) both depend on. It exposes a read-only GraphQL API that matches an organization's ontology. The user then writes GraphQL resolvers that translate incoming GraphQL requests into reads against their source systems. It is completely agnostic to these source systems -- could be data warehouses, operational DBs, or further API calls to other services. The DAL centralizes reads of all an organization's data into a single well-defined GraphQL API.
+    ### Why GraphQL?
+    GraphQL is a near-perfect fit for this use case for several reasons.
+    For one, thinking in graphs is both common & intuitive for investigative use cases. GraphQL puts this first. It's also an existing standard which means that humans & LLMs are familiar with it, and there's a large ecosystem.
+    GraphQL also natively supports [introspection](https://graphql.org/learn/introspection/) which lets callers dynamically learn the schema (i.e. the ontology) of a given API. This is great since this will vary depending on the org that's implementing the DAL. We may not need to build a custom API to share the ontology between services.
+    Finally, writing a GraphQL API that calls out to one or more source systems is very accessible. Developers write [resolvers](https://graphql.org/learn/execution/) for objects or fields, and the GraphQL library automatically handles running these to serve an incoming GraphQL query.
+    Using GraphQL for querying does not preclude MCP; they are different layers. It's still possible to build a GraphQL-over-MCP transport for LLM agents in the future if we wish, which may be useful for auth purposes.
+    The main alternatives would be REST, but this loses the graph-native shape and introspection, or more advanced graph query languages like [Cypher](https://en.wikipedia.org/wiki/Cypher_(query_language)), which is a lot more advanced -- more comparable to SQL than to REST, for example.
+    ## Scope
+    - The API will be read-only.
+    - We will re-use existing libraries rather than publishing something entirely new. The main work is the wiring and a great template, not a new GraphQL library.
+    - v1 targets Coop's MRT as the single consumer; Osprey integration is out of scope for this PRD.
+    - The (deployable) template repo + agent skill are in-scope deliverables.
+    ## Functional requirements
+. A read-only GraphQL API derived from a declared ontology (i.e. object types, properties, and relationships).
+. The API handles relationships between object types, including self-referential; reverse relationships are auto-generated.
+. The API supports GraphQL introspection, so consumers can discover the ontology at runtime.
+. Relationship connections with [Relay-style pagination](https://relay.dev/graphql/connections.htm).
+. Query depth is bounded by a configurable maximum.
+. Developers write resolvers per object type/field. This is source-system agnostic.
+. A template repo with resolver scaffolding, a test harness, examples, and support for dataloaders to prevent N+1s.
+. When configured, Coop enriches from the DAL instead of Partial Items.
+. Coop's Partial Items API is deprecated with a migration path.
+. Bearer-token auth -- a token grants access to the entire API.
+. Requests are audit-logged with caller identity and query content.
+. Structured logs, OTel traces (per-resolver, with source-system spans), and RED metrics (count, latency, error rate).
+    ## Non-functional requirements
+. The DAL adds no more than 100ms request latency (p95) on top of querying the source systems.
+. Coop stays functional when the DAL is absent or degraded.
+. Users can add TLS for all requests.
+. API tokens are revocable.
+. The DAL service is self-hosted by the org, deployable from the template with minimal custom code.
+. An agent skill guides ontology + resolvers.
+. The ontology is evolvable without coordinated downtime with Coop, as long as changes are backwards-compatible. However, since changing the ontology requires writing or modifying resolvers, it requires a code deploy.
+    ## Success criteria
+    - >=1 users running the DAL in production
+    - The Partial Items API is deprecated, and the DAL replaces it fully
+    ## Open questions
+. How should this interact with Coop's existing Item Types concept? Can we replace that entirely to avoid duplication? That would expand the scope but may be worth doing.
+. How important is ingestion-time enrichment in Coop? Do we actually need it in v1?
+    ## Risks
+. Solidifying an ontology and writing resolvers is an additional engineering lift that orgs may be reluctant to take on.
+        - This means that we need to make it worth it!
+        - We can formalize a longer-term vision: things like enrichment, Osprey integration, custom UIs for moderators in Coop, more flexible investigations in Coop (think Retool).
+. If the DAL is optional infrastructure, there is a risk that orgs will just not use it.
+        - Again, we have to make it worth it via product features.
+. GraphQL makes it easy to create N+1s.
+        - We will have to solve this the standard way using dataloaders, see e.g. [here](https://www.apollographql.com/tutorials/dataloaders-dgs/03-data-loaders-under-the-hood) and [here](https://strawberry.rocks/docs/guides/dataloaders).
+. GraphQL queries can get expensive, e.g. if they write underlying queries with a lot of expensive joins.
+        - We can mitigate this with query complexity/size limits, and great observability out of the box.
+. Source systems may be slow to respond with data.
+        - This can be a real problem, especially in high-volume use cases!
+        - We cannot control these source systems so the best thing we can do here is around observability, so it's easy to see *what* is slow.
+. The DAL may overload source systems if used heavily for T&S.
+        - This is an inherent risk. Orgs will have to manage this, and again we can ensure that it's easy to understand which queries the DAL is executing.
+. Organizations may be reluctant to build APIs that allow broad access to all or most data in their platform, for security reasons.
+        - We can mitigate this via some combination of audit logging, solid authentication + security fundamentals, and (in the future) field-level authorization.
+    ## Future work
+    There are many things we could do. To keep things simple, this initial PRD is tightly scoped. Future work can/should include:
+    - Integration with Osprey for ingest-time enrichment
+    - Ingest-time enrichment for Coop, i.e. fetching more data when an item is first sent in, before running rules
+    - Agentic moderators that can actually explore data via the DAL
+    - Caching queries for performance (with good headers so clients know if the response was cached)
+    - Field-level authorization.
+    ## Definitions
+    - **GraphQL**: The Graph Query Language, an API format originally built by Meta, and now used widely.
+    - **Ontology**: Can also be called a schema. A description of the object types that exist, what properties they have, and the possible relationships between them.

Uh oh!

Add PRD for data abstraction layer #71

Are you sure you want to change the base?

Uh oh!

Add PRD for data abstraction layer #71

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!