Proposal: Unify Backend Resource Management On Texera

[bobbai00]

How Texera manages resources today

Texera manages six kinds of resources: workflows, datasets, models, computing units, environments, and warehouses. By managing a resource I mean three things: creating, reading, updating, and deleting it (CRUD), keeping its versions, and controlling who can access it. Today each kind is managed by its own code: workflows and warehouses in the Dashboard Service, datasets and models in the File Service, computing units in the Computing Unit Managing Service, and environments in the computing unit itself.

Major Limitation: Resources Are Managed Ad-hoc

I think Texera's resource management has one major limitation: resources are managed ad-hoc. Each kind was built on its own, with its own tables, endpoints, and rules, and the codebase has no single definition of a resource that every kind follows. As a result, the same three things are done differently for each kind:

• CRUD is defined per kind: each kind has its own endpoints with its own conventions. For example, a workflow is created with POST /workflow/create, but an environment with POST /pve/db. A dataset is deleted with DELETE /dataset/{did}, but a workflow with POST /workflow/delete.
• Versions are defined per kind: each versioned kind has its own version logic. For example, the Dashboard Service stores a workflow version as a JSON patch on every save that changes the workflow. The File Service instead stores a dataset version as a lakeFS commit when the user asks for one, and keeps a second copy of that code for model versions.
• Access control is defined per kind: each kind has its own way to record who owns a resource and who can access it. For example, ownership is recorded in a workflow_of_user table for workflows, in an owner_uid column for datasets and models, and in a uid column for the other kinds. The rules differ too: a user with read access can remove themselves from a shared workflow, but not from a shared dataset, and environments and warehouses cannot be shared at all.

The codebase itself shows how far the kinds have drifted apart. ResourceTables.scala was added so that the access rules could be written once, but its own comment says that it fits only datasets and models, because workflows record ownership in a separate table and computing units have no public flag.

Consequences: Backend Code Is Duplicated, Hard to Maintain, and Hard to Extend

The current backend codebase suffers from three problems:

• Duplicate code: the same logic is written once per kind. For example, ModelResource.scala repeats 31 of the 32 endpoints of DatasetResource.scala, and the access rules are implemented three times: for workflows, for datasets and models, and for computing units. The database repeats itself in the same way: 17 of its 42 tables are per-kind copies.
• Hard to add new resource kinds: a new kind has to repeat the tables, the endpoints, the version logic, and the access rules. For example, adding models took 12 backend pull requests and 5 database migrations, and 5 of those pull requests were refactors that made dataset code reusable for models. Down the road I expect Texera to add more kinds, such as agents, conversations, and operators, and each of them will pay this cost again.
• Hard to develop new features on top of resources: a feature that works across kinds has to be connected to each kind separately. For example, search maps the columns of each searchable kind by hand into one schema of 28 columns, and the agent service can read and save only workflows, because that is the only kind it has a client for. A new feature that works with every kind, such as an agent or an MCP server, would need the same boilerplate for each kind.

Proposal: Unify the Resource Management

I propose to unify resource management. By unify I mean: designing and implementing common primitives that are shared across all kinds of resources.

Based on current Texera's position, I see three major primitives:

P1: Resource CRUD: every kind has an id, a name, and an owner, and every kind needs the same operations: create, list, read, update, and delete.

P2: Resource version CRUD: a workflow version, a dataset version, and a model version are the same concept, which is an immutable snapshot of a resource. The dataset_version and model_version tables already have the same columns.

P3: Resource access control: every kind needs the same decision, which is whether a user may read or write a resource. The four *_user_access tables already have the same three columns: the user, the resource, and the privilege.

What stays specific to a kind is its own fields and actions, such as executing a workflow or uploading a file to a dataset. Once the common parts exist in one copy, a new kind defines only what is specific to it, and a feature built on the common parts works for every kind.

Implementation Proposal

I see at least two layers whose implementation needs to be refactored: the database schema layer and the API definition layer. Each layer needs its own detailed discussion, so below I only give my proposal for each.

Layer 1: Database Schema Layer

At this layer, my proposal is to store the common parts once, under two supertypes: Resource and Resource Version.

Before: each kind has its own tables for the common parts. Each kind keeps its own relationships to User, ten own and access relationships in total, and each versioned kind keeps its own version table.

After: every kind is a subtype of Resource, which has a single access relationship to User with the levels own, read, and write. Every version is a subtype of Resource Version, keyed by (rid, vno), so one lineage table can link a version to the versions it was derived from. A new kind adds only its own fields under Resource.

Layer 2: API Definition Layer

My proposal at this layer is to give every kind the same endpoints for the three common parts. For each part, the table below lists the endpoints before and after. In the After column, {kind} is the plural name of a kind, such as workflows or datasets, {rid} is the id of a resource, {vno} is the number of a version, and {uid} is the id of a user.

Resource CRUD

Before, each kind spells the same operation in its own way, and some kinds lack an operation. After, every kind has the same five operations.

Operation	Before	After
Create	POST /workflow/create, POST /dataset/create, POST /model/create, POST /computing-unit/create, POST /pve/db, POST /warehouse	POST /{kind}
List	GET /workflow/list, GET /dataset/list, GET /model/list, GET /computing-unit, GET /pve/db, GET /warehouse/status	GET /{kind}
Read	GET /workflow/{wid}, GET /dataset/{did}, GET /model/{mid}, GET /computing-unit/{cuid}; none for environments and warehouses	GET /{kind}/{rid}
Update (rename)	POST /workflow/update/name, POST /dataset/update/name, POST /model/update/name, PUT /computing-unit/{cuid}/rename/{name}, PUT /pve/db/{veid}; none for warehouses	PATCH /{kind}/{rid}
Delete	POST /workflow/delete, DELETE /dataset/{did}, DELETE /model/{mid}, DELETE /computing-unit/{cuid}/terminate, DELETE /pve/db/{veid}, DELETE /warehouse/{whid}	DELETE /{kind}/{rid}

Resource Version CRUD

Before, workflow versions have their own root path and no endpoint to create one, because a version is made whenever a workflow is saved, while dataset and model versions put the operation in the path. After, every versioned kind has the same three operations. A version is numbered within its resource and immutable, so it has no update or delete.

Operation	Before	After
Create	POST /dataset/{did}/version/create, POST /model/{mid}/version/create; none for workflows, where POST /workflow/persist makes a version as a side effect	POST /{kind}/{rid}/versions
List	GET /version/{wid}, GET /dataset/{did}/version/list, GET /model/{mid}/version/list	GET /{kind}/{rid}/versions
Read	GET /version/{wid}/{vid}, GET /dataset/{did}/version/{dvid}/rootFileNodes, GET /model/{mid}/version/{mvid}/rootFileNodes	GET /{kind}/{rid}/versions/{vno}

Resource Access Control

Before, the same sharing endpoints are defined four times, in three services. Environments and warehouses have none, and no endpoint checks access, so each service checks it in its own code. After, every kind is shared and checked in the same way.

Operation	Before	After
List	GET /access/workflow/list/{wid}, plus a copy each for datasets, models, and computing units	GET /{kind}/{rid}/access
Grant or change	PUT /access/workflow/grant/{wid}/{email}/{privilege}, plus the same three copies	PUT /{kind}/{rid}/access/{uid}
Revoke	DELETE /access/workflow/revoke/{wid}/{email}, plus the same three copies	DELETE /{kind}/{rid}/access/{uid}
Check	No endpoint: each service checks access in its own code	POST /access/evaluation

[chenlica]

I support this idea. These resources were developed gradually over time. For each of them, the design and implementation were ad hoc, and we didn't have a "global picture" with a consistent and uniform principle. As we are moving toward to the direction of managing many resources, it becomes very critical to unify them for those benefits mentioned by @bobbai00.

[CaroFernando]

I support this idea. It would directly help #9362, which I’m looking into: loading operators at runtime instead of compiling them into the system. With the common primitives, operators could simply be another resource kind. Publishing an operator would create an immutable version, saved workflows could pin the operator version they were built with, and access control would decide who can use each operator, all without building a separate set of tables and endpoints just for operators. That would let the operator work focus on the engine side (registering and loading operators at runtime) instead of resource management.

[zyratlo]

I support this as well. Right now each resource kind works like its own small system, and like you pointed out, that shows up for users too, since sharing a workflow and sharing a dataset don't behave the same way. One definition of a resource that every kind follows makes sense, especially with agents, conversations, and operators likely coming later.

I'm curious about how resources refer to each other. A workflow already uses datasets, and the kinds you listed would probably reference other resources as well. The lineage table links a version to the versions it was derived from, but I didn't see where "this workflow uses this dataset version" would live. Is that meant to be one of the common primitives too, or left for each kind to handle on its own?

[Xiao-zhen-Liu]

I support this direction. Consolidating ownership, access control, and CRUD removes substantial duplication across services and makes adding new kinds like operators straightforward.

Two suggestions for the detailed design:

1. Decouple metadata and access primitives from the storage layer. A workflow version stores a JSON patch in PostgreSQL, while dataset and model versions commit files to LakeFS. Treating the version record as metadata referencing a kind-specific storage handler keeps the common schema clean.
2. Plan an incremental transition for REST routes. The frontend and agent service rely on endpoints like /workflow/create and /workflow/persist. Supporting legacy routes as adapters prevents breaking client flows while unified endpoints roll out.

[aicam]

I agree with this plan. I believe the codebase is in a good shape from design perspective and AI can easily do this refactor. For example, Amber engine is fully developed in its service and folder, its execution logic, caching and all features are finalized designs and implemented with human written test cases. This makes refactoring much easier because we know for sure that test cases are correct and human written so AI has an easy feedback loop. In this regard, I even support migrating to Go, if we want to refactor, it would be better to do it for a long term goal.

[mengw15]

I agree with this direction. This would be a great opportunity to consolidate and refactor our existing resource management mechanisms. I believe it would make our codebase much cleaner and easier to maintain.

One concern I have is that different resources may have their own resource-specific operations and permissions.

Would it make sense to keep the common CRUD and access control primitives while allowing each resource type to define additional operations and permission checks? This way, we could maintain a consistent foundation while preserving the flexibility needed for different resources.

[eugenegujing]

I support this proposal. Following @aicam's suggestion in #9377, here is what the agent changes proposed there need to read from the backend, so it can be considered in the design.

1. The outcome of a run. To answer "why did my run fail?" or "why is my result empty?", the agent needs the errors, the console messages and the row counts per operator of the user's last execution. Today fatal errors live only in the computing unit's memory, console messages are stored but no endpoint returns them for a past execution, and only row counts can be read. feat(frontend, agent-service): report user runs to the agent #9423 works around this by sending them from the browser. Executing is specific to workflows, so I don't expect the common endpoints to cover it. I am listing it so the design knows the agent needs to read an execution's outcome, with the same access check as the workflow it belongs to.
2. Datasets. To answer "which file should I scan?", the agent needs to list the datasets the user can read and the files of a dataset version. This is change 3 in Proposal: Making the workspace agent useful to a new user by seeing run errors, reading the docs, knowing the datasets #9377. It is not started, and I can build it on the unified endpoints.
3. The same access as the user. The agent service already calls the backend with the user's token, so a resource the user cannot read should be unreadable for the agent too, with no separate rule for the agent.

Also, I don't think my open agent PRs (#9423, #9424) conflict with this effort. Neither changes backend code or adds a call to a backend endpoint.