Architecture discussion: how should the frontend and agent-service share backend API contracts? (REST contract drift)

[aicam]

Summary — the contract drift problem

Texera now has two independent TypeScript clients of the same backend REST APIs: the Angular frontend and the agent-service (the LLM agent that operates Texera on a user's behalf). Both call the same Scala backend endpoints (file-service, dashboard-service, workflow-compiling-service, workflow-execution-service), and both hand-redeclare the same contract — URL paths, request shapes, and response DTOs.

That means each endpoint's contract exists in three places that must be kept in sync by hand:

1. The Scala backend (the actual source of truth — JAX-RS resources)
2. frontend (Angular HttpClient, types in common/type/*, URL constants in the dataset service)
3. agent-service (fetch-based clients in agent-service/src/api/*, with its own duplicated interfaces)

Concrete example — listing dataset versions:

• Backend: a JAX-RS resource in file-service
• Frontend: DATASET_VERSION_RETRIEVE_LIST_URL + a DatasetVersion type
• Agent: listDatasetVersions(...) building `/${did}/version/list` + its own DatasetVersion interface in agent-service/src/api/dataset-api.ts

The path string and the DTO are duplicated across the frontend and the agent, and both can silently drift from the Scala backend. When the backend renames a field or changes a route, nothing fails at build time — the two TS clients just quietly go stale until something breaks at runtime. As the agent-service grows to cover more of Texera's surface (datasets, workflow CRUD, execution, operator metadata), this overlap with the frontend keeps growing.

We'd like to align on a standard for how endpoints/contracts are shared across clients before the duplication spreads further.

Important: what does NOT solve this (and why MCP is orthogonal)

This is closely related to #5610 (inline agent tools vs. a dedicated TexeraMCP server), so it's worth being explicit about the relationship, because the two are easy to conflate.

#5610 is about a different axis. It asks how the agent (an LLM) should consume Texera's capabilities — inline function-calling tools vs. MCP. That is an LLM-facing delivery question. The drift problem here is about how two programmatic HTTP clients avoid re-declaring the backend's contract. That is a client-contract question. They are orthogonal layers.

Adopting MCP does not remove this redundancy. If Texera's actions move behind a TexeraMCP server:

• The agent-service stops hand-writing tool wrappers — but the endpoint+DTO knowledge just relocates into the MCP server, which still has to call the REST endpoints. The contract copy moves; it doesn't disappear.
• The frontend is unaffected. The Angular UI is a human-facing client and will never be an MCP client (MCP is a protocol for LLMs/agents). Its duplicate REST contract survives untouched.
• If the TexeraMCP server is hand-written, it becomes a fourth hand-maintained representation of the same endpoints — i.e. MCP can make drift worse, not better.

So MCP is a choice about how the agent talks to capabilities; it is not a contract-sharing strategy. The drift has to be solved one layer below MCP.

Note: the @AgentTool / GET /api/agent-tools manifest idea proposed in #5610 is actually a drift solution in disguise — it's a "backend declares the contract once, clients derive from it" approach (the same family as Option C below). That's the connection worth pursuing: solve the contract layer first, and then the inline-vs-MCP decision in #5610 becomes a cheap downstream choice, because every client (frontend, agent, and any future MCP server) derives from one definition.

A guiding principle

Both the frontend and the agent are clients of the same backend, which is the single source of truth. So the goal is not "share code between the frontend and the agent" — it's single-source the API contract and let every client derive from it.

And a key distinction: share the contract (types + endpoint paths), never the transport (the HTTP-calling code). The frontend uses Angular HttpClient/Observables; the agent uses fetch/Promises. The transport legitimately differs per client; only the request/response shapes and URL paths are identical and worth deduplicating.

Possible solutions

Option A — Status quo: hand-maintained copies

Keep duplicating manually.

• Pros: zero new tooling; each client fully independent.
• Cons: guaranteed drift; runtime-only failures; cost grows with every shared endpoint.

Option B — Shared TypeScript contract package

A small workspace package (e.g. @texera/api-contracts) holding only DTO interfaces + endpoint path builders (datasetVersionListPath(did)), consumed by both TS projects. No HTTP logic.

• Pros: eliminates 2 of the 3 copies (frontend + agent share one definition); low setup cost; can pilot on one API.
• Cons: still hand-maintained against the Scala source (backend can still drift from the package); requires wiring a shared module into the Angular build (the main integration risk).
• Variant (B'): define each contract once as a zod schema, derive the TS type, and validate responses at runtime on both sides. agent-service already uses zod. Adds runtime safety on top of B.

Option C — Contract-first codegen from the backend (the industry standard)

The Scala backend emits a machine-readable contract (OpenAPI), and we generate typed TS clients for both consumers (openapi-typescript / openapi-generator / orval). The backend is JAX-RS (Dropwizard/Jersey), which has first-class OpenAPI support via swagger annotations.

• Pros: eliminates all three copies — the spec is generated from the server, so a backend change breaks both TS clients at build time until updated; this is the standard answer for "same endpoints, multiple clients"; the same spec can later feed a TexeraMCP server so it isn't a 4th copy.
• Cons: highest setup cost (add OpenAPI emission to the Dropwizard services + wire codegen into both builds); we currently emit no spec.
• Connection to Architecture discussion: multiple agent tool calls vs. a dedicated TexeraMCP server for exposing Texera functionality to the agent #5610: this is the generalization of the @AgentTool manifest idea — one backend-declared contract that the frontend, the agent, and any MCP server all derive from.

Suggested direction (for discussion)

A pragmatic two-phase path:

1. Now: Option B/B' — extract a shared api-contracts module for one API (e.g. datasets) as a pilot, to prove out the Angular-build integration on a small surface.
2. Later, if overlap keeps growing: Option C — add OpenAPI emission to the backend and replace the hand-maintained package with generated clients, making the Scala backend the enforced single source for all clients (and feeding Architecture discussion: multiple agent tool calls vs. a dedicated TexeraMCP server for exposing Texera functionality to the agent #5610's tool layer from the same source).

Affected Area

• Workflow UI (frontend)
• agent-service
• Storage / Metadata (file-service / dashboard-service contracts)
• Deployment / Infrastructure (build + codegen tooling)

Questions for the group

1. Do we want to commit to backend-as-single-source (Option C) as the eventual target, or is a shared TS package (Option B) sufficient given the actual overlap?
2. Is there appetite to add OpenAPI emission to the Dropwizard/JAX-RS services? Any prior art or blockers?
3. Should the shared contract layer be designed so it can also feed the agent tool layer from Architecture discussion: multiple agent tool calls vs. a dedicated TexeraMCP server for exposing Texera functionality to the agent #5610 (one source → frontend client + agent tools + optional MCP)?
4. Where do we draw the line on what's worth sharing — just DTOs + paths, or also validation (zod)?

Related: #5610 (inline agent tools vs. TexeraMCP).

[bobbai00]

I suggest we put this discussion on hold, as there are ongoing efforts of refactoring the agent-service to make it well-designed, to name a few: #5753 #5751. After the refactoring, we can re-open the discussion,