Proposal: Making the workspace agent useful to a new user by seeing run errors, reading the docs, knowing the datasets

[eugenegujing]

The agent can build and run workflows, but it cannot see the errors from a run the user started, cannot read Texera's own documentation, and does not know which datasets the user has. I want to propose three small, independent changes, each tracked as its own issue and PR. The first two are ready to start, and the third is a sketch I would like feedback on first. This follows up on #9246.

flowchart TB
  subgraph browser [Browser]
    canvas["Workflow canvas<br/>run state, errors, row counts"]
    chat[Agent chat]
  end
  subgraph service [Agent service]
    ctx[Context sent to the model]
    tools[Agent tools]
  end
  docs[("docs/<br/>bundled in the image")]
  files[("file-service<br/>the user's datasets")]

  canvas == "1. report on the user's last run" ==> chat
  chat -- "prompt" --> ctx
  ctx -- "LLM" --> tools
  tools == "2. searchDocs, readDoc" ==> docs
  tools -. "3. listDatasets, listDatasetFiles (later)" .-> files

Loading

The thick arrows are new in changes 1 and 2. The dotted arrow is the sketch in change 3. Everything else exists today.

1. Let the agent see the errors from the user's own run

Today the agent only sees an error when it ran the operator itself through its executeOperator tool. When the user clicks Run and the workflow fails, the agent has no idea. So "why did this fail?" is the one question a new user most wants to ask it, and the one it cannot answer. The system prompt already tells the model how to fix a failing operator in place. What is missing is the input.

The change is on the frontend side. When the user sends a message, the frontend attaches a report on the last run the user started from that browser: the fatal errors, the last error-level console message of each operator (long stack traces truncated), and each operator's state and input and output row counts, as suggested in the comments below. The report is attached once the run has finished, or while it is still running if an operator has already logged an error. A run the agent started itself is not reported. The agent service then shows the report under the matching operator in the dataflow section of the context, the same way compilation errors already appear there. Two short prompt rules ask the model to explain an error in plain language, name the operator, suggest the fix and ask before editing anything, and to read the row counts when the user asks why a result is empty.

sequenceDiagram
  actor U as User
  participant B as Browser
  participant CU as Computing unit
  participant A as Agent service
  participant M as LLM

  U->>B: Click Run
  B->>CU: Execute the workflow
  CU-->>B: Run state, row counts, errors (websocket)
  Note over B: Today this stays in the browser
  U->>B: "Why did this fail?"
  rect rgba(46, 160, 67, 0.15)
    B->>A: Prompt + report on the user's last run
    A->>M: Context with the error and row counts under each operator
  end
  M-->>A: Explanation and a suggested fix
  A-->>B: Answer
  B-->>U: Answer

Loading

Why not read the errors from the backend? Fatal errors are not persisted anywhere. They live in the computing unit's in-memory state and reach the browser over the websocket. The frontend already holds exactly this data, and the AI Fix tab from #8637 reads it. Attaching it to the prompt needs no backend or database change. If fatal errors get persisted later, the source can switch to a REST call without changing the agent side.

This touches the frontend agent and execution services, the agent-service types and context serialization, two prompt rules, and tests. It edits the same file as #9244, so it will land after that.

2. Give the agent the documentation

The agent has no way to look anything up in Texera's own documentation, so a question about how Texera works is answered from the model's general knowledge. Meanwhile, most of the questions @zyratlo's researcher asked were "how do I do X in the UI", and the answers exist in docs/ and on texera.apache.org.

The change is to bundle docs/ into the agent-service image at build time and build an in-memory keyword index over the pages on startup, split by section. Two read-only tools go on top of it. searchDocs(query) returns the best matching sections with a link to the published page, and readDoc(path, section?) returns the text. What the tools return is kept in its own section of the model's context for the rest of the task, with a size limit. One new prompt rule says: for questions about how Texera works, search the docs, read the page, answer with a link, and do not edit the workflow.

flowchart LR
  subgraph build [Build time]
    md["docs/*.md"] -- "COPY docs" --> image[agent-service image]
  end
  subgraph startup [Startup]
    index["In-memory keyword index<br/>one entry per section"]
  end
  subgraph question [A user question]
    q["How do I upload a dataset?"] --> search[searchDocs]
    search --> read[readDoc]
    read --> answer["Answer with a link<br/>to the published page"]
  end
  image --> index
  index --> search
  index --> read

Loading

I want to keep this small. No embeddings, no model download, no new dependency. A plain keyword index over about 500 KB of markdown is enough. Docs are bundled rather than fetched at runtime so the answers match the deployed version and work offline. When #9031 regenerates the operator reference pages, the index picks them up with no change.

One side effect is useful on its own. Running a set of new-user questions against the index also shows which questions have no page at all, and that list can go to #8773.

This is two new files (the index and the tools), one new section in the context, one new prompt rule, one COPY docs line in the Dockerfile, a TEXERA_DOCS_DIR override for local runs, and tests. #4932 is related, since llms.txt could be generated from the same index later.

3. A sketch for letting the agent see the user's datasets

The agent does not know which datasets the user has or what a dataset file path looks like, so it guesses paths and gets them wrong (#4588).

The idea is two read-only tools backed by file-service endpoints that already exist, listDatasets() and listDatasetFiles(dataset, version?). The second returns each file's full path in the form the file scan operators accept, so the agent can fill it in directly. The tools call file-service with the user's own delegated token, so the user's access rules are enforced where they already are. This is discovery only. Reading a dataset from inside a Python UDF is the mount feature (#6866, #6896, #6897) and the questions raised in #9048.

There is prior work here. @aicam prototyped dataset tools (list, get, versions, and more) as part of a standalone MCP server on a fork branch in July, following the discussion in #5610, where the conclusion was that Texera's own actions should stay inline tools. The shape above is the inline version of that prototype's read-only subset. #5918 raised how the agent should call backend endpoints without duplicating the REST contract by hand, and #9395 now proposes one set of endpoints for every kind of resource. I would rather build these tools on that than add another hand-written client.

I am not proposing to start this yet. The agent service does not bind an agent to its owner (#5268) and is getting request authentication in #9043. A tool that lists a user's datasets should land after those, so I would rather agree on the shape now and open the issue once #9043 is in.

What I am asking

• Any suggestions on the three changes, or other ideas for what the agent should be able to see or do for a new user?
• For 2, bundling docs into the image versus fetching from the website at runtime. I prefer bundling for the reasons above.
• For 3, any objection to the shape?

I will open one issue per change and link the PRs here.

[zyratlo]

Thanks for writing this up! This lines up with what I saw with my collaborator, and I think all three are worth doing.

I have one comment on 1. Like I mentioned in #9246, my collaborator ran into runs that finished with no errors but still gave wrong results, like an empty dataset giving an empty result. If only fatal errors and error-level console messages get attached, the agent still can't answer "why is my result empty?" Could the run summary also include something like each operator's output row count from the user's latest run? Then the agent could see that, for example, a file scan produced 0 rows and point at the dataset. The agent already reads row counts for the runs it starts itself, so it's the same information coming from the user's run instead.

For 2, I agree on bundling. I'd expect the list of questions with no docs page to be the most useful part for #8773. A lot of my collaborator's questions weren't about building workflows at all (finding a shared workflow, selecting a file in a dataset, export failing because a dataset wasn't downloadable), and I checked docs/ and none of those are covered right now. Most of the pages are operator references or about installing and contributing, and only a few are written for end users. So for a new user, the docs tool will only help as much as those pages get written. I can share the questions they asked if that helps as a test set.

No objection on 3 right now.

[chenlica]

@eugenegujing Thanks for starting this discussion. If possible, can you draw some visual diagrams to illustrate the idea.

@bobbai00: can you chime in?

[aicam]

I agree with the changes, however I think it may have some conflict with the other effort in #9395. If we unify the endpoints, there is a high chance that agent get access to the execution result, workflow and other artifacts same way user does. I suggest you propose these requirements in that discussion so we consider them when doing design.
In general, I agree with the idea to unify all endpoints user use to do the work with ones agent use. I don't see any difference between user and agent in the way they use the platform.

[eugenegujing]

Thanks everyone for the feedback. Here is where things stand.

The PRs are open:

• feat(frontend, agent-service): report user runs to the agent #9423 reports the user's run to the agent (change 1)
• feat(agent-service): add docs search and read tools #9424 adds the docs search and read tools (change 2)
• feat(frontend): open agent chat links in a new tab #9426 opens links in agent chat messages in a new tab. It is small and separate, and it matters once the agent answers with docs links.

@chenlica I added three diagrams to the post above: an overview, and one flow for each of the first two changes. The PRs carry more detailed ones.

@zyratlo Both of your points are in. #9423 sends each operator's state and its input and output row counts from the user's last run, along with the errors, so the agent can see that a file scan produced 0 rows. On docs, you were right that the tools only help as far as the pages go. One example from testing: tutorials/create-dataset-upload-data.md still tells the user to open an environment tab and click Add Dataset, which the editor no longer has, and the agent repeated that step. The search evaluation in #9424 uses 25 questions I wrote myself, so your collaborator's questions might also be a better test set.

@aicam I agree with unifying the endpoints the user and the agent use, and I will post the agent's requirements in #9395. I don't think the open PRs conflict with that effort. Neither of them changes backend code or adds a call to a backend endpoint:

• feat(frontend, agent-service): report user runs to the agent #9423 changes only the frontend and the agent service. It takes the run's errors and row counts from the browser, because nothing serves them to the agent service today. The agent service only receives a validated report, so once Proposal: Unify Backend Resource Management On Texera #9395 lets it read a run the way the user does, the source can be switched without changing how the agent uses the report.
• feat(agent-service): add docs search and read tools #9424 reads the repository's docs/ folder.
• Change 3 (dataset tools) is the one that depends on Proposal: Unify Backend Resource Management On Texera #9395. It is not started, and I will wait for that design instead of building it on the current file-service endpoints.

[chenlica]

@bobbai00 Please chime in and confirm if these suggestions may conflict the agent improvement you are suggesting.