Design input: should a Python UDF mount multiple datasets, or just one?

[aicam]

We just added dataset mounting for Python UDFs (#6608, closes #6606): a UDF can pick a versioned dataset and read its files from a local path, streamed on demand via GeeseFS through a JWT-authenticated S3 proxy — no download step, version-pinned, and the computing-unit pod stays unprivileged.

Today it's one dataset per UDF. You pick one dataset@version, it gets mounted at /mnt/texera-mounts/<repo>/<commit>, and your code reads it from MOUNTED_DATASET_PATH.

Here is the current flow end to end:
Demo Video

The question

Should we keep this to one dataset per UDF, or let a single UDF mount several?

A concrete case for multiple: a UDF that loads a model from one dataset and reads an evaluation or reference dataset from another. Today that needs two operators or a workaround.

One dataset (what we have)

• Simple picker, a single MOUNTED_DATASET_PATH, easy to document
• Simplest mount lifecycle to reason about
• A UDF that needs two sources cannot do it in one operator

Multiple datasets

• Model + data + config in one UDF, without glue operators
• Needs a list picker in the UI, and a way to expose several paths — named mounts? a MOUNTED_DATASET_PATHS map or JSON?
• The infrastructure already allows it: the mounter keys each mount by repo/commit under the computing unit, so several mounts per pod work today. The open questions are really the UI and how the paths are surfaced to UDF code.

If we go with multiple, how would you want the paths exposed to your code — a JSON map keyed by dataset name, an ordered list, or separate named environment variables? And should there be a sensible upper bound?

Context: #6608 · #6606

---

Disclosure: this post was drafted with Claude (Opus 4.8) and reviewed before posting.

[aglinxinyuan]

I don't see the difference, and I don't get why do we need complex design for Multiple datasets.
The path for all datasets are unqiue. We can just use the dataset path in the code.

[aglinxinyuan]

Had a discussion; @aicam can provide an update once you have a new design.

[aicam]

@aglinxinyuan thanks for the feedback, the new UX is now added to the discussion description

[chenlica]

In terms of user experience, since we have the capability for a user to mount a single dataset in a UDF, we should allow multiple datasets in a single UDF to support general cases.

[aicam]

The new UX support that.