Mounting datasets and models: three open questions

[aicam]

Background

Since #6866, #6896 and #6897, a Python UDF can take a dataset or model version as a parameter (value=Resource.DATASET or value=Resource.MODEL) and read it as a local, read-only directory. When an execution starts, the engine asks access-control-service to mount each version the workflow names. A mounter DaemonSet on each node then runs GeeseFS against file-service's read-only S3 proxy, which checks the user's read access on every request. The mount reaches the computing unit through mount propagation. Files are fetched on demand, so a UDF can load a multi-GB model or walk a folder of many files without downloading anything first.

I'd like the community's input on three questions about where this goes next.

1. Do we still need the Python library for reading dataset files?

A Python UDF can also read a file with pytexera.DatasetFileDocument("/dataset/<owner>/<name>/<version>/<file>").read_file(). It gets a presigned URL from file-service and downloads the whole file into memory. It handles one file at a time and cannot list a folder. A mounted version covers the same need with plain open() and os.listdir(), and works with any library that takes a path (pandas, torch, PIL, ...).

However, mounting is Kubernetes-only and off by default, so single-node deployments and clusters that enforce the baseline or restricted Pod Security Standards can't use it. Should we keep DatasetFileDocument as the fallback, deprecate it once mounting can be on everywhere, or do something else?

2. What should mounting look like for native operators?

We have wanted operators that work on many files at once for a long time: a folder of images, a sharded CSV, a directory of documents. Today the native scan operators (CSV, JSONL, Arrow, File Scan, ...) each take a single file path and read it through a presigned URL. None of them accepts a folder or a pattern, and none uses mounting. File Scan can read many files only when another operator feeds it file names, or by unpacking a zip. The infrastructure for reading a whole folder now exists, but only Python UDFs use it.

Some possible directions:

• Scan operators take a dataset version (or a folder in it) plus a pattern such as *.csv, and either concatenate the matching files or emit one row per file.
• A "List files" source that outputs the paths in a mounted version for downstream operators to read.
• Operators read mounted files in place instead of copying them to temp files.

What would you want the experience to be, and which operators should come first?

3. Is the current design secure enough?

How it works today:

• The mounter DaemonSet is privileged (FUSE needs it) and listens on a host port on every node. It accepts requests only from access-control-service, which it verifies with a Kubernetes TokenReview.
• Access-control-service requires WRITE access to the computing unit and read access to the dataset or model version, using the token of the computing unit's owner.
• The computing-unit pod itself stays unprivileged. It does need a hostPath volume (its own per-unit folder under the node's mount root) with HostToContainer propagation, and that hostPath is why mounting is opt-in.
• GeeseFS authenticates to the S3 proxy with the user's JWT as its access key. The proxy validates the JWT, checks read access on each request, and re-signs the request with the deployment's LakeFS credentials.

Things I'm unsure about:

• A hostPath in user pods, plus a privileged agent on every node.
• The user's JWT is held by a long-lived GeeseFS process on the node.
• Mounts are released when the pod is deleted, not when an execution ends.
• With the optional disk cache, data stays on the node after access is revoked.

Is this an acceptable trade-off? Would an alternative be better, for example a CSI driver, a FUSE sidecar in each computing unit, or no FUSE at all with a streaming client?

@parshimers @chenlica Let me know if the questions are complete or you want to add anything to them.
@bobbai00 your feedback is welcome here since you developed the library.

[zyratlo]

I prefer keeping DatasetFileDocument as a fallback. My main concern is that a user shouldn't have to know how Texera was deployed to write a workflow that works. Right now mounting is off by default, so if we move UDFs or native operators over to it, the same workflow could run on one deployment and fail on another. Ideally users wouldn't have to pick between the two at all. Could the library use the mount when one exists and fall back to downloading when it doesn't? Then existing workflows keep working, and we can decide on deprecation once mounting is on everywhere.

The same idea applies to native operators. Reading a folder or a set of files matching a pattern is definitely needed, and I don't think it should only work on clusters that have mounting enabled. If listing files can be done without mounting, I'd build the user-facing part on that first and treat mounting as a faster way to read the files when it's available. For which operators come first, I think extending our current scan operators is the best direction. Would we enforce or check the user-defined patterns, or allow them to specify whatever they want?

On question 3, one thing I'd like to understand as a user: if someone loses access to a dataset, how long until they actually can't read it anymore? I understand the proxy checks access on every request, but with the disk cache, files that were already read could still be readable afterward. Mounts living until the pod is deleted seems to make that window longer than it needs to be. Is there a reason mounts can't end when the execution ends?

[chenlica]

@aicam These are three related but different topics. To make the communication efficient, I suggest we create a new discussion just to talk about the user experience for datasets that can be mounted. After we finish that discussion, we can talk about other related topics.

[chenlica]

@parshimers Please chime in.