Proposal - Supporting user-provided ML models in workflows

[tanishqgandhi1908]

What we're proposing

Let users bring their own trained ML models into Texera and use them in workflows, upload a model, keep versions of it, share it, and run inference on your data — just like datasets work today. Tracking issue: #6494.

What learned from research

Studied how other data/ML platforms handle this, and took a few clear lessons:

• Each resource type gets its own table and its own resolver. Don't force one shared table or one universal path rule to cover everything — give models their own home, separate from datasets.
• Tell resource types apart by a label in the path, then look up the right table from that label (e.g. /datasets/... vs /models/...).
• A model is just a folder of files with versions — the same shape as a dataset. So we can reuse the storage and versioning.
• To run a model, give the worker a real folder of its files (via a mount), because model loaders expect to open files from a directory.

The design

Models are a sibling of datasets: their own tables and their own storage repo, on top of the same storage engine we already use. The path label decides the type (/datasets/... vs /models/...), and a worker mounts the chosen model version to read its files.

Tables and storage

How a model reaches a worker when it runs

The plan

One small step per PR (see sub-tasks under #6494): add the path label → model tables → model storage & lookup → upload/version/access API → UI → use in a workflow → sharing & docs. We start with PyTorch; more frameworks and a no-code inference operator come later.

Questions for the community

1. Path label: we're making the label required (a leading datasets/models segment), with a one-time migration of existing dataset paths. Any concerns with requiring it, versus leaving datasets with no label as the default?
2. Separate model tables (mirroring the dataset tables) vs. reusing the dataset tables with a type column — any preference or downside we're missing?
3. Loading models by mounting the files into the worker — anyone have experience or concerns?

Feedback welcome!

[chenlica]

@tanishqgandhi1908 Thanks for starting the discussion. Our protocol is to create issues for a discussion, and raise PRs for each issue. Can you organize the content in this way?