Deprecate and remove per-user Python Virtual Environments (PVE) in favor of curated images?

[aicam]

Context

Texera currently has two mechanisms for providing the Python packages a Python UDF needs at execution time:

1. Per-user Python Virtual Environments (PVE). A user creates named environments from the dashboard ("Environments" tab) and installs packages into them; a Python UDF can then select an environment instead of the default one. Implemented in amber/.../web/resource/pythonvirtualenvironment/ (PveManager, PveResource, PveWebsocketResource), the virtual-environment frontend service and user-venv dashboard page, and the "Default Python Environment" option on the Python UDF operators.
2. Curated computing-unit images. An admin registers and validates a container image, and users select it when creating a computing unit. Implemented in CuratedImageResource and the admin cu-image UI.

These overlap in purpose: both exist so a workflow can run against a Python environment that has the packages it needs.

Proposal

Consider deprecating and eventually removing PVE, standardizing on curated images as the way to provide custom Python dependencies.

Reasons this may be worth doing:

• Two overlapping subsystems to maintain, document and test (PVE is roughly 900 lines of backend, ~1,500 lines of frontend, and ~2,300 lines of tests).
• Curated images produce reproducible, pre-built environments rather than packages installed ad hoc at runtime.
• Fewer moving parts in the execution path (no per-run package installation step).

Open questions / trade-offs

• Self-service gap. PVE is self-service; curated images are admin-managed. Do we want a self-service story before removing PVE, or is admin curation acceptable (and arguably preferable) for most deployments?
• Deployment coverage. Curated images are oriented around Kubernetes. What is the recommended path for single-node users who rely on PVE today?
• Migration. Are there active PVE users who would need a migration path or a deprecation notice period? Note the "Default Python Environment" / environment-name fields are part of saved workflow JSON, so removal touches stored workflows.
• Scope. Removal would span the backend resources, the dashboard "Environments" page, the operator-level option, and the virtual_environments table.

Request for comments

Is standardizing on curated images the right direction? If so, what is the minimum we would want in place (docs, a single-node story, a deprecation notice) before deprecating PVE? If not, how should the two features relate going forward?

[kunwp1]

I am concerned that this approach is not scalable because it gives a lot of burden to the admin. For PVE, users have a flexibility to create their own environment, but sounds like your suggestion relies on admin to create all the environments for the users.

[bobbai00]

I also think removing it and relying on Admin is not scalable. Ideally,

• Admin should provide a bare-minimal image to users
• To add on bespoke features like python packages, users should specify what they want and be able to install and manage them.
  I think PVE serves this purpose. Regarding whether it is implemented properly or designed properly, thats a separate topic

[chenlica]

Can we draw a conclusion? If there is no enough support, maybe we can close it?