-
Notifications
You must be signed in to change notification settings - Fork 547
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow scripted modules to declare pip requirements and wrap pip dependencies. #7697
Comments
A few things:
|
This was based on a misconception on my part. I think supporting this properly is out of scope for this. It is possible as long as the helper scripts are in sub-directories. The issue is that both This isn't so easy to fix since all factories search all module paths. We could change this so that the CLI module factory searches only CLI paths, and the loadable module factories search only loadable module paths - but this might have unintended consequences. eg. in-source module loading might break. Better probably to leave the module factory search paths as-is, and require helper scripts be placed in sub-directories or pip-installed. This is the same limitation we've always had in loadable modules. It might be nice to fix this or use some more declarative manifest for this, but either way is out of scope for this issue. |
Notes from the weekly hangout today:
Questions about conditional installation. Comments on usability difficulties with CLI modules. There's usually a CLI and a scripted wrapper for a CLI. Maybe instead we just push for scripted modules in general. Maybe a new module type that uses a loadable-style GUI/Widget combined with a CLI-style Logic?
Do we even need CLI modules at all? Originally the intent was to provide an easy on-ramp for people to add functionality. That's been superseded by Python interaction. Do we even want to keep the CLIs long-term? See https://napari.org/stable/plugins/first_plugin.html for another project plugin structure. Really the discriminator is in the environment for the logic of the thing. Need first-class:
"The module logic should almost always be a pip-installable module" Moving module logic into a separate file impacts reload functionality. Not an issue if the logic runs in a separate process (ie CLI) but it requires chain-reload or other unreliable complexity. |
I'm closing this issue and will open a revision, rather than heavily editing this one. Plan to pivot to managing environment for module logic rather than strictly for CLI. Implementation of this will hopefully be more straightforward and extensible in the future. It also avoids the issues around module discovery paths I described in #7697 (comment). Basically we split the module into a "loadable" part - module description and widget - and a "runtime" part - logic. The "loadable" part is very sensitive to import order and environment. Only the runtime part really needs the installable dependencies. Critical questions:
Critical support infrastructure:
Perhaps we install the "runtime" parts as distribution packages and use |
This is tightly related to #7171 and #6913, but since this is more an RFC for a particular solution, so I'm creating a separate issue for discussion specifically on this proposal.
Install scripted modules python dependencies during module discovery to support new use cases:
Safely
import
pip-installed dependencies at the global level withoutcatch ImportError: slicer.util.pip_install(...)
wrappers.Create scripted CLI modules which wrap python package entrypoints.
Supersede
slicer.util.pip_install
usage in scripted loadable modules to help prevent/debug environment breakage from incompatible dependencies.Create scripted CLI modules implemented in multiple Python source files.
Proposed solution
When scripted modules are discovered (ie.
$NAME.py
or$NAME.xml
is discovered) check for a corresponding$NAME-requirements.txt
. When the module is initialized (lazily), check the dependencies and install. Show a summary and confirmation dialog to the user, similar topip install
when-y
is not given. If installation is rejected, do not load the module.For an extension to install
$NAME-requirements.txt
, they should use theRESOURCES
argument toSlicerMacroBuildScriptedModule
. This requires the$NAME-requirements.txt
appear, with that name, in the source tree so that "installing" a module from source via module search path will still discover the file.SlicerMacroBuildScriptedCLI
does not currently have aRESOURCES
argument, so additionally: addSCRIPTS
andRESOURCES
arguments and update it to usectkMacroCompilePythonScript
in the same way asSlicerMacroBuildScriptedModule
. If neitherRESOURCES
norSCRIPTS
are provided, automatically add$NAME.py
and$NAME.xml
to each, respectively, for backwards compatibility.Note - this also allows splitting CLI modules into multiple files and multiple scripts. Only the script with a matching
$NAME.xml
is discovered at runtime.Add a convenience argument
ENTRYPOINT
toslicerMacroBuildScriptedCLI
(with the same semantics as setuptools console script entrypoints) which is mutually exclusive withSCRIPTS
. If provided, at configure time, generate a$NAME.py
which invokes the function identified byENTRYPOINT
.Examples
The current behavior would be unchanged:
Now it would be possible to explicitly list the script and xml for
ScriptedCLI
, or to declare additional scripts/resources:slicerMacroBuildScriptedCLI( NAME Foo SCRIPTS Foo.py ... RESOURCES Foo.xml ... )
Each module could declare a
requirements.txt
to be checked just before module initialization:And the CLI module could wrap a pip-installed entrypoint:
Alternatives considered
requirements.txt
which is renamed to$NAME-requirements.txt
.REQUIREMENTS_FILE
orPIP_DEPENDENCIES
or similar dedicated CMake macro argument.All these suffer issues when developers "install" modules by adding the source tree to Slicer module path. Placing the dependencies in a
$NAME-requirements.txt
forces them to be picked up at module discovery and ensures they are updated as the file is edited.Even if an extension developer does not build the project with CMake, the requirements are sure to be handled for users when the project is built and published via CI. If the developer forgets to declare the requirements file in
RESOURCES
, there would be a clear import error in user logs that the required packages are missing.pip_install
calls in the generated$NAME.py
file forENTRYPOINT
.This only supports the
ENTRYPOINT
mechanism. This invokes dependency resolution on every CLI invocation, not just on CLI instantiation. It is incompatible with installing modules from source via module path.Future Planning
CLI module virtual environments
In the future if CLI modules get their own virtual environment, module initialization must create the virtual environment and install
$NAME-requirements.txt
to that virtual environment, then launch$NAME.py
in the same environment. Behavior would be the same with or without therequirements.txt
.Slicer Constraints File
If a Slicer constraints file is provided to lock versions of built-in packages like
numpy
,vtk
,SimpleITK
, etc. it may be provided withpip -r $NAME-requirements.txt -c path-to-slicer-constraints.txt
. Should not be done for CLI modules if they have their own virtual environments.Dependency resolution errors
Discover all loadable module requirements.txt togethre and invoke pip once with all the requirements listed together? This would detect incompatibilities between modules and mitigate startup time cost - especially if using
uv
. Should not be done for CLI modules if they have their own virtual environments.e.g.
uv install -r Foo-requirements.txt -r Bar-requirements.txt -c path-to-slicer-constraints.txt
; might detect an incompatibility betweenFoo
andBar
dependencies. How should this error be handled? The user should have a means to downgrade or remove one or both of the infringing extensions.Deprecate
slicer.util.pip_install
Once
requirements.txt
is deemed mature, we should show a deprecation warning onslicer.util.pip_install
to encourage extension developers to migrate to this mechanism; else dependency resolution will not be useful.Questions
ENTRYPOINT
would not be supported by install-from-source, since it depends on CMake configure time. Is this a problem?Scripted Loadable modules are initialized during startup - this means all scripted modules dependencies would be resolved at startup. Is this overhead significant? We could mitigate this by:
uv
instead ofpip
-r
arguments. (This would also identify incompatibilities between extensions' dependencies).pip freeze
to avoid dependency resolution in the happy path when everything is already installed.Should pip installation be done at extension installation time? How would from-source modules be handled? How would extension installation time module discovery be handled?
Should
pyproject.toml
be involved in any way? How do we handle from-source installation and module discovery?tomllib
is available to Python in 3.11, or tomli for prior versions. Setuptools supports dynamic dependencies, but this is a beta feature.This technically allows installing multiple
.xml
files in a singleslicerMacroBuildScriptedCLI
call. Should this be explictly forbidden? Should this be explicitly allowed? Should multipleENTRYPOINT
arguments be allowed?I am working on a draft implementation of these changes but it is not yet complete. I invite comments and concerns in the meantime.
cc @jcfr @ebrahimebrahim
The text was updated successfully, but these errors were encountered: