diff --git a/docs/contributing.md b/docs/contributing.md index 75a67b0bb..72a06e652 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -266,6 +266,94 @@ git checkout develop git pull upstream develop ``` +##### Running your development instance {: #running_your_development_instance } + +There are multiple ways to deploy a local development copy of EasyBuild. This +section presents a method that is more relevant when modifying the framework and +easyblock repositories. + +To deploy a local copy of EasyBuild clone the +[easybuild-framework](https://github.com/easybuilders/easybuild-framework), +[easybuild-easyblocks](https://github.com/easybuilders/easybuild-easyblocks), and +[easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs) +repositories in the same directory, for instance a directory name `EasyBuild`. +After cloning these repositories the content of the `EasyBuild` directory +should be the following. + +``` console +$ tree -L 1 EasyBuild +EasyBuild +├── easybuild-easyblocks +├── easybuild-easyconfigs +└── easybuild-framework + +3 directories, 0 files +``` + +An environment variable with the path to the `EasyBuild` directory simplifies +calls to the development instance of EasyBuild significantly; define this +variable in the following manner. + +``` shell +export EB_DEVEL_ROOT=/path/to/EasyBuild +``` + +To run the development instance, add the paths of the framework and easyblock +repositories to the Python path, and the path of the easyconfig repository to +the robot search path. + +For instance, a command to install `ReFrame-4.3.3.eb` and its dependencies with +the necessary variables defined for the command environment would be the +following: + +``` shell +PYTHONPATH=${EB_DEVEL_ROOT}/easybuild-framework:${EB_DEVEL_ROOT}/easybuild-easyblocks:${PYTHONPATH} EASYBUILD_ROBOT_PATHS=${EB_DEVEL_ROOT}/easybuild-easyconfigs/easybuild/easyconfigs ${EB_DEVEL_ROOT}/easybuild-framework/eb --robot ReFrame-4.3.3.eb +``` + +To avoid adding the variables in the environment of every single command while +maintaining a clean shell environment, an alias can be defined as follows. + +``` shell +alias eb_devel='PYTHONPATH="${EB_DEVEL_ROOT}/easybuild-framework:${EB_DEVEL_ROOT}/easybuild-easyblocks:${PYTHONPATH}" EASYBUILD_ROBOT_PATHS="${EB_DEVEL_ROOT}/easybuild-easyconfigs/easybuild/easyconfigs" ${EB_DEVEL_ROOT}/easybuild-framework/eb' +``` + +With the alias defined, the command to install `ReFrame-4.3.3.eb` and its +dependencies now becomes the following: + +``` shell +eb_devel --robot ReFrame-4.3.3.eb +``` + +??? note "Controlling the robot search path in development instances" + + There is no default value for the [robot search + path](/using-easybuild/#controlling_robot_search_path) in the development + instance. The robot search path is set explicitly using the + `EASYBUILD_ROBOT_PATHS` environment variable. As a result, expect that some + features of the `EASYBUILD_ROBOT_PATHS` and its accompanying command line + option, `--robot-paths` will not work with `eb_devel`, the alias for the + EasyBuild development version. + + For instance, + + - you cannot use `EASYBUILD_ROBOT_PATHS` to set the robot path as it is + overridden in the alias `eb_devel`, and + - you cannot append to the default robot search path, with `--robot-paths`, + as it overrides the `EASYBUILD_ROBOT_PATHS` environment option. + + As a workaround, + + - use `--robot-paths` to override the robot search path set via the alias, and + - use the environment variable `EASYBUILD_ROBOT_PATHS` to access the + the robot search path set by the alias so that you can append to it. + + For instance to append to the development robot search path, use the + following command. + + ``` shell + eb_devel --robot-paths=${EASYBUILD_ROBOT_PATHS}:/path/to/dir_0:/path/to/dir_1 --robot ReFrame-4.3.3.eb + ``` + ### Opening a new pull request {: #contributing_creating_pull_requests } !!! note diff --git a/docs/partial-installations.md b/docs/partial-installations.md index 45d65cfda..402d4fb89 100644 --- a/docs/partial-installations.md +++ b/docs/partial-installations.md @@ -10,9 +10,9 @@ naming scheme). To stop the installation procedure *after* a specific step in the installation procedure, the `-s`/`--stop` command line option can be used; the name of the step must be supplied as an argument. -The following step names are recognized (listed in execution order): `fetch`, `ready`, `source`, `patch`, -`prepare`, `configure`, `build`, `test`, `install`, `extensions`, `package`, `postproc`, -`sanitycheck`, `cleanup`, `module`, `testcases`. +The following step names are recognized (listed in execution order): `fetch`, `ready`, `extract`, `patch`, +`prepare`, `configure`, `build`, `test`, `install`, `extensions`, `postiter`, `postproc`, +`sanitycheck`, `cleanup`, `module`, `permissions`, `package`, `testcases`. Example usage: diff --git a/docs/python-search-path.md b/docs/python-search-path.md new file mode 100644 index 000000000..7a766d581 --- /dev/null +++ b/docs/python-search-path.md @@ -0,0 +1,57 @@ +# Customizing Python search path + +In order to locate Python packages in modules, before v5.0.0, EasyBuild conventionally used the `PYTHONPATH` environment variable. However, this has several issues: + +1. `PYTHONPATH` has highest priority; it prevents users from making a custom virtual environment on top of modules and shadowing packages. +2. Packages are picked up even for incompatible python versions, e.g. an OS installed old python 3.6 will break if packages from 3.12 exists in `PYTHONPATH`. +3. Modules can't have optional dependencies on different Python versions, as PYTHONPATH points directly to the `site-packages` subdirectory. +4. Packages with `pth` files can't work with PYTHONPATH and requires being added to the site dir. + +Unfortunately, Python offers no environment variables to do the correct thing here. +To solve this (initially for Python `multi_deps`) EasyBuild has for a long time supported the use of the custom `EBPYTHONPREFIXES` via a `sitecustomize.py` script for the Python modules we build. +It is included in standard Python installations made with EasyBuild. You can opt out of using `sitecustomize.py` if you have strong technical reasons to avoid it. + +The `sitecustomize.py` script does the correct thing by only considering the correct Python version, and puts Python packages provided by the environment module at the lowest priority, allowing a user venv to override them. + +## Using `--prefer-python-search-path` + +Since v5.0.0 the new global configuration option `--prefer-python-search-path` can be used for EasyBuild to prefer the use of either `PYTHONPATH` or `EBPYTHONPREFIXES`. +For backwards compatibility with existing modules, EasyBuild was unfortunately required to keep `PYTHONPATH` as the default. +Note that the option is just the preferred option, if the package path doesn't follow the standard `lib/pythonY.X/site-packages` format then `PYTHONPATH` must be used. If multi-deps is used, then `EBPYTHONPREFIXES` is required. + +If you wish to switch to `EBPYTHONPREFIXES`, you should also traverse the existing Python bundles you have installed and make sure to rebuild the modules. +Failure to do so might leave you with a few easyconfigs that require shadowing of older package versions not working correctly due to the import priority changing. + +If you are building a new software stack from scratch, you can safely switch this option to `EBPYTHONPREFIXES` and enjoy the benefits. + +## Fixing existing modules + +You can convert existing modules by rebuilding them with `eb --module-only --rebuild ...`. +You can find the modules that use `PYTHONPATH` by e.g. grepping through your modules: + +```bash +grep --include '*.lua' -Rl 'PYTHONPATH.*site-packages' /path/to/modules/all +``` + +If you use the default module naming scheme + +```bash +cd $MODULEPATH +grep -Rl PYTHONPATH */*.lua | grep -v EasyBuild | sed -e 's+/+-+' | sed -e 's+lua$+eb+' | xargs eb --rebuild --module-only +``` + +If done correctly, your old module files will have gone from + +```lua +prepend_path("PYTHONPATH", pathJoin(root, "lib", "python3.12", "site-packages")) +# or +prepend_path("PYTHONPATH", pathJoin(root, "lib/python3.12/site-packages")) +``` + +to + +```lua +prepend_path("EBPYTHONPREFIXES", root) +``` + +Remember to back up all your module files first before rebuilding. diff --git a/mkdocs.yml b/mkdocs.yml index 6cb670066..1b4aa1b02 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -73,6 +73,7 @@ nav: - Cray support: cray-support.md - Customizing EasyBuild via hooks: hooks.md - Including Python modules: including-additional-python-modules.md + - Customizing Python search path: python-search-path.md - Packaging support: packaging-support.md - RPATH support: rpath-support.md - Using external modules: using-external-modules.md