diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index f4ccc74d6..835acf593 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -38,4 +38,5 @@ jobs: pyflakes shpc/main/modules pyflakes shpc/main/container/base.py pyflakes shpc/main/container/podman.py + pyflakes shpc/main/container/docker.py pyflakes shpc/main/container/singularity.py diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 2922f017f..9df786c44 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -73,6 +73,10 @@ jobs: shpc config set container_tech:${{ matrix.container_tech }} shpc config set module_sys:${{ matrix.module }} shpc config set enable_tty:false + + printf "\n\shpc test ============================================\n" + shpc test python:3.9.5-alpine + shpc install python:3.9.5-alpine module use ./modules @@ -96,7 +100,8 @@ jobs: cat test_output grep --quiet 'Python 3.9.5' test_output rm test_output - + shpc uninstall --force python:3.9.5-alpine + - name: Run python module tests (tcsh) shell: tcsh -e {0} run: | @@ -114,6 +119,7 @@ jobs: shpc config set container_tech:${{ matrix.container_tech }} shpc config set module_sys:${{ matrix.module }} shpc config set enable_tty:false + shpc install python:3.9.5-alpine module use ./modules @@ -137,3 +143,4 @@ jobs: cat test_output grep --quiet 'Python 3.9.5' test_output rm test_output + shpc uninstall --force python:3.9.5-alpine diff --git a/CHANGELOG.md b/CHANGELOG.md index 17cd9758b..c7a33d934 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,13 @@ and **Merged pull requests**. Critical items to know are: The versions coincide with releases on pip. Only major versions will be released as tags on Github. ## [0.0.x](https://github.scom/singularityhub/singularity-hpc/tree/master) (0.0.x) + - Add test for docker and podman (0.0.28) + - namespace as format string for command named renamed to repository + - shpc test/uninstall should be run for all tests + - bug with uninstall + - adding user and group ids to docker and podman commands + - remove conflict with entire module name, should only be for aliases + - fix documentation of commands in Lua module template - Cleanup of module files and docker requirement bugfix (0.0.27) - Docker support (0.0.26) - added an enable_tty setting to allow disabling add -t in recipes diff --git a/docs/getting_started/user-guide.rst b/docs/getting_started/user-guide.rst index 821e7fd06..48398a7f3 100644 --- a/docs/getting_started/user-guide.rst +++ b/docs/getting_started/user-guide.rst @@ -139,7 +139,7 @@ A summary table of variables is included below, and then further discussed in de - if defined, add to module script to load this Singularity module first - null * - module_name - - Format string for module commands exec,shell,run (not aliases) can include ``{{ registry }}``, ``{{ namespace }}``, ``{{ tool }}`` and ``{{ version }}`` + - Format string for module commands exec,shell,run (not aliases) can include ``{{ registry }}``, ``{{ repository }}``, ``{{ tool }}`` and ``{{ version }}`` - ``{{ tool }}`` * - bindpaths - string with comma separated list of paths to binds. If set, expored to SINGULARITY_BINDPATH @@ -305,12 +305,12 @@ A container identifier is parsed as follows: .. code-block:: console # quay.io /biocontainers/samtools:latest - # / / / + # / / / So by default, we use tool because it's likely closest to the command that is wanted. But let's say you had two versions of samtools - the namespaces would conflict! You -would want to change your format string to ``{{ namespace }}-{{ tool }}`` to be +would want to change your format string to ``{{ repository }}-{{ tool }}`` to be perhaps "biocontainers-samtools-exec" and "another-samtools-exec." If you change the format string to ``{{ tool }}-{{ version }}`` you would see: diff --git a/paper/paper.md b/paper/paper.md index 94d1416e8..b3cd1f8dd 100644 --- a/paper/paper.md +++ b/paper/paper.md @@ -25,14 +25,14 @@ bibliography: paper.bib Portability and reproducibility of complex software stacks is essential for researchers to perform their work. High Performance Computing (HPC) environments add another level of complexity, where possibly conflicting dependencies must co-exist. Although container technologies like Singularity [@Kurtzer2017-xj] make it possible to "bring your own environment," without any form of central strategy to manage containers, researchers that seek reproducibility via using containers are tasked with managing their own container collection, often not taking care to ensure that a particular digest or version is used. The reproducibility of the work is at risk, as they cannot easily install and use containers, nor can they share their software with others. -Singularity Registry HPC (shpc) is the first of its kind to provide an easy means for a researcher to add their research software for sharing and collaboration with other researchers to an existing collection of over 200 popular scientific libraries [@da2017biocontainers; @noauthor_undated-kp, @gorgolewski2017bids; @gamblin2015spack; @autamus]. The software installs these containers as environment modules [@McLay2011-wu] that are easy to use and read documentation for, and exposes aliases for commands in the container that the researcher can add to his or her pipeline without thinking about complex interactions with a container. The simple addition of an entry to the registry maintained by shpc comes down to adding a yaml file, and after doing this, another researcher can easily install the same software, down to the digest, to reproduce the original work. +Singularity Registry HPC (shpc) is the first of its kind to provide an easy means for a researcher to add their research software for sharing and collaboration with other researchers to an existing collection of over 200 popular scientific libraries [@da2017biocontainers; @noauthor_undated-kp, @gorgolewski2017bids; @gamblin2015spack; @autamus]. The software installs containers as environment modules [@McLay2011-wu] that are easy to use and read documentation for, and exposes aliases for commands in the container that the researcher can add to his or her pipeline without thinking about complex interactions with a container. The simple addition of an entry to the registry maintained by shpc comes down to adding a yaml file, and after doing this, another researcher can easily install the same software, down to the digest, to reproduce the original work. ## Statement of Need Using environment modules [@McLay2011-wu] on HPC clusters is a common trend. Although writing the recipes can be complex, it's a fairly common practice for cluster administrators to provide -a set of natively installed recipes for their users [@noauthor_undated-bt], or for researchers to develop and deploy their own software via containers. Even well-known package managers like Spack [@noauthor_undated-ae] and EasyBuild [@noauthor_undated-dj] expose software as modules. However, these package manager approaches don't always ensure reproducibility, or ease of development for the researcher. They typically require relying on some subset of system software, the underlying operating system, or even making changes to the system, which is not under the researcher's control. Although using containers in this context has been discussed previously [@noauthor_undated-rj; @noauthor_undated-rc], the majority of these approaches and tools do not make the process of developing and installing container modules easy. The single researcher must either convince a cluster administrator to install dependencies needed for their software, or build a container and manually move and interact with it on the cluster. All of these small challenges come together to make it harder for a researcher to develop and manage their own software, and subsequently to share their approach to reproduce the work. Using Singularity containers installed via Singularity Registry HPC offers a solution to this challenge. The only requirement is the Singularity software, and writing a simple configuration file for the registry. By clearly defining commands, and pinning exact versions of scientific software, researchers on high performance computing +a set of natively installed recipes for their users [@noauthor_undated-bt], or for researchers to develop and deploy their own software via containers. Even well-known package managers like Spack [@noauthor_undated-ae] and EasyBuild [@noauthor_undated-dj] expose software as modules. However, these package manager approaches don't always ensure reproducibility, or ease of development for the researcher. They typically require relying on some subset of system software, the underlying operating system, or even making changes to the system, which is not under the researcher's control. Although using containers in this context has been discussed previously [@noauthor_undated-rj; @noauthor_undated-rc], the majority of these approaches and tools do not make the process of developing and installing container modules easy. The single researcher must either convince a cluster administrator to install dependencies needed for their software, or build a container and manually move and interact with it on the cluster. All of these small challenges come together to make it harder for a researcher to develop and manage their own software, and subsequently to share their approach to reproduce the work. Using Singularity, Podman, or other container technologies installed via Singularity Registry HPC offers a solution to this challenge. The only requirement is the container technology software, and writing a simple configuration file for the registry. By clearly defining commands, and pinning exact versions of scientific software, researchers on high performance computing clusters can have more confidence in the reproducibility of their work [@Santana-Perez2015-wo; @Boettiger2014-cz; @Wandell2015-yt]. ## Usage diff --git a/shpc/client/config.py b/shpc/client/config.py index 0e275dea8..164ee2295 100644 --- a/shpc/client/config.py +++ b/shpc/client/config.py @@ -10,8 +10,6 @@ def main(args, parser, extra, subparser): from shpc.main import get_client - cli = get_client(quiet=args.quiet, settings_file=args.settings_file) - # If nothing provided, show help if not args.params: print(subparser.format_help()) @@ -20,6 +18,11 @@ def main(args, parser, extra, subparser): # The first "param" is either set of get command = args.params.pop(0) + validate = True if not command == "edit" else False + cli = get_client( + quiet=args.quiet, settings_file=args.settings_file, validate=validate + ) + # For each new setting, update and save! if command == "edit": return cli.settings.edit() diff --git a/shpc/main/__init__.py b/shpc/main/__init__.py index 95c5f86b8..12c60c861 100644 --- a/shpc/main/__init__.py +++ b/shpc/main/__init__.py @@ -21,9 +21,10 @@ def get_client(quiet=False, **kwargs): """ # The name of the module module = kwargs.get("module") + validate = kwargs.get("validate", True) # Load user settings to add to client, and container technology - settings = Settings(kwargs.get("settings_file")) + settings = Settings(kwargs.get("settings_file"), validate) container = kwargs.get("container_tech") or settings.container_tech # Use the user provided module OR the default diff --git a/shpc/main/client.py b/shpc/main/client.py index 5b26e96bf..86f4972f1 100644 --- a/shpc/main/client.py +++ b/shpc/main/client.py @@ -141,11 +141,7 @@ def cleanup(tmpdir): # Test all tags (this could be subsetted) for tag in tags: - # Install the recipe - sif = self.install(module_name, tag) - - # Assert that the container exists - assert os.path.exists(sif) + image = self.install(module_name, tag) # Do we want to test loading? if not skip_module and hasattr(self, "_test"): @@ -157,27 +153,17 @@ def cleanup(tmpdir): # Do we want to test the test commands? if test_commands and config.test: utils.write_file(test_file, config.test) - command = ["singularity", "exec", sif, "/bin/bash", test_file] - result = utils.run_command(command) - - # We can't run on incompatible hosts - if ( - "the image's architecture" in result["message"] - and result["return_code"] != 0 - ): - logger.warning( - "Cannot run test for container with incompatible architecture: %s" - % result["message"] - ) - elif result["return_code"] != 0: + return_code = self.container.test_script(image, test_file) + if return_code != 0: cleanup(tmpdir) logger.exit("Test of %s was not successful." % module_name) # Test the commands if not test_exec: continue + for alias in config.get_aliases(): - result = self._container.client.execute(sif, alias["command"]) + result = self.client.execute(image, alias["command"]) # Cleanup the test install if stage: diff --git a/shpc/main/container/base.py b/shpc/main/container/base.py index 9b21245ad..734ca4bbc 100644 --- a/shpc/main/container/base.py +++ b/shpc/main/container/base.py @@ -20,7 +20,7 @@ class ContainerName: def __init__(self, raw): self.raw = raw self.registry = None - self.namespace = None + self.repository = None self.tool = None self.version = None self.digest = None @@ -90,13 +90,17 @@ def container_dir(self, name): return os.path.join(self.settings.module_base, name) return os.path.join(self.settings.container_base, name) - def guess_tag(self, module_name): - """If a user asks for a name without a tag, try to figure it out.""" + def guess_tag(self, module_name, allow_fail=False): + """ + If a user asks for a name without a tag, try to figure it out. + """ if ":" in module_name: return module_name tags = os.listdir(os.path.join(self.settings.module_base, module_name)) - if not tags: + if not tags and allow_fail: logger.exit("%s does not have any tags installed." % module_name) + elif (tags or len(tags) > 1) and allow_fail: + return elif len(tags) > 1: logger.exit( "Multiple tags found for %s: %s." % (module_name, ", ".join(tags)) diff --git a/shpc/main/container/docker.py b/shpc/main/container/docker.py index 16ef4dac2..b65da498c 100644 --- a/shpc/main/container/docker.py +++ b/shpc/main/container/docker.py @@ -99,9 +99,13 @@ def get(self, module_name): Determine if a container uri exists. """ # If no module tag provided, try to deduce from install tree - module_name = self.guess_tag(module_name) + full_name = self.guess_tag(module_name, allow_fail=True) - uri = self.add_registry(module_name) + # The user already provided a tag + if not full_name: + full_name = module_name + + uri = self.add_registry(full_name) # If there isn't a tag in the name, add it back if ":" not in uri: uri = ":".join(uri.rsplit("/", 1)) @@ -132,6 +136,25 @@ def check(self, module_name, config): % (module_name, config.latest.name) ) + def test_script(self, image, test_script): + """ + Given a test file, run it and respond accordingly. + """ + command = [ + "docker", + "run", + "-i", + "--entrypoint", + "/bin/bash", + "-t", + image, + test_script, + ] + result = shpc.utils.run_command(command) + + # Return code + return result["return_code"] + def install( self, module_path, @@ -190,7 +213,7 @@ def install( name=name, tool=parsed_name.tool, registry=parsed_name.registry, - namespace=parsed_name.namespace, + repository=parsed_name.repository, envfile=self.settings.environment_file, command=self.command, tty=self.settings.enable_tty, diff --git a/shpc/main/container/singularity.py b/shpc/main/container/singularity.py index 61344fc96..e69c632d6 100644 --- a/shpc/main/container/singularity.py +++ b/shpc/main/container/singularity.py @@ -179,7 +179,7 @@ def install( name=name, tool=parsed_name.tool, registry=parsed_name.registry, - namespace=parsed_name.namespace, + repository=parsed_name.repository, envfile=self.settings.environment_file, ) shpc.utils.write_file(module_path, out) @@ -325,3 +325,23 @@ def _pull_github(self, uri, dest=None): dest = os.path.basename(url) name = os.path.basename(dest) return self.client.pull(url, name=name, pull_folder=os.path.dirname(dest)) + + def test_script(self, image, test_script): + """ + Given a test file, run it and respond accordingly. + """ + command = ["singularity", "exec", image, "/bin/bash", test_script] + result = shpc.utils.run_command(command) + + # We can't run on incompatible hosts + if ( + "the image's architecture" in result["message"] + and result["return_code"] != 0 + ): + logger.warning( + "Cannot run test for incompatible architecture: %s" % result["message"] + ) + return 0 + + # Return code + return result["return_code"] diff --git a/shpc/main/modules/__init__.py b/shpc/main/modules/__init__.py index 71f7ff5ae..f934313b8 100644 --- a/shpc/main/modules/__init__.py +++ b/shpc/main/modules/__init__.py @@ -84,6 +84,7 @@ def uninstall(self, name, force=False): """ # The name can either be a folder or an install directory name = self.add_namespace(name) + # folder paths do not have : name = name.replace(":", os.sep) module_dir = os.path.join(self.settings.module_base, name) diff --git a/shpc/main/modules/templates/docker.lua b/shpc/main/modules/templates/docker.lua index 0c020b725..db6551417 100644 --- a/shpc/main/modules/templates/docker.lua +++ b/shpc/main/modules/templates/docker.lua @@ -16,27 +16,27 @@ Container: Commands include: - {|module_name|}-run: - {{ command }} run --rm -i{% if tty %}t{% endif %} {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} -v ${PWD} -w ${PWD} {% endif %} + {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} -v ${PWD} -w ${PWD} {% endif %} "$@" - {|module_name|}-shell: - {{ command }} run --rm -i{% if tty %}t{% endif %} {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} -v ${PWD} -w ${PWD} {% endif %} + {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} --entrypoint {{ shell }} -v ${PWD} -w ${PWD} {% endif %} - {|module_name|}-exec: - {{ command }} run -i{% if tty %}t{% endif %} --rm --entrypoint {{ shell }} {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v ${PWD} -w ${PWD} "$@" + {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint "" {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v ${PWD} -w ${PWD} "$@" - {|module_name|}-inspect: {{ command }} inspect {% if aliases %}{% for alias in aliases %} - {{ alias.name }}: - {{ command }} run --rm -i{% if tty %}t{% endif %} --entrypoint {{ alias.entrypoint }} {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if alias.options %}{{ alias.options }} {% endif %} -v ${PWD} -w ${PWD} "{{ alias.args }}" + {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint {{ alias.entrypoint }} {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if alias.options %}{{ alias.options }} {% endif %} -v ${PWD} -w ${PWD} "{{ alias.args }}" {% endfor %}{% endif %} For each of the above, you can export: - - DOCKER_OPTS: to define custom options for {{ command }} - - DOCKER_COMMAND_OPTS: to define custom options for the command + - PODMAN_OPTS: to define custom options for {{ command }} + - PODMAN_COMMAND_OPTS: to define custom options for the command ]]) {% if podman_module %}load("{{ podman_module }}"){% endif %} -setenv ("DOCKER_OPTS", "") -setenv ("DOCKER_COMMAND_OPTS", "") +setenv ("PODMAN_OPTS", "") +setenv ("PODMAN_COMMAND_OPTS", "") -- we probably don't need this local MODULEPATH="{{ module_dir }}" @@ -44,17 +44,17 @@ local MODULEPATH="{{ module_dir }}" -- interactive shell to any container, plus exec for aliases local containerPath = '{{ image }}' -local shellCmd = "{{ command }} ${DOCKER_OPTS} run -i{% if tty %}t{% endif %} ${DOCKER_COMMAND_OPTS} --entrypoint {{ shell }} --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v ${PWD} -w ${PWD} " .. containerPath +local shellCmd = "{{ command }} ${PODMAN_OPTS} run -i{% if tty %}t{% endif %} ${PODMAN_COMMAND_OPTS} -u `id -u`:`id -g` --rm --entrypoint {{ shell }} {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v ${PWD} -w ${PWD} " .. containerPath -- execCmd needs entrypoint to be the executor -local execCmd = "{{ command }} ${DOCKER_OPTS} run ${DOCKER_COMMAND_OPTS} -i{% if tty %}t{% endif %} --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v ${PWD} -w ${PWD} " -local runCmd = "{{ command }} ${DOCKER_OPTS} run ${DOCKER_COMMAND_OPTS} -i{% if tty %}t{% endif %} --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v ${PWD} -w ${PWD} " .. containerPath -local inspectCmd = "{{ command }} ${DOCKER_OPTS} inspect ${DOCKER_COMMAND_OPTS} " .. containerPath +local execCmd = "{{ command }} ${PODMAN_OPTS} run ${PODMAN_COMMAND_OPTS} -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v ${PWD} -w ${PWD} " +local runCmd = "{{ command }} ${PODMAN_OPTS} run ${PODMAN_COMMAND_OPTS} -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v ${PWD} -w ${PWD} " .. containerPath +local inspectCmd = "{{ command }} ${PODMAN_OPTS} inspect ${PODMAN_COMMAND_OPTS} " .. containerPath -- set_shell_function takes bashStr and cshStr set_shell_function("{|module_name|}-shell", shellCmd, shellCmd) -- conflict with modules with the same name -conflict(myModuleName(){% if aliases %}{% for alias in aliases %},"{{ alias.name }}"{% endfor %}{% endif %}) +conflict(myModuleName(){% if aliases %}{% for alias in aliases %}{% if alias.name != name %},"{{ alias.name }}"{% endif %}{% endfor %}{% endif %}) -- exec functions to provide "alias" to module commands {% if aliases %}{% for alias in aliases %} diff --git a/shpc/main/modules/templates/docker.tcl b/shpc/main/modules/templates/docker.tcl index b4f5ed952..a950cfe5e 100644 --- a/shpc/main/modules/templates/docker.tcl +++ b/shpc/main/modules/templates/docker.tcl @@ -15,26 +15,26 @@ proc ModulesHelp { } { puts stderr " - {{ image }}" puts stderr "Commands include:" puts stderr " - {|module_name|}-run:" - puts stderr " {{ command }} run -i{% if tty %}t{% endif %} --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v . -w . " + puts stderr " {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v . -w . " puts stderr " - {|module_name|}-shell:" - puts stderr " {{ command }} run -i{% if tty %}t{% endif %} --rm --entrypoint {{ shell }}{% if envfile %} --env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v . -w . " + puts stderr " {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint {{ shell }}{% if envfile %} --env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v . -w . " puts stderr " - {|module_name|}-exec:" - puts stderr " {{ command }} run -i{% if tty %}t{% endif %} --rm --entrypoint {{ shell }}{% if envfile %} --env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v . -w . $*" + puts stderr " {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint \"\" {% if envfile %} --env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v . -w . $*" puts stderr " - {|module_name|}-inspect:" puts stderr " {{ command }} inspect " {% if aliases %}{% for alias in aliases %} puts stderr " - {{ alias.name }}:" - puts stderr " {{ command }} run -i{% if tty %}t{% endif %} --entrypoint {{ alias.entrypoint }} {% if envfile %}--envfile {{ module_dir }}/{{ envfile }} {% endif %}{% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if alias.options %}{{ alias.options }} {% endif %} -v . -w . {{ alias.args }}" + puts stderr " {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --entrypoint {{ alias.entrypoint }} {% if envfile %}--envfile {{ module_dir }}/{{ envfile }} {% endif %}{% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if alias.options %}{{ alias.options }} {% endif %} -v . -w . {{ alias.args }}" {% endfor %}{% endif %} puts stderr "For each of the above, you can export:" - puts stderr " - DOCKER_OPTS: to define custom options for {{ command }}" - puts stderr " - DOCKER_COMMAND_OPTS: to define custom options for the command" + puts stderr " - PODMAN_OPTS: to define custom options for {{ command }}" + puts stderr " - PODMAN_COMMAND_OPTS: to define custom options for the command" } # Environment -setenv DOCKER_OPTS "" -setenv DOCKER_COMMAND_OPTS "" +setenv PODMAN_OPTS "" +setenv PODMAN_COMMAND_OPTS "" # Variables @@ -50,18 +50,18 @@ set helpcommand "This module is a {{ docker }} container wrapper for {{ name }} {% if labels %}{% for key, value in labels.items() %}set {{ key }} {{ value }} {% endfor %}{% endif %} -# conflict with modules with the same name +# conflict with modules with the same alias name conflict {{ name }} {% if aliases %}{% for alias in aliases %}{% if alias.name != name %}conflict {{ alias.name }}{% endif %} {% endfor %}{% endif %} # interactive shell to any container, plus exec for aliases -set shellCmd "{{ command }} \${DOCKER_OPTS} run \${DOCKER_COMMAND_OPTS} --rm -i{% if tty %}t{% endif %} --entrypoint {{ shell }} {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v $workdir -w $workdir ${containerPath}" +set shellCmd "{{ command }} \${PODMAN_OPTS} run \${PODMAN_COMMAND_OPTS} -u `id -u`:`id -g` --rm -i{% if tty %}t{% endif %} --entrypoint {{ shell }} {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v $workdir -w $workdir ${containerPath}" # execCmd needs entrypoint to be the executor -set execCmd "{{ command }} \${DOCKER_OPTS} run -i{% if tty %}t{% endif %} \${DOCKER_COMMAND_OPTS} --rm {% if envfile %} --env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} -v $workdir -w $workdir {% endif %} " -set runCmd "{{ command }} \${DOCKER_OPTS} run -i{% if tty %}t{% endif %} \${DOCKER_COMMAND_OPTS} --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v $workdir -w $workdir ${containerPath}" -set inspectCmd "{{ command }} \${DOCKER_OPTS} inspect ${containerPath}" +set execCmd "{{ command }} \${PODMAN_OPTS} run -i{% if tty %}t{% endif %} \${PODMAN_COMMAND_OPTS} -u `id -u`:`id -g` --rm {% if envfile %} --env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} -v $workdir -w $workdir {% endif %} " +set runCmd "{{ command }} \${PODMAN_OPTS} run -i{% if tty %}t{% endif %} \${PODMAN_COMMAND_OPTS} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v $workdir -w $workdir ${containerPath}" +set inspectCmd "{{ command }} \${PODMAN_OPTS} inspect ${containerPath}" # set_shell_function takes bashStr and cshStr set-alias {|module_name|}-shell "${shellCmd}" diff --git a/shpc/main/modules/templates/docs.md b/shpc/main/modules/templates/docs.md index 8523920ba..834f45ea1 100644 --- a/shpc/main/modules/templates/docs.md +++ b/shpc/main/modules/templates/docs.md @@ -122,8 +122,8 @@ inspect aliases. For anycommands above, you can export: - SINGULARITY_OPTS: to define custom options for singularity (e.g., --debug) - SINGULARITY_COMMAND_OPTS: to define custom options for the command (e.g., -b) - - DOCKER_OPTS: to define custom options for podman or docker - - DOCKER_COMMAND_OPTS: to define custom options for the command + - PODMAN_OPTS: to define custom options for podman or docker + - PODMAN_COMMAND_OPTS: to define custom options for the command
diff --git a/shpc/main/modules/templates/singularity.lua b/shpc/main/modules/templates/singularity.lua index b70969d71..139a74d84 100644 --- a/shpc/main/modules/templates/singularity.lua +++ b/shpc/main/modules/templates/singularity.lua @@ -58,7 +58,7 @@ local inspectCmd = "singularity ${SINGULARITY_OPTS} inspect ${SINGULARITY_COMMAN set_shell_function("{|module_name|}-shell", shellCmd, shellCmd) -- conflict with modules with the same name -conflict(myModuleName(){% if aliases %}{% for alias in aliases %},"{{ alias.name }}"{% endfor %}{% endif %}) +conflict(myModuleName(){% if aliases %}{% for alias in aliases %}{% if alias.name != name %},"{{ alias.name }}"{% endif %}{% endfor %}{% endif %}) -- exec functions to provide "alias" to module commands {% if aliases %}{% for alias in aliases %} diff --git a/shpc/main/modules/templates/singularity.tcl b/shpc/main/modules/templates/singularity.tcl index 946c2bcbd..c02f332d9 100644 --- a/shpc/main/modules/templates/singularity.tcl +++ b/shpc/main/modules/templates/singularity.tcl @@ -56,7 +56,7 @@ set helpcommand "This module is a singularity container wrapper for {{ name }} v {% if labels %}{% for key, value in labels.items() %}set {{ key }} {{ value }} {% endfor %}{% endif %} -# conflict with modules with the same name +# conflict with modules with the same alias name conflict {{ name }} {% if aliases %}{% for alias in aliases %}{% if alias != name %}conflict {{ alias.name }}{% endif %} {% endfor %}{% endif %} diff --git a/shpc/main/settings.py b/shpc/main/settings.py index 72033aa8a..cf5fa73af 100644 --- a/shpc/main/settings.py +++ b/shpc/main/settings.py @@ -141,12 +141,13 @@ class Settings(SettingsBase): a dictionary-like class with extra functions. """ - def __init__(self, settings_file): + def __init__(self, settings_file, validate=True): """ Create a new settings object, which requires a settings file to load """ self.load(settings_file) - self.validate() + if validate: + self.validate() # Set an updated time, in case it's written back to file self._settings["updated_at"] = datetime.now().strftime("%Y-%m-%dT%H:%M:%SZ") diff --git a/shpc/settings.yml b/shpc/settings.yml index f2fbed6a0..82279a848 100644 --- a/shpc/settings.yml +++ b/shpc/settings.yml @@ -3,7 +3,7 @@ # currently specific to singularity, but can be extended to other technologies. # set a default module system -# Currently only lmod is supported. To request an additional system, +# Currently lmod and tcl are supported. To request an additional system, # please open an issue https://github.com/singularityhub/singularity-hpc module_sys: lmod @@ -21,7 +21,7 @@ registry: $root_dir/registry module_base: $root_dir/modules # Format string for module commands exec,run,shell (not aliases) (can also include: -# {{registry}} , {{ namespace }}, {{tool}}, {{version}} +# {{registry}} , {{ repository }}, {{tool}}, {{version}} # This is where you might add a prefix to your module names, if desired. module_name: '{{ tool }}' diff --git a/shpc/tests/test_client.py b/shpc/tests/test_client.py index ff22dc5f3..531ceaac9 100644 --- a/shpc/tests/test_client.py +++ b/shpc/tests/test_client.py @@ -123,8 +123,6 @@ def test_inspect(tmp_path, module_sys, container_tech): """Test inspect""" client = init_client(str(tmp_path), module_sys, container_tech) client.install("python:3.9.2-slim") - client.inspect("python") - # Python won't have much TODO we should test a custom container metadata = client.inspect("python:3.9.2-slim") if container_tech == "singularity": diff --git a/shpc/version.py b/shpc/version.py index 2833205a0..ebb951a6b 100644 --- a/shpc/version.py +++ b/shpc/version.py @@ -2,7 +2,7 @@ __copyright__ = "Copyright 2021, Vanessa Sochat" __license__ = "MPL 2.0" -__version__ = "0.0.27" +__version__ = "0.0.28" AUTHOR = "Vanessa Sochat" NAME = "singularity-hpc" PACKAGE_URL = "https://github.com/singularityhub/singularity-hpc"