start of work to add xserver support (#32)

* start of work to add xserver support

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

CHANGELOG.md

@@ -14,6 +14,7 @@ and **Merged pull requests**. Critical items to know are:
 The versions coincide with releases on pip. Only major versions will be released as tags on Github.
 
 ## [0.0.x](https://github.com/tunel-apps/tunel/tree/main) (0.0.x)
+ - support for xserver (singularity and slurm) (0.0.18)
  - bugfix in slurm stop and adding username to proxy (0.0.17)
  - bugfix template generation (0.0.16)
  - support for docker launcher (and intending to test podman) (0.0.15)

docs/getting_started/developer-guide.rst

@@ -44,7 +44,7 @@ Launchers include:
 
 
  - **singularity** run a singularity container on the server directly.
- - **docker** run a docker container on the server directly.
+ - **docker** run a docker container on the server directly (experimental support for podman)
  - **slurm**: submit jobs (or applications) via SLURM, either a job or a service on a node to forward back.
  - **condor**: submit jobs (or apps) to an HTCondor cluster.
 
@@ -61,6 +61,48 @@ The exception is docker, which we have a template variable to check for, e.g.,:
     {% endif %}
 
 
+Needs
+-----
+
+Each app can specify a set of boolean needs to indicate to the launcher how t do setup.
+A specification might look like the following in your ``app.yaml``:
+
+.. code-block:: console
+
+    needs:
+      xserver: true
+      socket: false
+
+If something is false, it's not required to include.
+
+socket
+^^^^^^
+
+If the app uses a socket, any existing *.sock files in the app directory on the remote will be cleaned up.
+In your application primary running script, you can easily create the socket relative to the script directory,
+and a helper include will ensure that we define the `SOCKET` environment variable and clean up any previous one.
+
+.. code-block:: console
+
+    SOCKET_DIR="{{ scriptdir }}"
+    mkdir -p ${SOCKET_DIR}
+
+    echo "Socket directory is ${SOCKET_DIR}"
+
+    # Remove socket if exists
+    {% include "bash/socket/set-socket.sh" %}
+
+It's helper to look at other application scripts.
+
+
+xserver
+^^^^^^^
+
+An xserver doesn't use typical ssh tunnel strategies like using a socket, but instead forwards with `-X`.
+This means we have support primarily for Singularity (running the container on the head node) and
+slurm (the same from a job node).
+
+
 Args
 ----
 

docs/getting_started/user-guide.rst

@@ -392,6 +392,7 @@ Our launchers include:
 Each launcher can be run via an app, meaning you do a ``run-app`` on an app.yaml that specifies the launcher,
 and we also provide courtesy functions (e.g., ``run-singularity``). These might be removed at some point, not decided yet.
 
+
 singularity
 ^^^^^^^^^^^
 

tunel/apps.py

@@ -60,6 +60,15 @@ def launchers(self):
             return [self.launcher]
         return list(set(self.launchers_supported + [self.launcher]))
 
+    @property
+    def has_xserver(self):
+        """
+        Boolean to indicate if an application has an xserver
+        """
+        if self.needs and self.needs.get("xserver", False):
+            return True
+        return False
+
     def get_script(self):
         return os.path.join(self.app_dir, self.script)
 

tunel/apps/xserver/neurodesk/app.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+JOB_NAME="{{ jobname }}"
+SCRIPT_DIR="{{ scriptdir }}"
+mkdir -p ${SCRIPT_DIR}
+
+# Sets $PORT envar from args.port then port
+{% include "bash/network/set-port.sh" %}
+
+# Include Singularity cachedir if not set
+{% include "bash/singularity/set-cache-tmp.sh" %}
+
+# Source ~/bash_profile or ~/.profile
+{% include "bash/source-profile.sh" %}
+
+# Working Directory
+{% include "bash/set-workdir.sh" %}
+cd $WORKDIR
+
+echo "Job is ${JOB_NAME}"
+echo "Port is ${PORT}"
+echo "Working directory is ${WORKDIR}"
+echo "Script directory is ${SCRIPT_DIR}"
+
+# Create .local folder for default modules, if doesn't exist
+{% include "bash/python/create-local.sh" %}
+
+# username and password for django to create
+{% include "bash/set-user-pass.sh" %}
+
+# Load modules requested by user
+{% for module in args.modules %}module load {{ module }} || printf "Could not load {{ module }}\n"
+{% endfor %}
+
+# Add variables to PATH
+{% for path in paths %}export PATH={{ path }}:${PATH}
+{% endfor %}
+
+{% if docker %}{% include "templates/run_docker.sh" %}{% else %}{% include "templates/run_singularity.sh" %}{% endif %}

tunel/apps/xserver/neurodesk/app.yaml

@@ -0,0 +1,24 @@
+launcher: singularity
+launchers_supported:
+ - singularity
+ - slurm
+script: app.sh
+description: Launch neurodesk (or an associated application)
+needs:
+  xserver: true
+examples: |
+  # Run app on login node with singularity
+  tunel run-app waffles neurodocker
+  # Run app on an interactive node (also with singularity)
+  tunel run-app waffles neurodocker --launcher slurm
+  # Force a new pull
+  tunel run-app waffles neurodocker --pull
+args:
+ - name: workdir
+   description: Working directory for app (and to show file explorer for)
+ - name: container
+   description: "Change the app container used (default is demo ghcr.io/neurodesk/caid/itksnap_3.8.0:20210322)"
+ - name: tag
+   description: "Tag of the container to use (defaults to latest)"
+ - name: pull
+   description: force a new pull (even if the container already exists).

tunel/apps/xserver/neurodesk/templates/run_singularity.sh

@@ -0,0 +1,20 @@
+SIF="${SINGULARITY_CACHEDIR}/neurodesk.sif"
+CONTAINER="docker://{% if args.container %}{{ args.container }}{% else %}ghcr.io/neurodesk/caid/itksnap_3.8.0:{% if args.tag %}{{ args.tag }}{% else %}20210322{% endif %}{% endif %}"
+
+if command -v singularity &> /dev/null
+then
+    printf "singularity pull ${CONTAINER}\n"
+
+    # Only pull the container if we do not have it yet, or the user requests it
+    if [[ ! -f "${SIF}" ]] || [[ "{{ args.pull }}" != "" ]]; then
+        singularity pull --force ${SIF} ${CONTAINER}
+    fi
+
+    # The false at the end ensures we aren't using nginx, but rather uwsgi just with sockets
+    printf "singularity run ${SIF}\n"
+    # Likely we could allow some custom binds here
+    singularity run ${SIF}
+
+else
+    printf "Singularity is not available.\n"
+fi

tunel/launcher/__init__.py

@@ -2,8 +2,8 @@
 
 from .base import Launcher
 from .docker import Docker
-from .podman import Podman
 from .htcondor import HTCondor
+from .podman import Podman
 from .singularity import Singularity
 from .slurm import Slurm
 

tunel/launcher/base.py

@@ -6,6 +6,7 @@
 
 import tunel.ssh
 import tunel.utils as utils
+from tunel.logger import logger
 
 here = os.path.dirname(os.path.abspath(__file__))
 
@@ -217,3 +218,52 @@ def scp_and_run(self, script):
         return self.ssh.execute(
             "chmod u+x %s; /bin/bash -l %s" % (remote_script, remote_script)
         )
+
+
+class ContainerLauncher(Launcher):
+    """
+    A container launcher has shared functions for launching a head node container.
+    """
+
+    def run_app(self, app):
+        """
+        A Singularity app means running a container directly with some arguments, etc.
+        """
+        # Make sure we set the username to the ssh
+        self.username
+
+        # Add any paths from the config
+        paths = self.settings.get("paths", [])
+
+        # Prepare dictionary with content to render into recipes
+        render = self.prepare_render(app, paths)
+
+        # Clean up previous sockets
+        self.ssh.execute(["rm", "-rf", "%s/*.sock" % render["scriptdir"]])
+
+        # Load the app template
+        template = app.load_template()
+        result = template.render(**render)
+
+        # Write script to temporary file
+        tmpfile = utils.get_tmpfile()
+        utils.write_file(tmpfile, result)
+
+        # Copy over to server
+        remote_script = os.path.join(self.remote_assets_dir, app.name, app.script)
+        self.ssh.scp_to(tmpfile, remote_script)
+
+        # Instead of a Singularity command, we run the script
+        command = "%s %s %s %s" % (
+            self.path,
+            self.environ,
+            self.ssh.settings.shell,
+            remote_script,
+        )
+
+        # An xserver launches the app directly
+        if not app.has_xserver:
+            logger.c.print()
+            logger.c.print("== INSTRUCTIONS WILL BE PRINTED with a delay ==")
+            self.print_tunnel_instructions(app, render["socket"])
+        self.ssh.execute(command, stream=True, xserver=app.has_xserver)

tunel/launcher/container.py

@@ -0,0 +1,87 @@
+__author__ = "Vanessa Sochat"
+__copyright__ = "Copyright 2021-2022, Vanessa Sochat"
+__license__ = "MPL 2.0"
+
+import os
+import shlex
+import threading
+import time
+
+import tunel.utils as utils
+from tunel.logger import logger
+
+from .base import Launcher
+
+
+class ContainerLauncher(Launcher):
+    """
+    A container launcher has shared functions for launching a head node container.
+    """
+
+    name = "container"
+
+    def run_app(self, app):
+        """
+        A Singularity app means running a container directly with some arguments, etc.
+        """
+        # Make sure we set the username to the ssh
+        self.username
+
+        # Add any paths from the config
+        paths = self.settings.get("paths", [])
+
+        # Prepare dictionary with content to render into recipes
+        render = self.prepare_render(app, paths)
+        render[self.name] = self.name
+
+        # Clean up previous sockets
+        self.ssh.execute(["rm", "-rf", "%s/*.sock" % render["scriptdir"]])
+
+        # Load the app template
+        template = app.load_template()
+        result = template.render(**render)
+
+        # Write script to temporary file
+        tmpfile = utils.get_tmpfile()
+        utils.write_file(tmpfile, result)
+
+        # Copy over to server
+        remote_script = os.path.join(self.remote_assets_dir, app.name, app.script)
+        self.ssh.scp_to(tmpfile, remote_script)
+
+        # Instead of a Singularity command, we run the script
+        command = "%s %s %s %s" % (
+            self.path,
+            self.environ,
+            self.ssh.settings.shell,
+            remote_script,
+        )
+
+        # An xserver launches the app directly
+        if not app.has_xserver:
+            logger.c.print()
+            logger.c.print("== INSTRUCTIONS WILL BE PRINTED with a delay ==")
+            self.print_tunnel_instructions(app, render["socket"])
+        self.ssh.execute(command, stream=True, xserver=app.has_xserver)
+
+    def print_tunnel_instructions(self, app, socket):
+        """
+        Start a separate thread to print connection details.
+        """
+        thread = threading.Thread(
+            target=post_commands,
+            name="Logger",
+            args=[self.ssh, app, socket],
+        )
+        thread.start()
+
+
+def post_commands(ssh, app, socket):
+    time.sleep(10)
+    ssh.tunnel_login_node(socket=socket, app=app)
+    if app.post_command:
+        logger.info("Found post command %s" % app.post_command)
+        post = app.post_command.replace("$socket_dir", os.path.dirname(socket))
+        ssh.execute(shlex.split(post), stream=True)
+    time.sleep(30)
+    ssh.tunnel_login_node(socket=socket, app=app)