Read-only filesystem errors from Snakemake in Singularity runtime

[tsibley]

The Snakemake upgrade in nextstrain/docker-base#136 caused failures for builds in the Singularity runtime due to new filesystem writes that Snakemake now tries to make under $HOME. Currently with our Singularity runner, $HOME and other non-working dirs are read-only.

The failure showed up as scheduled daily CI in this repo starting to fail between nextstrain/base:build-20230407T002437Z (last good) and build-20230411T103027Z (first bad).

I think we can address this by using the --writable-tmpfs flag in the Singularity runner to allow a writable (but temporary and ultimately discarded) filesystem outside of the working dir. This is similar to Docker's default behaviour. There may be other considerations with the change though.

[tsibley]

Maybe (or maybe not) related to this is that the nextstrain shell prompt also no longer shows up for the Singularity runtime. Instead, we get the default Singularity> shell prompt. But this might be Singularity version dependent also.

[tsibley]

I keep getting sidetracked on this. I'll pick it back up in a week or so, unless someone else gets to it. My extremely "for my eyes" notes on this:

diff --git a/nextstrain/cli/resources/bashrc b/nextstrain/cli/resources/bashrc
index 3456696..83a83a4 100644
--- a/nextstrain/cli/resources/bashrc
+++ b/nextstrain/cli/resources/bashrc
@@ -6,8 +6,13 @@ fi
 
 __NEXTSTRAIN_BASHRC=1
 
+OLDHOME="${HOME:-}"
+HOME=/nextstrain
+
 # Override PS1, this file's reason for being.
 OLDPS1="${PS1:-}"
+OLDPROMPT_COMMAND="${PROMPT_COMMAND:-}"
+unset PROMPT_COMMAND
 
 if [[ -n "${NEXTSTRAIN_PS1:-}" ]]; then
     PS1="$NEXTSTRAIN_PS1"
diff --git a/nextstrain/cli/runner/singularity.py b/nextstrain/cli/runner/singularity.py
index 9e7a671..3ba48cb 100644
--- a/nextstrain/cli/runner/singularity.py
+++ b/nextstrain/cli/runner/singularity.py
@@ -54,9 +54,15 @@ SINGULARITY_EXEC_ARGS = [
     # in this area are not available on older versions.
     #
     # ¹ e.g. <https://docs.sylabs.io/guides/latest/user-guide/singularity_and_docker.html#docker-like-compat-flag>
-    "--contain",
-    "--no-home",
-    "--cleanenv",
+    "--compat",
+
+    # XXX FIXME: --compat is 3.9.0
+    # XXX FIXME: --no-eval is 3.10.0 and included in --compat at that time
+    # 3.10.0 is a year old
+    # XXX FIXME: --writable-tmpfs is 3.0.0. we need this now
+    # XXX FIXME: do we still need --no-home? seems like no, not with --compat
+
+    # XXX FIXME: also need to adjust HOME to respect what the image sets… otherwise ends up weird
 
     # Since we use --no-home above, avoid warnings about not being able to cd
     # to $HOME (the default behaviour).  run() will override this by specifying
@@ -121,6 +127,11 @@ def run(opts, argv, working_volume = None, extra_env = {}, cpus: int = None, mem
         # Change the default working directory if requested
         *(("--pwd", "/nextstrain/%s" % working_volume.name) if working_volume else ()),
 
+        # XXX FIXME: Apply limits.
+        #   --cpus a la Docker <https://docs.sylabs.io/guides/latest/user-guide/cgroups.html#cpu-limits>
+        #   --memory a la Docker <https://docs.sylabs.io/guides/latest/user-guide/cgroups.html#memory-limits>
+        # Requires Singularity 3.10 (--apply-cgroups can do the same thing with only 3.9, but version diff seems marginal)
+
         str(image_path(image)),
         *argv,
     ], extra_env)