Bindmount

[prioux]

Adds the ability to specify bindmounts in a ToolConfig, for tasks launched with Apptainer.

The configuration is performed in the "Singularity Overlays" text area, where the admin can add lines like:

bindmount:/data1/user-data/prioux/misc/HbcdQcContainer:/inside/hbcd1
bindmount:/data1/user-data/prioux/misc/HbcdQcContainer:/inside/hbcd2:ro

There are two commits in this PR:

1. A draft work by Sergiy, commit 3d7816d
2. My work finishing the feature, commit bdb1723

[prioux]

Note, there is also a small irrelevant change in the "head" shortcut too, just wanted to make it go through too.

[prioux]

For this pull request, I intend to merge it using a merge commit instead of a squash commit, so that each developer's work is properly credited.

[MontrealSergiy]

seems fine to me.
Would you like me to test it?

[prioux]

Sure test it, it's easy

[prioux]

I just used the diagnostics tool with a plain linux container for it

[prioux]

You don't even need a boutiques task for this feature

[prioux]

I am going to add right now the commit for the OTHER feature too, since it's only one method changed. Good design.

[prioux]

This PR now closes TWO tickets: #1568 and #1569

[natacha-beck]

Tested on my side and it is a really nice addition.

[MontrealSergiy]

The first feature was tested, tool config seems working.

The second feature looks simple and elegant, though:

1. as Natacha mentioned, it leads to Boutiques descriptors that cannot be used outside CBRAIN unless a wrapper is mounted or otherwise made accessible.

2. We often have several Boutiques files for different versions of the same tool, and parameters come and go, we do not have a way to pick one particular wrapper, so the wrapper name would need a version-specific prefix or subfolder, which might get messy overtime with sometimes prefix update forgotten, sometimes not.

3. there is departure from CBRAIN approach of using installed_plugins folder for all the boutiques staff

4. Renaming the repository (e.g. alpha_beta_eeg_cbrain to cbrain-plugins-ab-eeg ) or merging into cbrain-plugins-neuro might fail, unless properties are adjusted

I do not have particularly elegant solution for all these bullet points, but at least we should be aware of challenges that may arise with new approach

[prioux]

Reply, point by point:

1. Yes that's true, but in the container_mnt folder in the plugins, it's easy to install a README that say "when running standalone, pleas mount this folder using -B or using the APPTAINER_BINDPATH environement variable
2. I don't see a problem. The folder we mount can have as many versions of wrappers as needed, and the version that is called is whatever you put in the "command-line" entry in the descriptor. So you can have exact 1-1 matches, or have a single wrapepr that works for multiple versions of the container, or have serveral wrappers that work with the same version of the container. All possibilities are available.
3. No there isn't. Read the code. We use the same system. It's true I resolve the symlink to go back to the original installation path of the plugins, but that's what is already happenign in the installed_plugins folder anyway.
4. If you rename the repo the paths will adjust automatically at run time for the next job.

I will consider maybe making it one more "install" step in the rake task that install plugins, but for the moment that's good enough, I think.

[MontrealSergiy]

BTW boutiques support creating files (see 'file-template' attribute), though it does not have execution right and only suits for small text files, not executable or libraries

[prioux]

BTW boutiques support creating files in workfolder 'file-template', though it does not have execution right and only suits for small text files, not executables or libraries

I don't understand what that means. Can you explain?

[MontrealSergiy]

the boutiques format has a feature to create additional files whenever bosh exec... is launched. They are supposed to create config file(s), but technically any textual files can be created. Example scripty.py:

    "command-line": "bash scripty.sh; du [CBRAIN_INPUT] | tee [DU_OUTPUT_NAME]"

    "output-files": [
        {
            "name":          "Output report",
            "id":            "du_report_out",
            "description":   "The output of the 'du' command",
            "optional":      false,
            "list":          false,
            "path-template": "[DU_OUTPUT_NAME]"
        },
	  {
            "name":         "scripty",
            "id":            "rapper",            
            "optional":      false,
            "list":          false,
            "path-template": "scripty.sh",
            "file-template": ["echo hello world", "echo bye bye cruel world"]
        }

[prioux]

In output-files ? Never heard of that before. Will check the BTQ doc.

[MontrealSergiy]

This one is not particularly straightforward too.

I did not find much in BTQ docs, yet it is in the schema and Tristan's paper Algorithm 1 pseudo-code

# Write all configuration files.
for output in outputs do
  if output has a file-template then
    Write file-template in path-template
    # Value keys have been substituted in file-template
        previously.
  end if
end for