RTE SINGULARITY: Use the SINGULARITY_OPTIONS input parameter instead of the environment variable if defined

[aldbr]

In the documentation of the Singularity RTE, 2 parameters are defined: SINGULARITY_IMAGE and SINGULARITY_OPTIONS.

Defining SINGULARITY_IMAGE in the submission script seems to have an effect on the execution and effectively uses the given singularity image. See in the source code: $2 is used here

Unfortunately, this does not seem to be the case for the SINGULARITY_OPTIONS. Using this variable in the submission script does not change anything during the execution: options are not used.

This PR aims to support the use of the SINGULARITY_OPTIONS parameter from the submission script.

• If not defined in the submission script, then the RTE Singularity script tries to use the environment variable
• If not defined, then no singularity option is defined

[manfuin]

Allowing end-user to define singularity options on-demand is a security hole.
In this case end-user can e.g. request to mount any host directory inside his own container.

But if you don't mind to open your system in this way, you can always create a "user defined" RTE script even with the same name and use it, no need to update the "system RTE" redistributed with ARC.

[aldbr]

Thanks for your prompt answer.

Is that really a security hole?
If I understand correctly, the user remains the same inside and outside the container: e.g. if a user is not authorized to access a specific directory, it will not be able to access it via the container.
A lot of supercomputers support singularity. Would they do it if there was such security hole?

Source: https://sylabs.io/guides/3.3/user-guide/security.html#singularity-runtime

[manfuin]

This is not about singlarity itself, it is about user origins and trust boundaries.

Local users indeed has access to the filesystem with or without singularity based on they permissions and this is defined by organization policies. However hear we are talking about grid users who are external to the resource and have nothing to do with organization policy, they just mapped to some accounts and should be handled in a more restricted way generally.

Using singularity with proper option allows to enforce better isolation (e.g. mounting only the job scratch area).
But making such an options configurable by external users themselves allows them to bypass isolation.

What is you use-case?
Why external user should be able to override the CE-admin defined way of singularity callout?

[maikenp]

An option here could be to modify the change you suggest to add a default None, then if the sys-admin wants to allow this option, he will set the parameter.

[maikenp]

Allowing end-user to define singularity options on-demand is a security hole.
In this case end-user can e.g. request to mount any host directory inside his own container.

But if you don't mind to open your system in this way, you can always create a "user defined" RTE script even with the same name and use it, no need to update the "system RTE" redistributed with ARC.

Note @manfuin that the SINGULARITY_OPTIONS parameter is already defined in the RTE, but not used.

[manfuin]

Note @manfuin that the SINGULARITY_OPTIONS parameter is already defined in the RTE, but not used.

There is a big difference between parameter defined by system owner vs defined in job xRSL by end-user

[aldbr]

Local users indeed has access to the filesystem with or without singularity based on they permissions and this is defined by organization policies. However hear we are talking about grid users who are external to the resource and have nothing to do with organization policy, they just mapped to some accounts and should be handled in a more restricted way generally.
Using singularity with proper option allows to enforce better isolation (e.g. mounting only the job scratch area).
But making such an options configurable by external users themselves allows them to bypass isolation.

Alright, I understand your concern, and indeed users could potentially exploit the SINGULARITY_OPTIONS parameter to bypass isolation.
But in the same way, they could also exploit the SINGULARITY_IMAGE parameter to bypass isolation, couldn't they?

What is your use-case?
Why external user should be able to override the CE-admin defined way of singularity callout?

We share an ARC instance connected to a supercomputer - with no external connectivity at all - with other VOs.
Our jobs need to interact with distinct singularity containers, and distinct directories to get their software dependencies.
For instance:

• VO1 will need: singularity exec --bind /path/to/dir1:/cvmfs image1.sif <command>
• VO2 will need: singularity exec --bind /path/to/dir2:/cvmfs image2.sif <command>

There is probably a workaround: I guess we could embed a given directory within our container to just have to call:
singularity exec image1.sif