Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: singularity sometimes uses system /tmp explanation #1588

Merged
merged 10 commits into from May 11, 2022

Conversation

Maarten-vd-Sande
Copy link
Contributor

@Maarten-vd-Sande Maarten-vd-Sande commented Apr 21, 2022

Description

The singularity conda packages dir does not respect the system tmp. Our servers have very small /tmp, so everyone makes their own tmp folder on /scratch/{user}. But snakemake does not respect the TMPDIR variable 馃槃

QC

  • The PR contains a test case for the changes or the changes are already covered by an existing test case.
  • The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

@Maarten-vd-Sande Maarten-vd-Sande changed the title Update conda.py fix: singularity respects tmpdir Apr 21, 2022
@Maarten-vd-Sande Maarten-vd-Sande changed the title fix: singularity respects tmpdir docs: singularity sometimes uses system /tmp explanation May 10, 2022
docs/project_info/faq.rst Outdated Show resolved Hide resolved
@sonarcloud
Copy link

sonarcloud bot commented May 11, 2022

Kudos, SonarCloud Quality Gate passed!聽 聽 Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.8% 0.8% Duplication

@johanneskoester johanneskoester merged commit 170c1d9 into snakemake:main May 11, 2022
7 checks passed
@nickp60
Copy link

nickp60 commented Jun 21, 2022

Hi @Maarten-vd-Sande can I get some clarification? I'm working on an HPC that uses LSF. I'm trying to convert a workflow to using per-rule conda envs in a global container. The pipeline works if executed with just --use-conda, but if I try to add --use-singularity I too am getting the following error:

$ snakemake --use-conda --use-singularity --snakefile ./vdb_16S/Snakefile --directory test_results/  --conda-frontend conda --singularity-args '--bind ${PWD}:${PWD},/data/brinkvd/:/data/brinkvd/'
onfig file /lila/home/watersn/GitHub/vdblab-pipelines/vdb_16S/runconfig.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Creating conda environment ../vdb_16S/envs/minimap2.yaml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /lila/home/watersn/GitHub/vdblab-pipelines/vdb_16S/rules/../envs/minimap2.yaml:
Command:
SINGULARITYENV_CONDA_PKGS_DIRS=/tmp/conda/9e5d92b4-e854-4e32-a81f-4360c633867a singularity  exec --home /lila/home/watersn/GitHub/vdblab-pipelines/test_results --bind ${PWD}:${PWD},/data/brinkvd/:/data/brinkvd/ /data/brinkvd/.singularity/487a72edc080f28be9d6fa74dcc90b96.simg sh -c 'conda config --set channel_priority strict && conda env create --quiet --file "/data/brinkvd/conda/0ce7b5045b512458b3f2c4cbac3d7203.yaml" --prefix "/data/brinkvd/conda/0ce7b5045b512458b3f2c4cbac3d7203"'
Output:
Collecting package metadata (repodata.json): ...working... failed

NoWritablePkgsDirError: No writeable pkgs directories configured.
  - /tmp/conda/9e5d92b4-e854-4e32-a81f-4360c633867a


When I saw this PR I tried adding --contain to singularity args, getting the following

Creating specified working directory test_results/.
Config file /lila/home/watersn/GitHub/vdblab-pipelines/vdb_16S/runconfig.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Creating conda environment ../vdb_16S/envs/minimap2.yaml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /lila/home/watersn/GitHub/vdblab-pipelines/vdb_16S/rules/../envs/minimap2.yaml:
Command:
SINGULARITYENV_CONDA_PKGS_DIRS=/tmp/conda/4d6f5119-423b-4121-af91-c53997323199 singularity  exec --home /lila/home/watersn/GitHub/vdblab-pipelines/test_results --bind ${PWD}:${PWD},/data/brinkvd/:/data/brinkvd/ --contain /data/brinkvd/.singularity/487a72edc080f28be9d6fa74dcc90b96.simg sh -c 'conda config --set channel_priority strict && conda env create --quiet --file "/data/brinkvd/conda/0ce7b5045b512458b3f2c4cbac3d7203.yaml" --prefix "/data/brinkvd/conda/0ce7b5045b512458b3f2c4cbac3d7203"'
Output:
Collecting package metadata (repodata.json): ...working... failed

NoSpaceLeftError: No space left on devices.


I seem to have plenty of space on my home directory and all bound locations. Is this something you ran into as well? Sorry for the lack of minimal example; any insight would be appreciated!

@Maarten-vd-Sande
Copy link
Contributor Author

Maarten-vd-Sande commented Jun 22, 2022

I'm not very experienced with singularity so I'm not sure. You can check if the mount temp option is on in the configuration (probably: /etc/singularity/singularity.conf).

When you use contain your home isn't mounted anymore, so perhaps you need to add -B $HOME as well?

@nickp60
Copy link

nickp60 commented Jun 27, 2022

@Maarten-vd-Sande I'm not exactly sure what the story is this this either. I think I'm having several issues at once, will link back when I get to the bottom of this. Binding HOME didn't resolve the issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants