Add Universc

[TomKellyGenetics]

Add the UniverSC single-cell pipeline

Implements a completely open-source single-cell RNA pipeline that supports > 30 technologies. Uses publicly available docker container will all dependencies installed.

✔️ near completion ✔️

PR checklist

Closes #1644

• This comment contains a description of changes (with reason).
• If you've fixed a bug or added code that should be tested, add tests!
• If you've added a new tool - have you followed the module conventions in the contribution docs
• If necessary, include test data in your PR.
• Remove all TODO statements.
• Emit the versions.yml file.
• Follow the naming conventions.
• Follow the parameters requirements.
• Follow the input/output options guidelines.
• Add a resource label
• Use BioConda and BioContainers if possible to fulfil software requirements.
• Ensure that the test works with either Docker / Singularity. Conda CI tests can be quite flaky:
  • PROFILE=docker pytest --tag <MODULE> --symlink --keep-workflow-wd --git-aware
  • PROFILE=singularity pytest --tag <MODULE> --symlink --keep-workflow-wd --git-aware
  • PROFILE=conda pytest --tag <MODULE> --symlink --keep-workflow-wd --git-aware

[TomKellyGenetics]

I'll share updates to the UniverSC module for feedback. It still needs tests but inputs and documentation have been updated.

Note this module requires a docker container and a conda environment is not configured yet.

[JoseEspinosa]

Hi @TomKellyGenetics, thank you for this PR! We are merging as many modules as possible now due to and impending restructuring of the entire repo. This will mean you will need to update the module to reflect these changes before it can be merged in the future. It appears like this module isn't ready to be merged so if applicable, we are converting it to draft and adding the WIP label. If this isn't the case please let us know and we will try to get the module in before the changes. Thanks again!

[TomKellyGenetics]

Hi @JoseEspinosa thanks for letting me know. I'm ok to mark it as Draft/WIP as it is but it is almost ready to review I think.

What is the timeframe for the restructuring? Our publication is accepted at the proofing stage so it would be good timing to add the tool to nf-core at the same time.

I still need to test the code but it is very similar to Cell Ranger so I wouldn't anticipate many problems there. The main reason for slow progress is I've changed jobs and I don't have much time to spend on this project anymore. I'd like to complete it anyway if possible. Thanks for your patience.

[JoseEspinosa]

No worries at all! 😄
The restructuring was planned for the end of this week or the beginning of next week. If you can't do it you can always adapt the module to this change after the changes. The change consists mainly in a change in the structure of the directories of the repository so that subworkflows can be tested consistently, so should not be a big deal anyway.

[TomKellyGenetics]

Thanks for clarifying. I was wondering if templates would change making it problematic to use older ones. I'll try resolving the linting syntax errors and testing it. If the finishing touches are not done in time, it should be no problem to move files on my branch as necessary to avoid conflicts.

[JoseEspinosa]

I think the only change will be from where modules are included in the test main.nf. Anyhow, as soon as you fix the linting and the tests please tag it as ready for review so that we are aware and eventually can be included in the repo before the restructuring comes in place.

[TomKellyGenetics]

Thanks for the explanation @JoseEspinosa, I'll update tests on the current version first and then pull updates and merge the next version from nf-core.

As you can see above I've resolved linting issues and GitHub Actions passes on these. I've updated the test jobs on our podman node and it now starts without syntax errors I checked the following commands in the repo root directory with nextflow version 22.04.5.

nextflow run ./tests/modules/cellranger/count -entry test_cellranger_count -c ./tests/config/nextflow.config  -c ./tests/modules/cellranger/count/nextflow.config
nextflow run tests/modules/universc -entry test_cellranger_10x -c tests/config/nextflow.config
nextflow run tests/modules/universc -entry test_universc_10x -c tests/config/nextflow.config

Currently with local testing, the UniverSC module gives the same "OCI runtime error" (when running with podman) as the already merged Cell Ranger tests. I've pushed it to GitHub to pull to another node with rootless Docker installed. This should allow testing without this error.

I built the UniverSC container with podman build --format docker on another server but it seems to still have problems with podman. Do you know if podman can be supported in addition to docker and conda for nextflow? I'll continue testing but I think it is almost ready to review.

[TomKellyGenetics]

Singularity tests seem to fail due to insufficient disk space in the GitHub runner as the rootless Docker image is larger.

I’ll remove changes to the GitHub Actions setting and share local test results to confirm current settings pass with Singularity or Podman. As already demonstrated, the updated image runs a container with “docker” profile without root access, resolving the issue discussed above.

[TomKellyGenetics]

updates

I checked this version of the tomkellygenetics/universc:1.2.5.1 docker image and nf-core/modules test configurations run without errors.

Docker

$ export PROFILE="docker"
$ which docker
/usr/bin/docker
$ nextflow run tests/modules/nf-core/universc -entry test_universc_10x -c tests/config/nextflow.config  -c tests/modules/nf-core/universc/nextflow.config
N E X T F L O W  ~  version 22.10.4
Launching `tests/modules/nf-core/universc/main.nf` [evil_sax] DSL2 - revision: df4b1bc3c7
executor >  local (3)
[77/3adb7e] process > test_universc_10x:CELLRANGER_MKGTF (genome.gtf)   [100%] 1 of 1 ?
[56/2cd612] process > test_universc_10x:CELLRANGER_MKREF (genome.fasta) [100%] 1 of 1 ?
[7f/42d1fc] process > test_universc_10x:UNIVERSC (123)                  [100%] 1 of 1 ?
Completed at: 23-Jan-2023 14:01:28
Duration    : 9m 17s
CPU hours   : 0.3
Succeeded   : 3

Podman

export PROFILE="podman"
$ which podman
/usr/bin/podman
$ nextflow run tests/modules/nf-core/universc -entry test_universc_10x -c /home/DATA/PROJECT/20221221_nf_test_KellyST/000_DOC/010_scripts/nextflow.config -c tests/config/nextflow.config  -c tests/modules/nf-core/universc/nextflow.config
N E X T F L O W  ~  version 22.10.1
Launching `tests/modules/nf-core/universc/main.nf` [drunk_davinci] DSL2 - revision: df4b1bc3c7
executor >  local (3)
[52/6bd688] process > test_universc_10x:CELLRANGER_MKGTF (gen... [100%] 1 of 1 ?
[e6/417c61] process > test_universc_10x:CELLRANGER_MKREF (gen... [100%] 1 of 1 ?
[93/e547a8] process > test_universc_10x:UNIVERSC (123)           [100%] 1 of 1 ?
Completed at: 23-Jan-2023 14:19:24
Duration    : 4m 55s
CPU hours   : 0.2
Succeeded   : 3

Singularity

$ export PROFILE="singularity"
$ which singularity
/usr/bin/singularity
$ nextflow run tests/modules/nf-core/universc -entry test_universc_10x  -c tests/config/nextflow.config  -c tests/modules/nf-core/universc/nextflow.config
N E X T F L O W  ~  version 22.10.4
Launching `tests/modules/nf-core/universc/main.nf` [nauseous_varahamihira] DSL2 - revision: df4b1bc3c7
executor >  local (3)
[18/daae7c] process > test_universc_10x:CELLRANGER_MKGTF (genome.gtf)   [100%] 1 of 1 ?
[07/5c3476] process > test_universc_10x:CELLRANGER_MKREF (genome.fasta) [100%] 1 of 1 ?
[6f/a56e41] process > test_universc_10x:UNIVERSC (123)                  [100%] 1 of 1 ?
Completed at: 23-Jan-2023 14:29:24
Duration    : 4m 25s
CPU hours   : 0.1
Succeeded   : 3

summary

Therefore the docker image is ready to mirror on nfcore. Sorry for the inconvenience while testing this. Root permissions are no longer necessary.

[TomKellyGenetics]

Parameters to run singularity without "--user root" or "writable" are not supported by GitHub Actions: https://github.com/nf-core/modules/actions/runs/3983764318/jobs/6829355099

I've disabled automated testing and updated the code to call the "nfcore/universc:1.2.5.1" container. Once the mirror is set up this version should be ready to merge with all issues discussed above resolved.

@emiller88 redundant tests have been removed as suggested. Thanks for @ggabernet for explaining how to handle this.

@apeltzer could we get the mirrored image updated to v1.2.5.1? Thank you.

[TomKellyGenetics]

@apeltzer Thanks for your help. Friendly reminder that this is on hold until the image is mirrored at nfcore/universc:1.2.5.1 (https://hub.docker.com/r/nfcore/universc/tags)

Once this is available, I'll trigger GH Actions to confirm Docker runs without errors and ensure it is safe to merge. Thank you for your help.

[apeltzer]

Sorry @TomKellyGenetics - been quite a bit inactive these weeks (crazy new year start here at work...) --> mirror is now done ✅

[TomKellyGenetics]

Thanks @apeltzer. It is understandable, I am often in the same situation.

I've merged updates from the master branch. This resolves conflicts and confirms the updated module runs with the mirrored image.

@emiller88 please note the above changes have been made later during discussions with other reviewers. In my opinion it is ready to merge.

[ggabernet]

Looks good to me now

As far as I can tell, Tom updated everything and made the code much nicer + two extra reviews.

[maxulysse]

Why was this merged with linting failing?

[TomKellyGenetics]

@maxulysse it wasn’t? A docker image version was bumped before merging but automated checks were passing again at the time of merging (as the image was mirrored on the nfcore dockerhub page). Check the above commits to verify this.

Note that several commits, such as this one that failed linting were not made to this repository. They’re changes to the scRNA pipeline that referenced this PR in their commit messages.

Please tell me if you find any issues that need to be addressed with this module. Currently I don’t see any so I believe this is a misunderstanding.