Snakemake now failing with Slurm profile

[BournSupremacy]

Hi Job,

I've been trying to run the new release of the pipeline, but unfortunately have not been able to get it to work at all this time. I can download and install Snakemake fine and follow the instructions exactly in the sci-rocket docs, and I have a Slurm-specific Snakemake configuration that has worked perfectly for me before (I have even remade it a few times to try and fix it but it hasn't worked). Now when I try to run the snakemake --use-conda --profile <profile_name> --configfile <path_config> command, I get the error:

snakemake --use-conda --profile embl_slurm --configfile /g/dorrity/sci_rna_expts/medaka_apr2023/Nova/sci-rocket/run2/config_run2.yaml`

Using profile embl_slurm for setting default command line arguments.`

usage: snakemake [-h] [--dry-run] [--profile PROFILE] [--workflow-profile WORKFLOW_PROFILE] [--cache [RULE ...]] [--snakefile FILE] [--cores N] [--jobs N] [--local-cores N]  [--resources NAME=INT [NAME=INT ...]] [--set-threads RULE=THREADS [RULE=THREADS ...]] [--max-threads MAX_THREADS] [--set-resources RULE:RESOURCE=VALUE [RULE:RESOURCE=VALUE ...]] [--set-scatter NAME=SCATTERITEMS [NAME=SCATTERITEMS ...]] [--set-resource-scopes RESOURCE=[global|local] [RESOURCE=[global|local] ...]] [--default-resources [NAME=INT ...]] [--preemptible-rules [PREEMPTIBLE_RULES ...]] [--preemptible-retries PREEMPTIBLE_RETRIES] [--config [KEY=VALUE ...]] [--configfile FILE [FILE ...]] [--envvars VARNAME [VARNAME ...]] [--directory DIR] [--touch] [--keep-going]  [--rerun-triggers {code,input,mtime,params,software-env} [{code,input,mtime,params,software-env} ...]] [--force] [--executor {local,dryrun,touch}] [--forceall]  [--forcerun [TARGET ...]] [--prioritize TARGET [TARGET ...]] [--batch RULE=BATCH/BATCHES] [--until TARGET [TARGET ...]] [--omit-from TARGET [TARGET ...]] [--rerun-incomplete] [--shadow-prefix DIR] [--scheduler [{ilp,greedy}]] [--wms-monitor [WMS_MONITOR]] [--wms-monitor-arg [NAME=VALUE ...]]  [--scheduler-ilp-solver {COIN_CMD}] [--scheduler-solver-path SCHEDULER_SOLVER_PATH] [--conda-base-path CONDA_BASE_PATH] [--no-subworkflows] [--precommand PRECOMMAND] [--groups GROUPS [GROUPS ...]] [--group-components GROUP_COMPONENTS [GROUP_COMPONENTS ...]] [--report [FILE]] [--report-stylesheet CSSFILE] [--draft-notebook TARGET] [--edit-notebook TARGET] [--notebook-listen IP:PORT] [--lint [{text,json}]] [--generate-unit-tests [TESTPATH]] [--containerize] [--export-cwl FILE] [--list-rules] [--list-target-rules] [--dag] [--rulegraph] [--filegraph] [--d3dag] [--summary] [--detailed-summary] [--archive FILE] [--cleanup-metadata FILE [FILE ...]] [--cleanup-shadow] [--skip-script-cleanup] [--unlock] [--list-changes {params,input,code}] [--list-input-changes] [--list-params-changes] [--list-untracked] [--delete-all-output | --delete-temp-output] [--keep-incomplete] [--drop-metadata] [--deploy-sources QUERY CHECKSUM] [--version] [--printshellcmds] [--debug-dag] [--nocolor] [--quiet [{all,progress,rules} ...]] [--print-compilation] [--verbose] [--force-use-threads] [--allow-ambiguity] [--nolock] [--ignore-incomplete] [--max-inventory-time SECONDS] [--latency-wait SECONDS] [--wait-for-files [FILE ...]] [--wait-for-files-file FILE] [--notemp] [--all-temp] [--unneeded-temp-files FILE [FILE ...]] [--keep-storage-local-copies] [--target-files-omit-workdir-adjustment] [--allowed-rules ALLOWED_RULES [ALLOWED_RULES ...]]  [--target-jobs TARGET_JOBS [TARGET_JOBS ...]] [--local-groupid LOCAL_GROUPID] [--max-jobs-per-second MAX_JOBS_PER_SECOND] [--max-status-checks-per-second MAX_STATUS_CHECKS_PER_SECOND] [--seconds-between-status-checks SECONDS_BETWEEN_STATUS_CHECKS] [--retries RETRIES] [--attempt ATTEMPT] [--wrapper-prefix WRAPPER_PREFIX] [--default-storage-provider DEFAULT_STORAGE_PROVIDER] [--default-storage-prefix DEFAULT_STORAGE_PREFIX] [--local-storage-prefix LOCAL_STORAGE_PREFIX] [--shared-fs-usage {input-output,persistence,software-deployment,source-cache,sources,storage-local-copies,none} [{input-output,persistence,software-deployment,source-cache,sources,storage-local-copies,none} ...]] [--scheduler-greediness SCHEDULER_GREEDINESS] [--no-hooks] [--debug] [--runtime-profile FILE] [--mode {subprocess,remote,default}] [--show-failed-logs] [--log-handler-script FILE] [--log-service {none,slack,wms}] [--job-deploy-sources] [--container-image IMAGE] [--immediate-submit] [--jobscript SCRIPT] [--jobname NAME] [--flux] [--software-deployment-method {apptainer,conda,env-modules} [{apptainer,conda,env-modules} ...]] [--container-cleanup-images] [--use-conda] [--conda-not-block-search-path-envvars] [--list-conda-envs] [--conda-prefix DIR] [--conda-cleanup-envs] [--conda-cleanup-pkgs [{tarballs,cache}]] [--conda-create-envs-only] [--conda-frontend {conda,mamba}] [--use-apptainer] [--apptainer-prefix DIR] [--apptainer-args ARGS] [--use-envmodules] [targets ...]

snakemake: error: unrecognized arguments: --cluster-cancel=scancel --cluster=/home/bourn/.config/snakemake/embl_slurm/slurm-submit.py --cluster-status=/home/bourn/.config/snakemake/embl_slurm/slurm-status.py

It seems it is not recognising the Slurm config properly but I am not sure how to fix it given that the exact same configuration has worked perfectly many times before, and I'm downloading the Slurm template with cookiecutter (as before).

I notice that the commands for installing sci-rocket are different from what they used to be. Now you are using Micromamba to create a Snakemade enivornment, where before I used the command micromamba create --prefix /g/dorrity/bin/condaEnvs/sci-rocket -f workflow/envs/sci-rocket.yaml to create a sci-rocket environment. I am guessing this is where my problem is. Do you perhaps have any insight into what might be going wrong? Any ideas would be appareciated!

Best,
Jess

[BournSupremacy]

Ah. According to this issue on the Snakemake-Profiles repo, It seems there has been an update to Snakemake but not yet the profiles. So they're currently incompatible. Nothing to do with sci-rocket after all! I will either just wait until that's been fixed or use an older version of Snakemake and try again. Sorry for wasting your time!