pyp slurm submission

[FlorianBeckOle]

Hi,

the pyp command from the tutorial just runs locally.
Anthing I missed in the config or the pyp command ?

best

Florian

pyp -data_mode spr -scope_voltage 300  -scope_pixel 0.66 -data_path='/fs/pool/pool-fbeck/misc/testPYP/T20S/nextpyp_spr_tutorial/14sep05c_00024sq_00003h*.tif' -gain_reference='/fs/pool/pool-fbeck/misc/testPYP/T20S/nextpyp_spr_tutorial/Gain.mrc' -gain_flipv -detect_rad 80 -detect_method all -detect_dist 40 -slurm_tasks 7 -slurm_memory 70 -slurm_merge_tasks 7 -slurm_merge_memory 70
Launching pyp ...
Reading configuration file at: /fs/pool/pool-bmapps/hpcl8/app/soft/PYP/0.7.1/config.toml
2025-08-07 16:25:57 [INFO] /opt/pyp/bin/run/pyp:3992 | Job 585738843_1 running nextPYP v0.7.1 on hpcl9301 using 1 task(s) ?
2025-08-07 16:25:57 [INFO] /opt/pyp/bin/run/pyp:5846 | Running on directory /fs/pool/pool-fbeck/misc/testPYP
2025-08-07 16:25:58 [INFO] /opt/pyp/bin/run/pyp:600 | Attempting to load existing parameters and metadata from /fs/pool/pool-fbeck/misc/testPYP/T20S/nextpyp_spr_tutorial
2025-08-07 16:25:58 [INFO] /opt/pyp/bin/run/pyp:642 | /fs/pool/pool-fbeck/misc/testPYP/T20S/nextpyp_spr_tutorial/14sep05c_00024sq_00003h*.tif found 2 file(s) to link into raw/ folder
2025-08-07 16:25:58 [INFO] /opt/pyp/bin/run/pyp:650 | Linking 2 files into raw/*.tif
2025-08-07 16:25:58 [INFO] pyp/system/project_params.py:284 | Found 2 unique file(s) for processing
2025-08-07 16:28:40 [INFO] pyp/system/local_run.py:116 | /bin/bash -c "export SLURM_CPUS_PER_TASK=7; SLURM_NTASKS=7; export OMP_NUM_THREADS=7; export MKL_NUM_THREADS=7; export sprmerge=sprmerge; cd swarm; mkdir -p /fs/pool/pool-briggs-scratch/PYP/fbeck; /opt/pyp/bin/run/pyp "
2025-08-07 16:28:42 [INFO] pyp/system/local_run.py:77 | 2025-08-07 16:28:42 [INFO] /opt/pyp/bin/run/pyp:3992 | Job 585738843_1 running nextPYP v0.7.1 on hpcl9301 using 7 task(s) ?
2025-08-07 16:28:42 [INFO] pyp/system/local_run.py:77 | 2025-08-07 16:28:42 [INFO] /opt/pyp/bin/run/pyp:4125 | Running merge from /fs/pool/pool-fbeck/misc/testPYP
2025-08-07 16:28:42 [INFO] pyp/system/local_run.py:77 | 2025-08-07 16:28:42 [INFO] pyp/system/project_params.py:157 | Looking for failed jobs
2025-08-07 16:28:42 [INFO] pyp/system/local_run.py:77 | Progress:   0%|          | 0/2 [00:00<?, ?it/s]
2025-08-07 16:28:42 [INFO] pyp/system/local_run.py:77 | Progress: 100%|##########| 2/2 [00:00<00:00, 1078.50it/s]
2025-08-07 16:28:42 [INFO] pyp/system/local_run.py:77 | 2025-08-07 16:28:42 [INFO] /opt/pyp/bin/run/pyp:777 | All jobs finished successfully
2025-08-07 16:28:42 [INFO] pyp/system/local_run.py:77 | 2025-08-07 16:28:42 [INFO] /opt/pyp/bin/run/pyp:779 | Total number of micrograph(s) = 2
2025-08-07 16:28:42 [INFO] pyp/system/local_run.py:77 | 2025-08-07 16:28:42 [WARNING] pyp:680 | Updating micrograph/tilt-series order according to number of particles
2025-08-07 16:28:42 [INFO] pyp/system/local_run.py:77 | 2025-08-07 16:28:42 [INFO] /opt/pyp/bin/run/pyp:783 | Number of micrographs used = 2
2025-08-07 16:28:44 [INFO] pyp/system/local_run.py:77 | 2025-08-07 16:28:44 [INFO] /opt/pyp/bin/run/pyp:75 | Total time elapsed (spr_merge): 2.2s
2025-08-07 16:28:44 [INFO] pyp/system/local_run.py:77 | 2025-08-07 16:28:44 [INFO] /opt/pyp/bin/run/pyp:4132 | nextPYP (sprmerge) finished successfully
2025-08-07 16:28:45 [INFO] /opt/pyp/bin/run/pyp:5881 | nextPYP (launch) finished successfully
cat /fs/pool/pool-bmapps/hpcl8/app/soft/PYP/0.7.1/config.toml

[pyp]

# path to the PYP container file, from a SLURM compute node's point of view
container = '/fs/pool/pool-bmapps/hpcl8/app/soft/PYP/0.7.1/pyp.sif'

# fast storage for temporary files, ideally local to the compute node
scratch = '/fs/pool/pool-briggs-scratch/PYP'

# Bind network locations as needed
binds = ['/fs/pool/pool-fbeck/misc']

[slurm]

path = '/usr/bin'

queues = ['p.hpcl9','p.hpcl8','p.hpcl93']

gpuQueues = ['p.hpcl9','p.hpcl8','p.hpcl93']

[abartesaghi]

It looks like you are missing the slurm.host entry from the configuration file, see here:

[slurm]

host = 'hostname_of_slurm_login_node'

I just noticed that this is not mentioned in the docs, so I will make a note to update the documentation.

[FlorianBeckOle]

that worked. thanks Florian

…

________________________________ Von: Alberto Bartesaghi ***@***.***> Gesendet: Donnerstag, 7. August 2025 17:02:17 An: nextpyp/.github Cc: Beck, Florian; Author Betreff: Re: [nextpyp/.github] pyp slurm submission (Discussion #149) It looks like you are missing the slurm.host entry from the configuration file, see here<https://nextpyp.app/files/pyp/latest/docs/reference/config.html#host>: [slurm] host = 'hostname_of_slurm_login_node' I just noticed that this is not mentioned in the docs, so I will make a note to update the documentation. — Reply to this email directly, view it on GitHub<#149 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/APAUYRJ4WCXNQ5OSEGR2Y2T3MNS7TAVCNFSM6AAAAACDLGIEMGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTIMBTGYYDKMQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>