Feature/psij scripts by jlnav · Pull Request #849 · Libensemble/libensemble

jlnav · 2022-05-31T20:09:41Z

Addresses #807

Introduces a new pair of utilities for preparing and launching libEnsemble workflows onto almost any machine and almost any scheduler, using a PSI/J backend. Should help alleviate some responsibility of maintaining batch submission scripts and determining how to submit jobs on each system.

liberegister

Creates an initial, platform-independent PSI/J representation of a libEnsemble run. Run this utility on a calling script in a familiar manner:

liberegister run_libe_forces.py --comms local --nworkers 4

This will produce a run_libe_forces.json serialization from PSI/J, conforming to their specification:

{
    "version": 0.1,
    "type": "JobSpec",
    "data": {
        "name": "libe-job",
        "executable": "python",
        "arguments": [
            "run_libe_forces.py",
            "--comms",
            "local",
            "--nworkers",
            "4"
        ],
        "directory": null,
        "inherit_environment": true,
        "environment": {
            "PYTHONNOUSERSITE": "1"
        },
        "stdin_path": null,
        "stdout_path": null,
        "stderr_path": null,
        "resources": {
            "node_count": 1,
            "process_count": null,
            "process_per_node": null,
            "cpu_cores_per_process": null,
            "gpu_cores_per_process": null,
            "exclusive_node_use": true
        },
        "attributes": {
            "duration": "30",
            "queue_name": null,
            "project_name": null,
            "reservation_id": null,
            "custom_attributes": {}
        },
        "launcher": null
    }
}

This json file is easy to make but also easy to share for use by anyone else using libEnsemble.

libesubmit

Further parameterizes a PSI/J serialization then submits the resulting Job to the specified scheduler. (for example):

libesubmit run_libe_forces.json -q debug -A project -s slurm

This will also produce a Job-specific serialization, like 8ba9de56.run_libe_forces.json:

{
    "version": 0.1,
    "type": "JobSpec",
    "data": {
        "name": "libe-job",
        "executable": "/Users/jnavarro/miniconda3/envs/libe/bin/python3.8",
        "arguments": [
            "run_libe_forces.py",
            "--comms",
            "local",
            "--nworkers",
            "4"
        ],
        "directory": "/Users/jnavarro/Desktop/libensemble/libensemble/libensemble/tests/scaling_tests/forces/forces_simple",
        "inherit_environment": true,
        "environment": {
            "PYTHONNOUSERSITE": "1"
        },
        "stdin_path": null,
        "stdout_path": "8ba9de56.run_libe_forces.out",
        "stderr_path": "8ba9de56.run_libe_forces.err",
        "resources": {
            "node_count": 4,
            "process_count": null,
            "process_per_node": null,
            "cpu_cores_per_process": null,
            "gpu_cores_per_process": null,
            "exclusive_node_use": true
        },
        "attributes": {
            "duration": "30",
            "queue_name": "debug",
            "project_name": "project",
            "reservation_id": null,
            "custom_attributes": {}
        },
        "launcher": null
    }
}

If libesubmit is run on a .json serialization from liberegister and can't find the specified calling script, it'll help find matching candidate scripts:

*** libEnsemble 0.9.1+dev ***
Imported PSI/J serialization: run_libe_forces.json. Preparing submission...
Calling script: run_libe_forces.py
... not found in Job working directory!
Check somewhere else? (Y/N): y
/Users/jnavarro:
0. Music
1. Pictures
2. Desktop
3. Library
4. Public
5. Movies
6. Documents
7. Downloads
8. miniconda3
Specify a starting directory: 2
preparing... ctrl+c to abort.
132168it [00:02, 58050.10it/s]
detecting... ctrl+c to abort.
/Users/jnavarro:
1. .../libensemble/gitlab/libensemble/libensemble/tests/scaling_tests/forces/run_libe_forces.py                                                                                                                  
2. .../libensemble/gitlab/libensemble/examples/calling_scripts/run_libe_forces.py                                                                                                                                
3. .../libensemble/libensemble/libensemble/tests/scaling_tests/forces/forces_gpu/run_libe_forces.py                                                                                                              
4. .../libensemble/libensemble/libensemble/tests/scaling_tests/forces/forces_adv/run_libe_forces.py                                                                                                              
5. .../libensemble/libensemble/libensemble/tests/scaling_tests/forces/forces_simple/run_libe_forces.py                                                                                                           
6. .../libensemble/libensemble/examples/calling_scripts/run_libe_forces.py                                                                                                                                       
7. .../libensemble/libensemble/examples/tutorials/forces_with_executor/run_libe_forces.py                                                                                                                        
8. .../libensemble/software/libensemble/libensemble/tests/scaling_tests/forces/run_libe_forces.py                                                                                                                
9. .../libensemble/software/libensemble/examples/calling_scripts/run_libe_forces.py                                                                                                                              
10. .../libensemble/old_hpc/calling/templates/run_libe_forces.py                                                                                                                                                 
11. .../libensemble/old_hpc/hpc_tests/summit/templates/run_libe_forces.py                                                                                                                                        
12. .../libensemble/old_hpc/hpc_tests/old/calling/templates/run_libe_forces.py                                                                                                                                   
13. .../libensemble/old_hpc/hpc_tests/bebop/templates/run_libe_forces.py                                                                                                                                         
14. .../libensemble/old_hpc/hpc_tests/cori/templates/run_libe_forces.py                                                                                                                                          
15. .../libensemble/old_hpc/hpc_tests/theta/templates/run_libe_forces.py                                                                                                                                         
16. .../libensemble/newgitlab/libensemble/libensemble/tests/scaling_tests/forces/run_libe_forces.py                                                                                                              
17. .../libensemble/newgitlab/libensemble/examples/calling_scripts/run_libe_forces.py                                                                                                                            
18. .../libensemble/libE-templater/platforms/all/run_libe_forces.py                                                                                                                                              
19. .../libensemble/src/libensemble/libensemble/tests/scaling_tests/forces/run_libe_forces.py                                                                                                                    
20. .../libensemble/src/libensemble/examples/calling_scripts/run_libe_forces.py                                                                                                                                  
21. .../sc21/libensemble/libensemble/tests/scaling_tests/forces/run_libe_forces.py                                                                                                                               
22. .../sc21/libensemble/examples/calling_scripts/run_libe_forces.py                                                                                                                                             
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 132168/132168 [00:01<00:00, 95904.14it/s]
Specify a detected script: 5

Should these two scripts really be combined into one script?
Create custom launcher to flexibly launch libEnsemble w/ MPI onto allocated resources

… a PSI/J spec of a libEnsemble run, and run that spec on a system

…utor

…ubmit to crawl for calling script if matching script not found in current directory

…for calling script

…w...

…orks

jlnav · 2022-06-14T16:45:39Z

Getting started:

git clone https://github.com/ExaWorks/psi-j-python.git, then pip install
pip install tqdm
Check out this branch, then pip install the libEnsemble repository again, so scripts are copied into the environment's bin

jlnav · 2022-06-14T16:46:37Z

Probably ready for the first round of feedback. Still WIP of course

shuds13

Following the discussion/demo, I am happy with how this is going so far. It currently is working for
local comms on the tested platforms. Looks like we need a psi/j update for mpi4py comms.

We can decide best location for utility scripts.

jlnav · 2022-07-11T21:19:12Z

This iteration may be sufficient for a portable multiprocessing solution. As has been stated, extensions to PSI/J (which I'm looking into) will be necessary to support MPI comms.

Thoughts?

…scripts

…ister

…ork isn't this script's job

jlnav · 2022-07-14T16:32:33Z

Document

jlnav added 24 commits May 27, 2022 17:08

initial commit for making a pair of scripts that respectively produce…

4ef9318

… a PSI/J spec of a libEnsemble run, and run that spec on a system

add remaining options, with sensible defaults.

434cb5c

renames, some cleanup, begin submission script

605fdc6

black first script

2618db8

add launcher choices

e5a5f06

replace some unportable defaults, small fixes, instantiate psi/j exec…

c5cd5fd

…utor

Merge branch 'develop' into feature/psij_scripts

938ef2d

small adjusmtments, auto-specify PYTHONNOUSERSITE, add logic to libes…

750d522

…ubmit to crawl for calling script if matching script not found in current directory

small fixes

076972e

small adjustments

0d18e26

various small fixes, plus allow user to narrow down search directory …

a9991a5

…for calling script

black, and create stdout/stderr files

6eef8f0

proper python headers?

cabb019

small reformattings, set new script as installable in environment's bin

f867030

make default nnodes 1

8a1a847

probably giving up on evaluating and waiting for exit criteria for no…

dca2923

…w...

black

66a5cf6

fix reference to other psi-j script

85cb75a

first attempts to define MPI resources for an MPI run

f54c786

black

6d94da0

specify processes per node

6cb26bb

try libesubmit with ppn=1

111375f

just experimenting until some arrangement of resources for MPI runs w…

be55f73

…orks

trying out dry-run feature and adding nnodes option to libesubmit

613c8ba

jlnav requested a review from shuds13 June 14, 2022 16:46

print help and exit if libesubmit didn't take in a json file

9d9d1e3

shuds13 reviewed Jun 15, 2022

View reviewed changes

Merge branch 'develop' into feature/psij_scripts

5ba3733

jlnav added 6 commits June 30, 2022 15:34

move psij scripts into base scripts directory

8d95433

fix symlink

c1aea4d

Merge branch 'develop' into feature/psij_scripts

4a8d9ac

oMerge branch 'develop' into feature/psij_scripts

f585498

odd symlink in repo base?

177c16f

some fixes, add if name == main line to simple forces

4e23b95

jlnav requested a review from shuds13 July 11, 2022 21:17

jlnav marked this pull request as ready for review July 11, 2022 21:19

some attempts to catch and help resolve missing dependencies in psij …

e9edbb9

…scripts

jlnav marked this pull request as draft July 13, 2022 19:05

jlnav added 3 commits July 13, 2022 16:16

first attempt at including nsim_workers and nresource_sets in libereg…

4748b79

…ister

Merge branch 'develop' into feature/psij_scripts

20e0071

fix logic so args are just passed though - evaluating how well they w…

2dd65b7

…ork isn't this script's job

jlnav marked this pull request as ready for review July 14, 2022 16:17

shuds13 approved these changes Jul 14, 2022

View reviewed changes

jlnav added 3 commits July 15, 2022 16:26

Merge branch 'develop' into feature/psij_scripts

7777f2c

first approach to documenting psi/j scripts

25c5a86

mention new optional dependencies in README

41ad54d

jlnav changed the title ~~[WIP] Feature/psij scripts~~ Feature/psij scripts Jul 19, 2022

jlnav merged commit 0075ea3 into develop Jul 20, 2022

jlnav deleted the feature/psij_scripts branch July 20, 2022 20:08

jlnav mentioned this pull request Jul 20, 2022

Implement the PSI/J interface into libEnsemble #807

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/psij scripts#849

Feature/psij scripts#849
jlnav merged 39 commits intodevelopfrom
feature/psij_scripts

jlnav commented May 31, 2022 •

edited

Loading

Uh oh!

jlnav commented Jun 14, 2022 •

edited

Loading

Uh oh!

jlnav commented Jun 14, 2022

Uh oh!

shuds13 left a comment

Uh oh!

jlnav commented Jul 11, 2022

Uh oh!

jlnav commented Jul 14, 2022 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jlnav commented May 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jlnav commented Jun 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jlnav commented Jun 14, 2022

Uh oh!

shuds13 left a comment

Choose a reason for hiding this comment

Uh oh!

jlnav commented Jul 11, 2022

Uh oh!

jlnav commented Jul 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jlnav commented May 31, 2022 •

edited

Loading

jlnav commented Jun 14, 2022 •

edited

Loading

jlnav commented Jul 14, 2022 •

edited

Loading