Skip to content

Parallel Processing Coordinator. Splits dataset processing to run parallel cluster jobs and aggregates outputs

License

Notifications You must be signed in to change notification settings

VictoriaBeilsten-Edmands/ParProcCo

 
 

Repository files navigation

ParProcCo

Requires a YAML configuration file in grandparent directory of package, CONDA_PREFIX/etc or /etc

--- !PPCConfig
allowed_programs:
    rs_map: msmapper_utils
    blah1: whatever_package1
    blah2: whatever_package2
project_env_var: CLUSTER_PROJECT
cluster_help_msg: Please module load blah
clusters:
    cluster_one: !PPCCluster
        default_queue: basic.q
        user_queues:
            better.q: middle_user1
            best.q: power_user1, power_user2
    cluster_two: !PPCCluster
        default_queue: only.q
        resources:
            cpu_model: arm64

About

Parallel Processing Coordinator. Splits dataset processing to run parallel cluster jobs and aggregates outputs

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%