New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multisite support #48

Closed
yadudoc opened this Issue Dec 8, 2017 · 1 comment

Comments

Projects
None yet
1 participant
@yadudoc
Contributor

yadudoc commented Dec 8, 2017

This is to track ongoing work on the multisite branch.

Why is this important:

  1. This will allow us to manage apps with different resource requirements in a single workflow, such as apps that only need a core and should be executed on the login node vs, apps that run on multiple
    nodes with MPI.
  2. Light weight apps executing on threads could perform flowcontrol and launch heavy tasks.
  • Accept multiple executors in the dfk
  • Launch multiple executors in the DFK from the config definition
  • Support for multiple IPP executors
  • Attach specific tasks to sites via enhancements to the decorator to take sites kwarg :
@App ('bash', dfk, sites=['local'])
def foo (x):
    return 'echo $(({0} * 2))

@App ('bash', dfk, sites='all' ) # Sites = 'all is the default
def sleep_compute (x) :
    return 'sleep x'

@yadudoc yadudoc added the enhancement label Dec 8, 2017

@yadudoc yadudoc added this to the Parsl-0.4.0 milestone Dec 8, 2017

@yadudoc yadudoc self-assigned this Dec 8, 2017

@yadudoc

This comment has been minimized.

Contributor

yadudoc commented Dec 12, 2017

Here's a test from the test_suite that shows how all of this would work :
https://github.com/Parsl/parsl/blob/multisite/parsl/tests/test_scaling/local_multisite/test_ipp_None_local.py

@yadudoc yadudoc referenced this issue Dec 19, 2017

Closed

Run directories #58

yadudoc added a commit that referenced this issue Dec 22, 2017

Adding Multi site tests. More tests needed but these tests should
demonstrate key functionality for #48, #58 and potentially fix #39

@yadudoc yadudoc closed this Dec 23, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment