New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add exclude modules argument to map #140

Merged
merged 4 commits into from Jul 7, 2017

Conversation

Projects
None yet
3 participants
@Vaishaal
Collaborator

Vaishaal commented May 17, 2017

Exclude modules in map based on prefix

Fix to #136

@ooq

This comment has been minimized.

Show comment
Hide comment
@ooq

ooq May 30, 2017

Collaborator

Let's maybe have an example to document this?

Collaborator

ooq commented May 30, 2017

Let's maybe have an example to document this?

@Vaishaal

This comment has been minimized.

Show comment
Hide comment
@Vaishaal

Vaishaal May 31, 2017

Collaborator

Here is an example

library.py:

import big_module # lets say this includes some stupid dependency tree that is like > 100 mb in size
def foo(x):
      return big_module.foo(x)

def bar(x):
       return x

driver.py:

import library 
pwex = pywren.default_executor()
pwex.map(library.bar, [0,1,2,3]) # will fail
pwex.map(library.bar, [0,1,2,3], exclude_modules=["big_module"]) # will work
Collaborator

Vaishaal commented May 31, 2017

Here is an example

library.py:

import big_module # lets say this includes some stupid dependency tree that is like > 100 mb in size
def foo(x):
      return big_module.foo(x)

def bar(x):
       return x

driver.py:

import library 
pwex = pywren.default_executor()
pwex.map(library.bar, [0,1,2,3]) # will fail
pwex.map(library.bar, [0,1,2,3], exclude_modules=["big_module"]) # will work
@Vaishaal

This comment has been minimized.

Show comment
Hide comment
@Vaishaal

Vaishaal Jun 1, 2017

Collaborator

I've also added a test. @ericmjonas could you see if this is sufficient?

Collaborator

Vaishaal commented Jun 1, 2017

I've also added a test. @ericmjonas could you see if this is sufficient?

fut = self.wrenexec.map(foo, [x], exclude_modules=["extmodule"])
try:
fut[0].result()
self.fail("shouldn't happen")

This comment has been minimized.

@ooq

ooq Jun 2, 2017

Collaborator

If exclusion does not work, there might still be some exception because extmodule does not really exist right?

@ooq

ooq Jun 2, 2017

Collaborator

If exclusion does not work, there might still be some exception because extmodule does not really exist right?

This comment has been minimized.

@Vaishaal

Vaishaal Jun 2, 2017

Collaborator

What do you mean? In this case extmodule exists right?

@Vaishaal

Vaishaal Jun 2, 2017

Collaborator

What do you mean? In this case extmodule exists right?

This comment has been minimized.

@ooq

ooq Jun 2, 2017

Collaborator

Did you import it?

@ooq

ooq Jun 2, 2017

Collaborator

Did you import it?

This comment has been minimized.

@Vaishaal

Vaishaal Jun 2, 2017

Collaborator

at the top. line 6.

@Vaishaal

Vaishaal Jun 2, 2017

Collaborator

at the top. line 6.

This comment has been minimized.

@ooq

ooq Jun 2, 2017

Collaborator

Ah okay, I missed it.

@ooq

ooq Jun 2, 2017

Collaborator

Ah okay, I missed it.

@shivaram

This comment has been minimized.

Show comment
Hide comment
@shivaram

shivaram Jul 7, 2017

Collaborator

I think this looks fine to me. @ooq do you want to take one final look ?

Collaborator

shivaram commented Jul 7, 2017

I think this looks fine to me. @ooq do you want to take one final look ?

@ooq

This comment has been minimized.

Show comment
Hide comment
@ooq

ooq Jul 7, 2017

Collaborator

Looks good. Will merge.

Collaborator

ooq commented Jul 7, 2017

Looks good. Will merge.

@ooq ooq merged commit 9363fe8 into master Jul 7, 2017

1 check passed

continuous-integration/travis-ci/push The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment