Skip to content

add exclude modules argument to map #140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jul 7, 2017
Merged

add exclude modules argument to map #140

merged 4 commits into from
Jul 7, 2017

Conversation

Vaishaal
Copy link
Collaborator

Exclude modules in map based on prefix

Fix to #136

@ooq
Copy link
Collaborator

ooq commented May 30, 2017

Let's maybe have an example to document this?

@Vaishaal
Copy link
Collaborator Author

Vaishaal commented May 31, 2017

Here is an example

library.py:

import big_module # lets say this includes some stupid dependency tree that is like > 100 mb in size
def foo(x):
      return big_module.foo(x)

def bar(x):
       return x

driver.py:

import library 
pwex = pywren.default_executor()
pwex.map(library.bar, [0,1,2,3]) # will fail
pwex.map(library.bar, [0,1,2,3], exclude_modules=["big_module"]) # will work

@Vaishaal
Copy link
Collaborator Author

Vaishaal commented Jun 1, 2017

I've also added a test. @ericmjonas could you see if this is sufficient?

fut = self.wrenexec.map(foo, [x], exclude_modules=["extmodule"])
try:
fut[0].result()
self.fail("shouldn't happen")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If exclusion does not work, there might still be some exception because extmodule does not really exist right?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean? In this case extmodule exists right?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you import it?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at the top. line 6.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah okay, I missed it.

@shivaram
Copy link
Collaborator

shivaram commented Jul 7, 2017

I think this looks fine to me. @ooq do you want to take one final look ?

@ooq
Copy link
Collaborator

ooq commented Jul 7, 2017

Looks good. Will merge.

@ooq ooq merged commit 9363fe8 into master Jul 7, 2017
@Vaishaal Vaishaal deleted the vaishaal/exclude_modules branch November 28, 2018 04:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants