Skip to content

add exclude modules argument to map#140

Merged
ooq merged 4 commits into
masterfrom
vaishaal/exclude_modules
Jul 7, 2017
Merged

add exclude modules argument to map#140
ooq merged 4 commits into
masterfrom
vaishaal/exclude_modules

Conversation

@Vaishaal

Copy link
Copy Markdown
Collaborator

Exclude modules in map based on prefix

Fix to #136

@ooq

ooq commented May 30, 2017

Copy link
Copy Markdown
Collaborator

Let's maybe have an example to document this?

@Vaishaal

Vaishaal commented May 31, 2017

Copy link
Copy Markdown
Collaborator Author

Here is an example

library.py:

import big_module # lets say this includes some stupid dependency tree that is like > 100 mb in size
def foo(x):
      return big_module.foo(x)

def bar(x):
       return x

driver.py:

import library 
pwex = pywren.default_executor()
pwex.map(library.bar, [0,1,2,3]) # will fail
pwex.map(library.bar, [0,1,2,3], exclude_modules=["big_module"]) # will work

@Vaishaal

Vaishaal commented Jun 1, 2017

Copy link
Copy Markdown
Collaborator Author

I've also added a test. @ericmjonas could you see if this is sufficient?

Comment thread tests/test_modules.py
fut = self.wrenexec.map(foo, [x], exclude_modules=["extmodule"])
try:
fut[0].result()
self.fail("shouldn't happen")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If exclusion does not work, there might still be some exception because extmodule does not really exist right?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean? In this case extmodule exists right?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you import it?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at the top. line 6.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah okay, I missed it.

@shivaram

shivaram commented Jul 7, 2017

Copy link
Copy Markdown
Collaborator

I think this looks fine to me. @ooq do you want to take one final look ?

@ooq

ooq commented Jul 7, 2017

Copy link
Copy Markdown
Collaborator

Looks good. Will merge.

@ooq
ooq merged commit 9363fe8 into master Jul 7, 2017
@Vaishaal
Vaishaal deleted the vaishaal/exclude_modules branch November 28, 2018 04:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants