How to handle definitions in the same file? #235
Description
I'm still playing around integrating pympipool with pyiron_workflow in this PR. I had been struggling with a couple of tests that hang local, and on the CI gave errors like ModuleNotFoundError: no module named 'unit' or no module named 'test_macro'.
I managed to whittle things down to this minimal(ish) working example:
Suppose I run a test file, e.g. test_pympipool.py, with the following contents:
from unittest import TestCase
from pympipool import PyMPIExecutor
def add_two(x):
return x + 2
class TestPyMPIExecutor(TestCase):
def test_local(self):
# Works perfectly
def add_one(x):
return x + 1
class Foo:
def add(self, x):
return add_one(x)
foo = Foo()
executor = PyMPIExecutor()
fs = executor.submit(foo.add, 1)
self.assertEqual(2, fs.result())
def test_semi_local(self):
# Hangs
# CI logs make it look like cloudpickle is trying to import from this module,
# but can't find it
class Foo:
def add(self, x):
return add_two(x)
foo = Foo()
executor = PyMPIExecutor()
fs = executor.submit(foo.add, 1)
self.assertEqual(3, fs.result())When the function and class are defined in exactly the same scope, everything works perfectly. Not shown but also working perfectly is if I imported the add_x function from some other accessible location. The only thing that seems to be breaking is when I make a local definition that relies on something else defined in a different scope but the same file.
In a very hand-wavey way this makes sense to me -- the executor sees that add_two is not defined in this scope and so decides its something that can be pickled by reference; the only problem is that the file it's defined in is not in my import path! In contrast, when add_one is defined in exactly the same scope as the object being pickled, it's clear to the code that this object also needs to be pickled by value.
My questions are:
- Should this count as a bug that can be fixed here in
pympipool, or is what I'm asking for unreasonable? In the latter case, what is the recommended syntax for same-file-different-scope definitions like this, i.e. how to succinctly make sure everything is added to path, or can I flag particular stuff for by-value treatment? - On the CI it's super helpful to get the hard crash and the import error message -- why is it only hanging on my local machine, and can we force a crash under these conditions there too?