Shared modules race condition in python3

[bsteffensmeier]

There is an error when using shared modules with multiple threads where the module can be shared before it has finished initializing. This results in missing attributes. The following code can be used to demonstrate the error.

import java.util.function.Consumer;
import jep.Jep;
import jep.JepConfig;

import java.lang.Thread;
import java.lang.Exception;


public class Test extends Thread{

    public static void main(String[] args) throws Throwable{
        Test[] t = new Test[10];
        for(int i = 0 ; i < t.length ; i += 1){
            t[i] = new Test();
            t[i].start();
        }
        for(int i = 0 ; i < t.length ; i += 1){
            t[i].join();
        }
    }

    @Override
    public void run() {
        try(Jep jep = new Jep(new JepConfig().addSharedModules("numpy"))){
            jep.eval("import numpy");
            jep.eval("numpy.array(1)");
        } catch (Exception e){
            e.printStackTrace();
        }
    }

}

The result of running the code:

jep.JepException: <class 'AttributeError'>: module 'numpy' has no attribute 'array'
	at <string>.<module>(<string>:1)
jep.JepException: <class 'AttributeError'>: module 'numpy' has no attribute 'array'
	at <string>.<module>(<string>:1)
jep.JepException: <class 'AttributeError'>: module 'numpy' has no attribute 'array'
	at <string>.<module>(<string>:1)
jep.JepException: <class 'AttributeError'>: module 'numpy' has no attribute 'array'
	at <string>.<module>(<string>:1)
jep.JepException: <class 'AttributeError'>: module 'numpy' has no attribute 'array'
	at <string>.<module>(<string>:1)
jep.JepException: <class 'AttributeError'>: module 'numpy' has no attribute 'array'
	at <string>.<module>(<string>:1)
jep.JepException: <class 'AttributeError'>: module 'numpy' has no attribute 'array'
	at <string>.<module>(<string>:1)
array(1)
array(1)
array(1)

As you can see the first 7 threads failed to lookup the attribute, however after numpy finished initializing the last 3 threads succeeded.

[pradeepvb]

Is it possible to enhance the module loader to just log a warning if the configured shared module is not available instead of throwing an exception?

jep.JepException: <class 'RuntimeError'>: jep.JepException: No module named 'scipy'
at /usr/lib64/python3.5/site-packages/jep/shared_modules_hook.load_module(shared_modules_hook.py:103)
at <frozen importlib._load_backward_compatible(:634)
at <frozen importlib._load_unlocked(:664)
at <frozen importlib._find_and_load_unlocked(:958)
at <frozen importlib._find_and_load(:969)
at .(:1)

[bsteffensmeier]

If I understand correctly then you made scipy a shared module but it is not found by the shared module hook? Which I assume means that scipy is not installed or at least not on the default PYTHONPATH?

The problem is that according to PEP-302 the loader must return a module or throw an exception. If the shared import fails and logs it then we don't have a module to return either.

[pradeepvb]

yes, that's correct.

Can this be handled inside Jep (Java layer) if the module loader has to follow PEP-302?

Please correct me if i have misunderstood.

[bsteffensmeier]

For the scipy issues you are seeing I think the current behavior is correct. If you tell jep that a module should be shared and that module is not found on the PYTHONPATH then the correct behavior is to throw an exception.

The only alternative I see would be using the include path for a specific jep instance to load a shared scipy but since that scipy would not necessarily be valid for a jep instance with a different path it may not be valid to share across all jep instances.

[bsteffensmeier]

The original problem with missing module attributes has been resolved on the dev_3.7 branch with 7eb9028

[bsteffensmeier]

Closing this because the original issue is fixed in dev_3,7 which has been merged into master.