After loading the JEP library, it fails with "No module named 'jep'" on OSX

[dportabella]

I managed to use JEP on Ubuntu:
https://github.com/dportabella/scala-python-example/tree/fix

but I fail to make it work on OSX 10.12.

First I install the JEP library for OSX as follows:

pip install jep
# http://www.macworld.com/article/2986118/security/how-to-modify-system-integrity-protection-in-el-capitan.html
sudo cp /Users/david/.virtualenvs/py3-data/lib/python3.6/site-packages/jep/libjep.jnilib /Library/Java/Extensions/

And then I try to run a simple test, Test.java:

public class Test {
  public static void main(String[] args) throws Exception {  
    System.loadLibrary("jep");  // just for testing that it is being loaded
    System.out.println("+++ jep library loaded!");
    new jep.Jep();
  }
}

as follows:

$ javac -cp /Users/david/.virtualenvs/py3-data/lib/python3.6/site-packages/jep/jep-3.6.3.jar Test.java
$ java -cp .:/Users/david/.virtualenvs/py3-data/lib/python3.6/site-packages/jep/jep-3.6.3.jar Test
+++ jep library loaded!
Exception in thread "main" jep.JepException: <class 'ModuleNotFoundError'>: No module named 'jep'
	at <string>.<module>(<string>:1)

So, it loads the library, but then it fails with No module named 'jep'.

Any idea?

[dportabella]

Instead of installing JEP using pip, I also tried building it from the source code:

$ git clone https://github.com/mrj0/jep.git
$ cd jep
$ python setup.py build install

But I get the same error when running Test.java: No module named 'jep'

[dportabella]

Solved, by setting up the PYTHONHOME env, as proposed here:
sushant-hiray/scala-python-example#3

[ndjensen]

Thank you for sharing your solution.

[dukkona]

Hello,
I have the same problem, setting up the PYTHONHOME solved the No module named 'jep' problem, but it caused another quite common problem:

Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

I'm trying to use Jep with Spark, on a Yarn cluster, with Jep installed in a python virtualenvironment, my spark-submit looks like this:

spark-submit \
--deploy-mode cluster \
--master yarn \
--conf spark.yarn.maxAppAttempts=1 \
--conf spark.yarn.dist.archives=hdfs://<hdfs_path>/jep_venv_36.tar.gz#environment \
--conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./environment/bin/python \
--conf spark.yarn.executorEnv.PYSPARK_PYTHON=./environment/bin/python \
--conf spark.yarn.appMasterEnv.PYTHONHOME=./environment/bin/python \
--conf spark.yarn.executorEnv.PYTHONHOME=./environment/bin/python \
--conf spark.yarn.appMasterEnv.JAVA_LIBRARY_PATH=./environment/lib/python3.6/site-packages/jep \
--conf spark.yarn.executorEnv.JAVA_LIBRARY_PATH=./environment/lib/python3.6/site-packages/jep \
--conf spark.yarn.appMasterEnv.LD_LIBRARY_PATH=./environment/lib/python3.6/site-packages/jep  \
--conf spark.yarn.executorEnv.LD_LIBRARY_PATH=./environment/lib/python3.6/site-packages/jep \
--class <main_class> \
<script>.jar

Does anybody have any idea how to submit the job correctly to avoid both problems?

[bsteffensmeier]

The PYTHONHOME env var is compatible with a virtualenvironment, typically the activation script will unset this. Are you calling the activate script before running your program?

[dukkona]

No, I don't call explicitly the activation script before the program execution