'JavaPackage' object is not callable when importing bigdl layers

[raphaelrbr]

I am having trouble importing bigdl keras layers
from bigdl.nn.keras.layer import Input, Dense, merge, Convolution1D, MaxPooling1D, Reshape, Flatten

I am receiving

TypeErrorTraceback (most recent call last)
<ipython-input-5-06f013d33c29> in <module>()
----> 1 from bigdl.nn.keras.layer import Input, Dense, merge, Convolution1D, MaxPooling1D, Reshape, Flatten

/opt/anaconda2/lib/python2.7/site-packages/bigdl/nn/keras/__init__.py in <module>()
     16 from bigdl.util.common import *
     17 
---> 18 init_engine()
     19 redire_spark_logs()
     20 show_bigdl_info_logs()

/opt/anaconda2/lib/python2.7/site-packages/bigdl/util/common.pyc in init_engine(bigdl_type)
    415 
    416 def init_engine(bigdl_type="float"):
--> 417     callBigDlFunc(bigdl_type, "initEngine")
    418     # Spark context is supposed to have been created when init_engine is called
    419     get_spark_context()._jvm.org.apache.spark.bigdl.api.python.BigDLSerDe.initialize()

/opt/anaconda2/lib/python2.7/site-packages/bigdl/util/common.pyc in callBigDlFunc(bigdl_type, name, *args)
    577     gateway = _get_gateway()
    578     error = Exception("Cannot find function: %s" % name)
--> 579     for jinvoker in JavaCreator.instance(bigdl_type, gateway).value:
    580         # hasattr(jinvoker, name) always return true here,
    581         # so you need to invoke the method to check if it exist or not

/opt/anaconda2/lib/python2.7/site-packages/bigdl/util/common.pyc in instance(cls, bigdl_type, *args)
     54             with cls._lock:
     55                 if not cls._instance:
---> 56                     cls._instance = cls(bigdl_type, *args)
     57         return cls._instance
     58 

/opt/anaconda2/lib/python2.7/site-packages/bigdl/util/common.pyc in __init__(self, bigdl_type, gateway)
     91             jclass = getattr(gateway.jvm, creator_class)
     92             if bigdl_type == "float":
---> 93                 self.value.append(getattr(jclass, "ofFloat")())
     94             elif bigdl_type == "double":
     95                 self.value.append(getattr(jclass, "ofDouble")())

TypeError: 'JavaPackage' object is not callable

I've been reading about this issue and thinking if maybe is a environment variable problem, but i'm not having sucess debugging this.

I tried to set BIGDL_JAR variable as a system environment variable but didn't work.

Someone can help me?

[raphaelrbr]

This is my spark conf

sparkConf = SparkConf() \
    .setAppName('Test') \
    .setAll([ \
#            ("spark.dynamicAllocation.enabled", "true"), 
#            ("spark.shuffle.service.enabled", "true")          
             
             ("spark.dynamicAllocation.enabled", "false"), 
             ("spark.shuffle.service.enabled", "false"),
             ("spark.executor.memory", "15g"), 
             ("spark.submit.deployMode", "client"),
             ("spark.driver.memory", "15g"), 
             ("spark.executor.instances", "75"),
             ("spark.executor.cores", "32") 
             
             
             ]) \
    .setMaster("yarn") 


sc = SparkContext(appName="Neural Network", conf=sparkConf)

[zhangxiaoli73]

hi, @raphaelrbr it seems you have not install bigdl~
please refer to following page for how to set bigdl jar.
without pip install: https://bigdl-project.github.io/0.7.0/#PythonUserGuide/run-without-pip/#precondition
with pip install: https://bigdl-project.github.io/0.7.0/#PythonUserGuide/run-from-pip/#precondition

[yeamusic21]

In the documentation you have the location as (...) for BIGDL_JARS and BIGDL_PACKAGES. Where can we find these jars?

export BIGDL_JARS=...
export BIGDL_PACKAGES=...

Thanks :)

[yeamusic21]

I tried to set BIGDL_JAR variable as a system environment variable but didn't work.

I tried the same thing and still getting the same error. I also followed the instructions on the documentation page, installed via pip, and then downloaded the jars from Release 0.7.0 and pointed to them in ...

export BIGDL_JARS=...
export BIGDL_PACKAGES=...

And this still does not work for me. Does anyone have any updates?

https://bigdl-project.github.io/0.7.0/#release-download/
https://bigdl-project.github.io/0.7.0/#PythonUserGuide/run-from-pip/#precondition

[yeamusic21]

I think this answers my question

https://groups.google.com/forum/#!msg/bigdl-user-group/xIzBGfHmn-A/PREWlj_HCgAJ

if installing from pip you use python command not spark-submit.

You can use python to start a REPL and call from zoo.common.nncontext import *&sc = init_nncontext() to create a SparkContext.

And most importantly

Pip install only supports local mode.

Would be interesting if I understand this correctly.

[wzhongyuan]

@yeamusic21 please let me know if you still face this issue or have any question ?