Could not find SavedModel .pb or .pbtxt at supplied export directory path

[shuDaoNan9]

My code works well on Windows10, but I can't reload the model on Linux by 'tf.SavedModelBundle.load(modelPath,"serve")':

[hadoop@ip-172-17-1-1 ~]$ spark-shell --master yarn --jars s3://jars/jasypt-1.9.2.jar,s3://jars/commons-pool2-2.0.jar,s3://jars/tensorflow-core-api-0.3.1.jar,s3://jars/tensorflow-core-api-0.3.1-linux-x86_64-mkl.jar,s3://jars/ndarray-0.3.1.jar,s3://jars/javacpp-1.5.4.jar,s3://jars/tensorflow-core-platform-0.3.1.jar
Spark context available as 'sc' (master = yarn, app id = application_1619156563605_0010).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.7-amzn-1
      /_/
         
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_282)
Type in expressions to have them evaluated.
Type :help for more information.

scala> import org.{tensorflow => tf}
import org.{tensorflow=>tf}

scala> val modelPath="s3://TF/model/xDeepFM"
modelPath: String = s3://TF/model/xDeepFM

scala> val bundle = tf.SavedModelBundle.load(modelPath,"serve")
Warning: Could not load Loader: java.lang.UnsatisfiedLinkError: no jnijavacpp in java.library.path
Warning: Could not load Pointer: java.lang.UnsatisfiedLinkError: no jnijavacpp in java.library.path
Warning: Could not load BytePointer: java.lang.UnsatisfiedLinkError: no jnijavacpp in java.library.path
Warning: Could not load PointerPointer: java.lang.UnsatisfiedLinkError: no jnijavacpp in java.library.path
2021-04-23 06:46:30.410844: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: s3://TF/model/xDeepFM
2021-04-23 06:48:15.648373: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: fail: Not found: Could not find SavedModel .pb or .pbtxt at supplied export directory path: s3://TF/model/xDeepFM. Took 105237551 microseconds.
org.tensorflow.exceptions.TensorFlowException: Could not find SavedModel .pb or .pbtxt at supplied export directory path: s3://TF/model/xDeepFM
  at org.tensorflow.internal.c_api.AbstractTF_Status.throwExceptionIfNotOK(AbstractTF_Status.java:101)
  at org.tensorflow.SavedModelBundle.load(SavedModelBundle.java:418)
  at org.tensorflow.SavedModelBundle.access$000(SavedModelBundle.java:59)
  at org.tensorflow.SavedModelBundle$Loader.load(SavedModelBundle.java:68)
  at org.tensorflow.SavedModelBundle.load(SavedModelBundle.java:242)
  ... 49 elided

scala> 

[hadoop@ip-172-17-1-1 ~]$ hadoop fs -ls  s3://TF/model/xDeepFM
Found 2 items
-rw-rw-rw-   1 hadoop hadoop    1598718 2021-04-23 06:33 s3://TF/model/xDeepFM/saved_model.pb
drwxrwxrwx   - hadoop hadoop          0 1970-01-01 00:00 s3://TF/model/xDeepFM/variables

Thanks!

[shuDaoNan9]

I tried to load a local model on Linux, and the error info changed:

[hadoop@ip-172-17-1-1 ~]$ spark-shell  --master local[2] --jars s3://jars/jasypt-1.9.2.jar,s3://jars/commons-pool2-2.0.jar,s3://jars/tensorflow-core-api-0.3.1.jar,s3://jars/tensorflow-core-api-0.3.1-linux-x86_64-mkl.jar,s3://jars/ndarray-0.3.1.jar,s3://jars/javacpp-1.5.4.jar,s3://jars/tensorflow-core-platform-0.3.1.jar,s3://jars/protobuf-java-3.8.0.jar
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.7-amzn-1
      /_/
         
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_282)
Type in expressions to have them evaluated.
Type :help for more information.

scala> import org.{tensorflow => tf}
import org.{tensorflow=>tf}

scala> val bundle = tf.SavedModelBundle.load("/home/hadoop/xDeepFM","serve")
2021-04-23 07:32:56.223881: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: /home/hadoop/xDeepFM
2021-04-23 07:32:56.266424: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve }
2021-04-23 07:32:56.266468: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: /home/hadoop/xDeepFM
2021-04-23 07:32:56.346757: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle.
2021-04-23 07:32:56.873838: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:190] Running initialization op on SavedModel bundle at path: /home/hadoop/xDeepFM
2021-04-23 07:32:56.928656: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 704788 microseconds.
java.lang.NoSuchMethodError: com.google.protobuf.Parser.parseFrom(Ljava/nio/ByteBuffer;)Ljava/lang/Object;
  at org.tensorflow.proto.framework.MetaGraphDef.parseFrom(MetaGraphDef.java:3067)
  at org.tensorflow.SavedModelBundle.load(SavedModelBundle.java:422)
  at org.tensorflow.SavedModelBundle.access$000(SavedModelBundle.java:59)
  at org.tensorflow.SavedModelBundle$Loader.load(SavedModelBundle.java:68)
  at org.tensorflow.SavedModelBundle.load(SavedModelBundle.java:242)
  ... 49 elided

[Craigacp]

It looks like your class path isn't right, you're missing the protobuf jars in the local load case.

In the s3 load case I'm not sure we compile the native tf binary with s3 support. You can modify the configuration and rebuild tf-java from source to fix that.

[shuDaoNan9]

It looks like your class path isn't right, you're missing the protobuf jars in the local load case.

In the s3 load case I'm not sure we compile the native tf binary with s3 support. You can modify the configuration and rebuild tf-java from source to fix that.

My protobuf-java-3.8.0.jar was added at 's3://jars/protobuf-java-3.8.0.jar'. By using '--jars' option, I added the dependencies while start running spark shell(scala). I can 'import com.google.protobuf.Parser' but spark(java or scala) can't find 'com.google.protobuf.Parser.parseFrom()' and I tried it in a single Java file ‘Testtf.java’, everything was right then.

By the way, I find I can load the model by java like that:

vi Testtf.java 

import org.tensorflow.SavedModelBundle;
public class Testtf {
    public static void main(String[] args) {
        System.out.println("test load...");
        SavedModelBundle bundle = org.tensorflow.SavedModelBundle.load("/home/hadoop/xDeepFM","serve");
        System.out.println("loaded bundle...");
        System.out.println(bundle);
    }
}

[hadoop@ip-172-17-1-1 ~]$ javac -cp javacpp-1.5.4-linux-x86_64.jar:tensorflow-core-api-0.3.1-linux-x86_64.jar:tensorflow-core-api-0.3.1.jar:ndarray-0.3.1.jar:javacpp-1.5.4.jar:tensorflow-core-platform-0.3.1.jar:tensorflow-core-platform-mkl-0.3.1.jar:protobuf-java-3.8.0.jar Testtf.java
[hadoop@ip-172-17-1-1 ~]$ java -cp javacpp-1.5.4-linux-x86_64.jar:tensorflow-core-api-0.3.1-linux-x86_64.jar:tensorflow-core-api-0.3.1.jar:ndarray-0.3.1.jar:javacpp-1.5.4.jar:tensorflow-core-platform-0.3.1.jar:tensorflow-core-platform-mkl-0.3.1.jar:protobuf-java-3.8.0.jar:. Testtf
test load...
2021-04-25 02:33:47.247120: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: /home/hadoop/xDeepFM
2021-04-25 02:33:47.275349: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve }
2021-04-25 02:33:47.275405: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: /home/hadoop/xDeepFM
2021-04-25 02:33:47.275480: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-25 02:33:47.370208: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle.
2021-04-25 02:33:47.381656: I external/org_tensorflow/tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2300065000 Hz
2021-04-25 02:33:47.667941: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:190] Running initialization op on SavedModel bundle at path: /home/hadoop/xDeepFM
2021-04-25 02:33:47.725458: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 478354 microseconds.
loaded bundle...
org.tensorflow.SavedModelBundle@80503
[hadoop@ip-172-17-1-1 ~]$ 

[Craigacp]

Yeah it looks like there is a dependency conflict between spark's version of protobuf and the one TensorFlow uses. Shading all of TF might work, or upgrading spark if that's feasible.

[shuDaoNan9]

Yeah it looks like there is a dependency conflict between spark's version of protobuf and the one TensorFlow uses. Shading all of TF might work, or upgrading spark if that's feasible.

Yes, I tried Amazon flink just now, and the model was loaded correctly. So, I waiting for 'Amazon Spark'
team's solution now.

[shuDaoNan9]

I tried it on Ubuntu( Original spark ), and every was right then! it looks like Amazon spark or cdh spark changed something of spark which cause this error.
Thanks to Amazon team's help. They find the reason is Amazon emr's default protobuf-java-[version].jar is too old. The solution is update it's version.

Thank you all!

[feifeiontheway]

@JWenBin 您好，请问 protobuf-java- [version]需要更新到那个版本？

[shuDaoNan9]

@JWenBin 您好，请问 protobuf-java- [version]需要更新到那个版本？

protobuf-java-3.8.0.jar

[desertzebra]

Hi,

I had the same issue running Spark 3.2.0 with scala 2.13.5 on Hadoop 3.3.1 in cluster mode on yarn.

Since I had the custom setup so Amazon had nothing to do with me. I fixed the problem with my setup, by backing up the protobuf jar at my installation point:

$ mv /usr/local/spark/jars/protobuf-java-2.5.0.jar /usr/local/spark/jars/protobuf-java-2.5.0.jar.bk

and copying the newer protobuf jar (in my case protobuf-java-3.19.3.jar) into the same folder.

Nothing else worked for me. Even sending the protobuf jar in with --jars didnt work because I suspect spark gives priority to the jars in its own folder, rather than the ones passed as an argument.

P.S. found this configuration after resolving my problem:

spark.driver.userClassPathFirst (default: false) | (Experimental) Whether to give user-added jars precedence over Spark's own jars when loading classes in the driver. This feature can be used to mitigate conflicts between Spark's dependencies and user dependencies. It is currently an experimental feature. This is used in cluster mode only.

[feifeiontheway]

您好，      来信已经收到，我将尽快查阅，给您回复！                                               陈飞