//tensorflow/tools/api/tests:api_compatibility_test fails with protobuf > 3.20.3 #59932

elfringham · 2023-03-08T10:59:57Z

Click to expand!

Issue Type

Bug

Have you reproduced the bug with TF nightly?

Yes

Source

source

Tensorflow Version

git HEAD

Custom Code

No

OS Platform and Distribution

Ubuntu 20.04

Mobile device

n/a

Python version

3.9.16

Bazel version

5.3.0

GCC/Compiler version

9.4.0

CUDA/cuDNN version

n/a

GPU model and memory

n/a

Current Behaviour?

The test fails

Standalone code to reproduce the issue

bazel test --test_timeout=300,500,-1,-1 --flaky_test_attempts=3 --test_output=all --cache_test_results=no --noremote_accept_cached --test_env=TF_ENABLE_ONEDNN_OPTS=1 --build_tag_filters=-no_oss,-oss_serial,-gpu,-tpu,-benchmark-test,-v1only,-requires-gpu --test_tag_filters=-no_oss,-oss_serial,-gpu,-tpu,-benchmark-test,-v1only,-requires-gpu --verbose_failures --build_tests_only --jobs=16 -- //tensorflow/tools/api/tests:api_compatibility_test

Relevant log output

======================================================================
FAIL: testAPIBackwardsCompatibility (__main__.ApiCompatibilityTest)
ApiCompatibilityTest.testAPIBackwardsCompatibility
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 511, in testAPIBackwardsCompatibility
    self._checkBackwardsCompatibility(
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 488, in _checkBackwardsCompatibility
    self._AssertProtoDictEquals(
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 395, in _AssertProtoDictEquals
    self.fail('%d differences found between API and golden.' % diff_count)
AssertionError: 1 differences found between API and golden.

======================================================================
FAIL: testAPIBackwardsCompatibilityV1 (__main__.ApiCompatibilityTest)
ApiCompatibilityTest.testAPIBackwardsCompatibilityV1
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 528, in testAPIBackwardsCompatibilityV1
    self._checkBackwardsCompatibility(
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 488, in _checkBackwardsCompatibility
    self._AssertProtoDictEquals(
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 395, in _AssertProtoDictEquals
    self.fail('%d differences found between API and golden.' % diff_count)
AssertionError: 4 differences found between API and golden.

======================================================================
FAIL: testAPIBackwardsCompatibilityV2 (__main__.ApiCompatibilityTest)
ApiCompatibilityTest.testAPIBackwardsCompatibilityV2
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 550, in testAPIBackwardsCompatibilityV2
    self._checkBackwardsCompatibility(
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 488, in _checkBackwardsCompatibility
    self._AssertProtoDictEquals(
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 395, in _AssertProtoDictEquals
    self.fail('%d differences found between API and golden.' % diff_count)
AssertionError: 1 differences found between API and golden.

----------------------------------------------------------------------

SuryanarayanaY · 2023-03-13T13:55:15Z

Hi @elfringham ,

I tried the same commands but my build is success and test also passes.Please refer to attached log below.Please check and confirm if any deviation.Please note I have not installed any protobuf specifically but gone with default from master branch.I observed 3.19.6 is default protobuf version with Pip install and same might be considered by Bazel. Hence for my case there is no error right?

(bazel) suryanarayanay@surya-ubuntu20:~/tensorflow$ bazel test --test_timeout=300,500,-1,-1 --flaky_test_attempts=3 --test_output=all --cache_test_results=no --noremote_accept_cached --test_env=TF_ENABLE_ONEDNN_OPTS=1 --build_tag_filters=-no_oss,-oss_serial,-gpu,-tpu,-benchmark-test,-v1only,-requires-gpu --test_tag_filters=-no_oss,-oss_serial,-gpu,-tpu,-benchmark-test,-v1only,-requires-gpu --verbose_failures --build_tests_only --jobs=16 -- //tensorflow/python/util/protobuf:protobuf_compare_test
WARNING: Running Bazel server needs to be killed, because the startup options are different.
Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=100
INFO: Reading rc options for 'test' from /home/suryanarayanay/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'test' from /home/suryanarayanay/tensorflow/.bazelrc:
  Inherited 'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils
INFO: Found applicable config definition build:short_logs in file /home/suryanarayanay/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /home/suryanarayanay/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:linux in file /home/suryanarayanay/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --distinct_host_configuration=false --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /home/suryanarayanay/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/XNNPACK/archive/5eff7fb0cf49215d9f68646512797a2258aa00cc.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
INFO: Analyzed target //tensorflow/python/util/protobuf:protobuf_compare_test (555 packages loaded, 34029 targets configured).
INFO: Found 1 test target...
INFO: From Testing //tensorflow/python/util/protobuf:protobuf_compare_test:
==================== Test output for //tensorflow/python/util/protobuf:protobuf_compare_test:
2023-03-13 12:55:09.249927: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-03-13 12:55:09.298935: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Running tests under Python 3.9.16: /home/suryanarayanay/miniconda3/envs/bazel/bin/python3
[ RUN      ] AssertTest.testAssertEqualWithStringArg
[       OK ] AssertTest.testAssertEqualWithStringArg
[ RUN      ] AssertTest.testCheckInitialized
[       OK ] AssertTest.testCheckInitialized
[ RUN      ] AssertTest.testLargeProtoData
[       OK ] AssertTest.testLargeProtoData
[ RUN      ] AssertTest.testMessage
[       OK ] AssertTest.testMessage
[ RUN      ] AssertTest.testMsgPassdown
[       OK ] AssertTest.testMsgPassdown
[ RUN      ] AssertTest.testNormalizesFloat
[       OK ] AssertTest.testNormalizesFloat
[ RUN      ] AssertTest.testNormalizesNumbers
[       OK ] AssertTest.testNormalizesNumbers
[ RUN      ] AssertTest.testPrimitives
[       OK ] AssertTest.testPrimitives
[ RUN      ] AssertTest.testRepeatedMessage
[       OK ] AssertTest.testRepeatedMessage
[ RUN      ] AssertTest.testRepeatedPrimitives
[       OK ] AssertTest.testRepeatedPrimitives
[ RUN      ] MixinTests.testAssertEqualWithStringArg
[       OK ] MixinTests.testAssertEqualWithStringArg
[ RUN      ] NormalizeNumbersTest.testNormalizesDoubles
[       OK ] NormalizeNumbersTest.testNormalizesDoubles
[ RUN      ] NormalizeNumbersTest.testNormalizesFloats
[       OK ] NormalizeNumbersTest.testNormalizesFloats
[ RUN      ] NormalizeNumbersTest.testNormalizesInts
[       OK ] NormalizeNumbersTest.testNormalizesInts
[ RUN      ] NormalizeNumbersTest.testNormalizesMaps
[       OK ] NormalizeNumbersTest.testNormalizesMaps
[ RUN      ] NormalizeNumbersTest.testNormalizesRepeatedFloats
[       OK ] NormalizeNumbersTest.testNormalizesRepeatedFloats
[ RUN      ] NormalizeNumbersTest.testNormalizesRepeatedInts
[       OK ] NormalizeNumbersTest.testNormalizesRepeatedInts
[ RUN      ] ProtoEqTest.testEmpty
[       OK ] ProtoEqTest.testEmpty
[ RUN      ] ProtoEqTest.testIsClose
[       OK ] ProtoEqTest.testIsClose
[ RUN      ] ProtoEqTest.testMessage
[       OK ] ProtoEqTest.testMessage
[ RUN      ] ProtoEqTest.testNans
[       OK ] ProtoEqTest.testNans
[ RUN      ] ProtoEqTest.testNansMap
[       OK ] ProtoEqTest.testNansMap
[ RUN      ] ProtoEqTest.testNansRepeatedField
[       OK ] ProtoEqTest.testNansRepeatedField
[ RUN      ] ProtoEqTest.testNestedMessage
[       OK ] ProtoEqTest.testNestedMessage
[ RUN      ] ProtoEqTest.testPrimitiveFields
[       OK ] ProtoEqTest.testPrimitiveFields
[ RUN      ] ProtoEqTest.testPrimitives
[       OK ] ProtoEqTest.testPrimitives
[ RUN      ] ProtoEqTest.testRepeatedPrimitives
[       OK ] ProtoEqTest.testRepeatedPrimitives
[ RUN      ] ProtoEqTest.testTagOrder
[       OK ] ProtoEqTest.testTagOrder
----------------------------------------------------------------------
Ran 28 tests in 0.363s

OK
================================================================================
Target //tensorflow/python/util/protobuf:protobuf_compare_test up-to-date:
  bazel-bin/tensorflow/python/util/protobuf/protobuf_compare_test
INFO: Elapsed time: 5496.134s, Critical Path: 456.21s
INFO: 14759 processes: 1651 internal, 13108 local.
INFO: Build completed successfully, 14759 total actions
//tensorflow/python/util/protobuf:protobuf_compare_test                  PASSED in 1.2s

INFO: Build completed successfully, 14759 total actions
(bazel) suryanarayanay@surya-ubuntu20:~/tensorflow$

elfringham · 2023-03-13T14:24:17Z

Default install of protobuf is 4.22.1, see https://pypi.org/project/protobuf/ - I do not understand where you got 3.19.6 from.
Supported versions of protobuf for use with TensorFlow are shown here

tensorflow/tensorflow/tools/pip_package/setup.py

Line 100 in 3ad732b

'protobuf>=3.20.3,<5.0.0dev,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5',

which shows a minimum version of 3.20.3 but also includes 4.x with some exceptions early in the 4.x range.
The important thing is that protobuf versions > 3.20.3 are supported for use with TensorFlow and this test will fail with those versions. The test should pass with all supported versions.
Your test with protobuf==3.19.6 is not a supported configuration.
Yes there is an error.

SuryanarayanaY · 2023-03-14T11:43:33Z

@elfringham ,

Actually earlier I did not downloaded any protobuf version explicitly. I tried to test the command by installing latest protobuf version (4.22.1) now and the test is failing.The log is attached below for reference.

#59932.log

elfringham · 2023-03-14T11:49:04Z

@SuryanarayanaY You are missing an install of keras and tensorflow-estimator. Or alternatively keras-nightly and tf-estimator-nightly.

SuryanarayanaY · 2023-03-14T17:20:44Z

@elfringham ,
Thanks for that.It seems I actually missed this step.I will crosscheck all the dependencies mentioned in setup.py and update.

SuryanarayanaY · 2023-03-15T12:02:59Z

@elfringham ,

I am unable to replicate the reported problem.For me build is success and test also passed. I am attaching the logs for your review. Please have a look and confirm if anything missing. Thanks!
#59932-r1.txt

elfringham · 2023-03-15T18:43:03Z

Hi @SuryanarayanaY it is hard to see what is the difference in our environments that is causing your test to pass. Your use of the 'pip' command masks which version of Python is used and that may not be the same as the Python that is picked up by the TensorFlow test build. Also as you have not run 'configure' the log does not show which Python is in use. Also there may be environment variables such as PYTHON_BIN_PATH or PYTHON_LIB_PATH that affects things.
I am trying to put a gist together to show the issue but that may be tomorrow before its ready.

SuryanarayanaY · 2023-03-16T03:58:11Z

Thanks @elfringham ,I Will definitely wait for your gist.

elfringham · 2023-03-16T14:37:11Z

api_compat_test_fail.zip
Attached log of fail with Python pip etc commands at beginning to show that protobuf 4.22.1 was indeed installed into the place that TensorFlow used in its build.
I think the gist is not going to happen, sorry, I only have a free account and its going to hit the runtime limit well before it finishes.

elfringham · 2023-03-16T14:40:10Z

You can see the gist here https://colab.research.google.com/drive/1r8ssFOenTKYlWf0Ca00y2ffg8ifq4ntN?usp=sharing in case you can run it yourself with more resource privileges.

SuryanarayanaY · 2023-03-17T15:54:40Z

@elfringham ,

I too don't have privilege access to google colab.It will time out before completion. I can test only in VM for now. I am surprised that with same environment I replicated the issue mentioned at #59931 but not here. Could you please refer the issue there and confirm whether any changes

If you ask me I can test with python==3.9.16 explicitly using .configure step and will give it a try.

elfringham · 2023-03-17T16:20:17Z

@SuryanarayanaY I found the three issues, #59930 , #59931 and this one #59932 all at the same time with the same environment. So I too am surprised that you see different behaviour between #59931 and here. For me they all pass if I install protobuf==3.20.3 and all fail with protobuf > 3.20.3. I only entered three separate issues as they all have different errors.

SuryanarayanaY · 2023-03-18T05:28:10Z

@elfringham ,

I have used same build that has been used for 59931 and tested all the three reported issues 59930,59931 and 59932 and here also 59932 is success but remaining two tests failed with same error as reported by you. I am attaching the log below for all three in same file.

It shows Python 3.9.16 being used for running tests.But I noticed one test being skipped automatically.

Please check the tests(6/7) that are executed.

Running tests under Python 3.9.16: /home/suryanarayanay/miniconda3/envs/bazel/bin/python3
[ RUN      ] ApiCompatibilityTest.testAPIBackwardsCompatibility
INFO:tensorflow:No differences found between API and golden.
I0318 04:49:06.249608 140189728887424 api_compatibility_test.py:398] No differences found between API and golden.
INFO:tensorflow:time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibility): 1.03s
I0318 04:49:06.251941 140189728887424 test_util.py:2467] time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibility): 1.03s
[       OK ] ApiCompatibilityTest.testAPIBackwardsCompatibility
[ RUN      ] ApiCompatibilityTest.testAPIBackwardsCompatibilityV1
WARNING:tensorflow:From /home/suryanarayanay/.cache/bazel/_bazel_suryanarayanay/e93ef42332b258ebf8397106d59e39e3/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/python/util/decorator_utils.py:153: GraphKeys.VARIABLES (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.GraphKeys.GLOBAL_VARIABLES` instead.
W0318 04:49:06.337129 140189728887424 deprecation.py:364] From /home/suryanarayanay/.cache/bazel/_bazel_suryanarayanay/e93ef42332b258ebf8397106d59e39e3/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/python/util/decorator_utils.py:153: GraphKeys.VARIABLES (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.GraphKeys.GLOBAL_VARIABLES` instead.
INFO:tensorflow:No differences found between API and golden.
I0318 04:49:07.151268 140189728887424 api_compatibility_test.py:398] No differences found between API and golden.
INFO:tensorflow:time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibilityV1): 0.9s
I0318 04:49:07.154043 140189728887424 test_util.py:2467] time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibilityV1): 0.9s
[       OK ] ApiCompatibilityTest.testAPIBackwardsCompatibilityV1
[ RUN      ] ApiCompatibilityTest.testAPIBackwardsCompatibilityV2
INFO:tensorflow:No differences found between API and golden.
I0318 04:49:07.826159 140189728887424 api_compatibility_test.py:398] No differences found between API and golden.
INFO:tensorflow:time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibilityV2): 0.67s
I0318 04:49:07.829506 140189728887424 test_util.py:2467] time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibilityV2): 0.67s
[       OK ] ApiCompatibilityTest.testAPIBackwardsCompatibilityV2
[ RUN      ] ApiCompatibilityTest.testNoSubclassOfMessage
INFO:tensorflow:time(__main__.ApiCompatibilityTest.testNoSubclassOfMessage): 0.28s
I0318 04:49:08.110209 140189728887424 test_util.py:2467] time(__main__.ApiCompatibilityTest.testNoSubclassOfMessage): 0.28s
[       OK ] ApiCompatibilityTest.testNoSubclassOfMessage
[ RUN      ] ApiCompatibilityTest.testNoSubclassOfMessageV1
INFO:tensorflow:time(__main__.ApiCompatibilityTest.testNoSubclassOfMessageV1): 0.13s
I0318 04:49:08.236025 140189728887424 test_util.py:2467] time(__main__.ApiCompatibilityTest.testNoSubclassOfMessageV1): 0.13s
[       OK ] ApiCompatibilityTest.testNoSubclassOfMessageV1
[ RUN      ] ApiCompatibilityTest.testNoSubclassOfMessageV2
INFO:tensorflow:time(__main__.ApiCompatibilityTest.testNoSubclassOfMessageV2): 0.13s

After above log , I observed in the below immediate log where one of the test being skipped.

INFO:tensorflow:time(__main__.ApiCompatibilityTest.testNoSubclassOfMessageV2): 0.13s
I0318 04:49:08.367611 140189728887424 test_util.py:2467] time(__main__.ApiCompatibilityTest.testNoSubclassOfMessageV2): 0.13s
[       OK ] ApiCompatibilityTest.testNoSubclassOfMessageV2
[ RUN      ] ApiCompatibilityTest.test_session
[  SKIPPED ] ApiCompatibilityTest.test_session
----------------------------------------------------------------------
Ran 7 tests in 3.145s

OK (skipped=1)

The same test being skipped in earlier build also where I have attached logs in comment-1469880441.

Could you please cross check whether the above skipped test in your environment also skipped or executed?

Thanks!

elfringham · 2023-03-20T10:29:21Z

I see the same skipped test. The failures I see are in testAPIBackwardsCompatibility, testAPIBackwardsCompatibilityV1 and testAPIBackwardsCompatibilityV2. Please see the log I attached above.

[ RUN      ] ApiCompatibilityTest.test_session
[  SKIPPED ] ApiCompatibilityTest.test_session
======================================================================
FAIL: testAPIBackwardsCompatibility (__main__.ApiCompatibilityTest)
ApiCompatibilityTest.testAPIBackwardsCompatibility
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 511, in testAPIBackwardsCompatibility
    self._checkBackwardsCompatibility(
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 488, in _checkBackwardsCompatibility
    self._AssertProtoDictEquals(
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 395, in _AssertProtoDictEquals
    self.fail('%d differences found between API and golden.' % diff_count)
AssertionError: 1 differences found between API and golden.

======================================================================
FAIL: testAPIBackwardsCompatibilityV1 (__main__.ApiCompatibilityTest)
ApiCompatibilityTest.testAPIBackwardsCompatibilityV1
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 528, in testAPIBackwardsCompatibilityV1
    self._checkBackwardsCompatibility(
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 488, in _checkBackwardsCompatibility
    self._AssertProtoDictEquals(
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 395, in _AssertProtoDictEquals
    self.fail('%d differences found between API and golden.' % diff_count)
AssertionError: 4 differences found between API and golden.

======================================================================
FAIL: testAPIBackwardsCompatibilityV2 (__main__.ApiCompatibilityTest)
ApiCompatibilityTest.testAPIBackwardsCompatibilityV2
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 550, in testAPIBackwardsCompatibilityV2
    self._checkBackwardsCompatibility(
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 488, in _checkBackwardsCompatibility
    self._AssertProtoDictEquals(
  File "/root/.cache/bazel/_bazel_root/e953b164f58eb4c9598ad736d787ff39/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 395, in _AssertProtoDictEquals
    self.fail('%d differences found between API and golden.' % diff_count)
AssertionError: 1 differences found between API and golden.

----------------------------------------------------------------------
Ran 7 tests in 6.893s

FAILED (failures=3, skipped=1)

SuryanarayanaY · 2023-03-21T05:20:55Z

CC- @learning-to-play

learning-to-play · 2023-03-22T05:25:36Z

Hi @vam-google, Could you please take a look?

SuryanarayanaY · 2023-04-03T08:34:38Z

I have cross checked with r2.12 branch and here also the test passed for me.Please refer to attached logs below.

59932_r2.12_logs.txt

vam-google · 2023-04-03T19:59:26Z

I'm struggling to understand what is going on in all three bugs reported here.
Do the same tests pass in our CI?

If during the tests it somehow got resolved to <= protobuf-3.19 it means already is something wrong in the setup, as it should have failed earlier than running the test itself. On a proper python environment there should be no protobuf of version older than 3.20.3. I'm confused why protobuf 3.19 is being discussed and tried here, as it is irrelevant, that version is not supported so it does not matter if it works or not (it may work for most cases because 3.19 is actually the first version of prorotobuf with compatibility leayer between 3.x and 4.x versions, but we chose to not support it for TF for various reasons).

Currently TF bazel build is not hermetic in terms of python dependencies (we are working on fixing it), so whatever gets actually installed on your machine will be used by your TF build.

In terms of compatibility. Honestly I don't know what those failing tests are testing for, but in general protobuf 3.18, 3.19, 3.20 and 4.21 are very different versions, and they represent a transition path to a non-compatible versions (4.21 is not compatible with 3.18). So in principle some breackage in terms of compatibility is logical here.

One other important piece: we depend on protobuf in both phases: generaiton/compilation and runtime. This is important because generation time is what is responsible for creation of those *_pb2.py stub files. TF build currently uses protoc 3.21.9 which corresponds to protobuf 4.21.9 runtime dependency. The stubs produced by protoc 3.21.x are not compatible with older versions of protobuf runtime and in general the files themselves (the _pb2.py ones) look different, so it could be related.

elfringham · 2023-04-04T14:17:22Z

Theses tests do pass the CI but that is because that is run using protobuf==3.20.3, see

tensorflow/tensorflow/tools/tf_sig_build_dockerfiles/devel.requirements.txt

Line 18 in 968a8dc

protobuf ~= 3.20.3

Yes the tests using 3.19 are irrelevant, it is not supported.
The unit tests that fail are being run in the source tree and so of course they are using the same protobuf for the build and the test. This is not testing a wheel installed into a separate environment.
My point is that TensorFlow claims to be compatible with protobuf 4.x and yet it fails these tests when protobuf > 3.20.3 is installed.

tensorflow/tensorflow/tools/pip_package/setup.py

Line 100 in 025f1b2

    
           'protobuf>=3.20.3,<5.0.0dev,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5',

google-ml-butler bot added the type:bug Bug label Mar 8, 2023

google-ml-butler bot assigned pjpratik Mar 8, 2023

pjpratik added subtype: ubuntu/linux Ubuntu/Linux Build/Installation Issues comp:apis Highlevel API related issues labels Mar 10, 2023

pjpratik assigned SuryanarayanaY and unassigned pjpratik Mar 10, 2023

SuryanarayanaY added the type:build/install Build and install issues label Mar 13, 2023

SuryanarayanaY added the stat:awaiting response Status - Awaiting response from author label Mar 13, 2023

google-ml-butler bot removed the stat:awaiting response Status - Awaiting response from author label Mar 13, 2023

SuryanarayanaY added the stat:awaiting response Status - Awaiting response from author label Mar 15, 2023

google-ml-butler bot removed the stat:awaiting response Status - Awaiting response from author label Mar 15, 2023

SuryanarayanaY added the stat:awaiting response Status - Awaiting response from author label Mar 17, 2023

google-ml-butler bot removed the stat:awaiting response Status - Awaiting response from author label Mar 17, 2023

SuryanarayanaY added the stat:awaiting response Status - Awaiting response from author label Mar 18, 2023

google-ml-butler bot removed the stat:awaiting response Status - Awaiting response from author label Mar 20, 2023

learning-to-play assigned vam-google Mar 22, 2023

SuryanarayanaY added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Mar 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

//tensorflow/tools/api/tests:api_compatibility_test fails with protobuf > 3.20.3 #59932

//tensorflow/tools/api/tests:api_compatibility_test fails with protobuf > 3.20.3 #59932

elfringham commented Mar 8, 2023 •

edited by google-ml-butler bot

Issue Type

Have you reproduced the bug with TF nightly?

Source

Tensorflow Version

Custom Code

OS Platform and Distribution

Mobile device

Python version

Bazel version

GCC/Compiler version

CUDA/cuDNN version

GPU model and memory

Current Behaviour?

Standalone code to reproduce the issue

Relevant log output

SuryanarayanaY commented Mar 13, 2023

elfringham commented Mar 13, 2023

SuryanarayanaY commented Mar 14, 2023

elfringham commented Mar 14, 2023

SuryanarayanaY commented Mar 14, 2023

SuryanarayanaY commented Mar 15, 2023

elfringham commented Mar 15, 2023

SuryanarayanaY commented Mar 16, 2023

elfringham commented Mar 16, 2023

elfringham commented Mar 16, 2023

SuryanarayanaY commented Mar 17, 2023

elfringham commented Mar 17, 2023

SuryanarayanaY commented Mar 18, 2023

elfringham commented Mar 20, 2023

SuryanarayanaY commented Mar 21, 2023

learning-to-play commented Mar 22, 2023

SuryanarayanaY commented Apr 3, 2023

vam-google commented Apr 3, 2023 •

edited

elfringham commented Apr 4, 2023

//tensorflow/tools/api/tests:api_compatibility_test fails with protobuf > 3.20.3 #59932

//tensorflow/tools/api/tests:api_compatibility_test fails with protobuf > 3.20.3 #59932

Comments

elfringham commented Mar 8, 2023 • edited by google-ml-butler bot

Issue Type

Have you reproduced the bug with TF nightly?

Source

Tensorflow Version

Custom Code

OS Platform and Distribution

Mobile device

Python version

Bazel version

GCC/Compiler version

CUDA/cuDNN version

GPU model and memory

Current Behaviour?

Standalone code to reproduce the issue

Relevant log output

SuryanarayanaY commented Mar 13, 2023

elfringham commented Mar 13, 2023

SuryanarayanaY commented Mar 14, 2023

elfringham commented Mar 14, 2023

SuryanarayanaY commented Mar 14, 2023

SuryanarayanaY commented Mar 15, 2023

elfringham commented Mar 15, 2023

SuryanarayanaY commented Mar 16, 2023

elfringham commented Mar 16, 2023

elfringham commented Mar 16, 2023

SuryanarayanaY commented Mar 17, 2023

elfringham commented Mar 17, 2023

SuryanarayanaY commented Mar 18, 2023

elfringham commented Mar 20, 2023

SuryanarayanaY commented Mar 21, 2023

learning-to-play commented Mar 22, 2023

SuryanarayanaY commented Apr 3, 2023

vam-google commented Apr 3, 2023 • edited

elfringham commented Apr 4, 2023

elfringham commented Mar 8, 2023 •

edited by google-ml-butler bot

vam-google commented Apr 3, 2023 •

edited