Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing to find `-ltensorflow_framework` #7903

Open
Eric-Arellano opened this issue Jun 19, 2019 · 1 comment

Comments

Projects
None yet
2 participants
@Eric-Arellano
Copy link
Contributor

commented Jun 19, 2019

CI is consistently failing due to the below TensorFlow issue. It did not start flaking until today (June 19, 2019). The most recent commit was 32d8180, which seems extremely unlikely to have caused the change. This is likely due to an external change that now impacts us.


==================== FAILURES ====================
                     ____ TestProjectsIntegrationTest.test_shard_7 ____
                     
                     self = <pants_test.projects.test_testprojects_integration.TestProjectsIntegrationTest testMethod=test_shard_7>
                     
                         def test_shard_7(self):
                     >     self.run_shard(7)
                     
                     .pants.d/pyprep/sources/69446c4993914909469884ba16737382e81a451a/pants_test/projects/test_testprojects_integration.py:165: 
                     _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
                     .pants.d/pyprep/sources/69446c4993914909469884ba16737382e81a451a/pants_test/pants_run_integration_test.py:108: in wrapper
                         f(self, *args, **kwargs)
                     .pants.d/pyprep/sources/69446c4993914909469884ba16737382e81a451a/pants_test/projects/test_testprojects_integration.py:136: in run_shard
                         self.assert_success(pants_run)
                     .pants.d/pyprep/sources/69446c4993914909469884ba16737382e81a451a/pants_test/pants_run_integration_test.py:483: in assert_success
                         self.assert_result(pants_run, PANTS_SUCCEEDED_EXIT_CODE, expected=True, msg=msg)
                     .pants.d/pyprep/sources/69446c4993914909469884ba16737382e81a451a/pants_test/pants_run_integration_test.py:504: in assert_result
                         assertion(value, pants_run.returncode, error_msg)
                     E   AssertionError: 0 != 1 : /home/travis/build/pantsbuild/pants/pants --no-pantsrc --pants-workdir=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmpbnrv9tg9.pants.d --kill-nailguns --print-exception-stacktrace=True test examples/src/python/example/hello/greet:greet examples/src/python/example/hello/main:hello-app examples/src/python/example/hello/main:main examples/src/python/example/pants_publish_plugin:pants-publish-plugin examples/src/python/example/tensorflow_custom_op:tensorflow-zero-out-op examples/src/python/example/tensorflow_custom_op:tensorflow-zero-out-op-wrapper examples/src/python/example/tensorflow_custom_op:tensorflow_custom_op examples/src/python/example:3rdparty_py --jvm-platform-default-platform=java7
                     E   returncode: 1
                     E   stdout:
                     E   	
                     E   	19:48:19 00:00 [main]
                     E   	               (To run a reporting server: ./pants server)
                     E   	19:48:20 00:01   [setup]
                     E   	19:48:22 00:03     [parse]
                     E   	               Executing tasks in goals: unpack-wheels -> bootstrap -> imports -> unpack-jars -> deferred-sources -> native-compile -> link -> jvm-platform-validate -> gen -> resolve -> resources -> compile -> pyprep -> test
                     E   	19:48:22 00:03   [unpack-wheels]
                     E   	19:48:22 00:03     [unpack-wheels]
                     E   	                   Invalidated 1 target.
                     E   	19:48:22 00:03       [extract-native-wheels]
                     E   	19:48:33 00:14   [background]
                     E   	19:48:33 00:14     [workdir_build_cleanup]
                     E   	19:48:33 00:14   [bootstrap]
                     E   	19:48:33 00:14     [substitute-aliased-targets]
                     E   	19:48:33 00:14     [jar-dependency-management]
                     E   	19:48:34 00:15     [bootstrap-jvm-tools]
                     E   	19:48:34 00:15     [provide-tools-jar]
                     E   	19:48:34 00:15     [workdir_build_cleanup]
                     E   	19:48:34 00:15   [imports]
                     E   	19:48:34 00:15     [ivy-imports]
                     E   	19:48:34 00:15   [unpack-jars]
                     E   	19:48:34 00:15     [unpack-jars]
                     E   	19:48:34 00:15     [workdir_build_cleanup]
                     E   	19:48:34 00:15   [deferred-sources]
                     E   	19:48:34 00:15     [deferred-sources]
                     E   	19:48:40 00:21   [native-compile]
                     E   	19:48:40 00:21     [conan-prep]
                     E   	19:48:40 00:21     [conan-fetch]
                     E   	19:48:40 00:21     [c-for-ctypes]
                     E   	19:48:40 00:21     [workdir_build_cleanup]
                     E   	19:48:40 00:21     [cpp-for-ctypes]
                     E   	                   Invalidated 1 target.
                     E   	                   selected compiler exe name: 'g++'
                     E   	19:48:41 00:22       [cpp-compile]
                     E   	                     
                     E   	19:48:44 00:25     [workdir_build_cleanup]
                     E   	19:48:44 00:25   [link]
                     E   	19:48:44 00:25     [shared-libraries]
                     E   	                   Invalidated 1 target.
                     E   	                   selected linker exe name: 'g++'
                     E   	19:48:44 00:25       [link-shared-libraries]
                     E   	                     ==== stdout ====
                     E   	                     
                     E   	                     ==== stderr ====
                     E   	                     /home/travis/.cache/pants/bin/binutils/linux/x86_64/2.30/binutils/bin/ld: cannot find -ltensorflow_framework
                     E   	                     collect2: error: ld returned 1 exit status
                     E   	                     
                     E   	FAILURE: Error linking native objects with command ['g++', '-shared', '-o', '/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmpbnrv9tg9.pants.d/link/shared-libraries/75beb223f481/examples.src.python.example.tensorflow_custom_op.tensorflow-zero-out-op/current/libtensorflow-zero-out-operator.so', '-L/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmpbnrv9tg9.pants.d/unpack-wheels/unpack-wheels/252d64521cf9/examples.3rdparty.python.tensorflow-native-libs/current/', '-ltensorflow_framework', '/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmpbnrv9tg9.pants.d/native-compile/cpp-for-ctypes/bf77967d723d/examples.src.python.example.tensorflow_custom_op.tensorflow-zero-out-op/current/zero_out_op.o'] and environment {'PATH': '/home/travis/.cache/pants/bin/gcc/linux/x86_64/7.3.0/gcc/bin:/home/travis/.cache/pants/bin/binutils/linux/x86_64/2.30/binutils/bin:/home/travis/.cache/pants/bin/binutils/linux/x86_64/2.30/binutils/bin', 'LD_LIBRARY_PATH': '/home/travis/.cache/pants/bin/gcc/linux/x86_64/7.3.0/gcc/lib64:/home/travis/.cache/pants/bin/gcc/linux/x86_64/7.3.0/gcc/lib:/home/travis/.cache/pants/bin/gcc/linux/x86_64/7.3.0/gcc/lib/gcc:/home/travis/.cache/pants/bin/gcc/linux/x86_64/7.3.0/gcc/lib/gcc/x86_64-pc-linux-gnu/7.3.0', 'LDSHARED': 'g++', 'LIBRARY_PATH': '/usr/lib/x86_64-linux-gnu'} for request LinkSharedLibraryRequest(linker<Exactly(Linker)>=Linker(path_entries=['/home/travis/.cache/pants/bin/gcc/linux/x86_64/7.3.0/gcc/bin', '/home/travis/.cache/pants/bin/binutils/linux/x86_64/2.30/binutils/bin', '/home/travis/.cache/pants/bin/binutils/linux/x86_64/2.30/binutils/bin'], exe_filename=g++, runtime_library_dirs=['/home/travis/.cache/pants/bin/gcc/linux/x86_64/7.3.0/gcc/lib64', '/home/travis/.cache/pants/bin/gcc/linux/x86_64/7.3.0/gcc/lib', '/home/travis/.cache/pants/bin/gcc/linux/x86_64/7.3.0/gcc/lib/gcc', '/home/travis/.cache/pants/bin/gcc/linux/x86_64/7.3.0/gcc/lib/gcc/x86_64-pc-linux-gnu/7.3.0'], linking_library_dirs=[], extra_args=[], extra_object_files=['/usr/lib/x86_64-linux-gnu/crti.o']), object_files<Exactly(tuple)>=('/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmpbnrv9tg9.pants.d/native-compile/cpp-for-ctypes/bf77967d723d/examples.src.python.example.tensorflow_custom_op.tensorflow-zero-out-op/current/zero_out_op.o',), native_artifact<Exactly(NativeArtifact)>=NativeArtifact(lib_name=tensorflow-zero-out-operator), output_dir=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmpbnrv9tg9.pants.d/link/shared-libraries/75beb223f481/examples.src.python.example.tensorflow_custom_op.tensorflow-zero-out-op/current, external_lib_dirs<Exactly(tuple)>=('/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmpbnrv9tg9.pants.d/unpack-wheels/unpack-wheels/252d64521cf9/examples.3rdparty.python.tensorflow-native-libs/current/',), external_lib_names<Exactly(tuple)>=('tensorflow_framework',)). Exit code was: 1.
                     E   	
                     E   	
                     E   	               Waiting for background workers to finish.
                     E   	19:48:45 00:26   [complete]
                     E   	               FAILURE
                     E   stderr:
                     E   	19:48:31 [WARN] /home/travis/.pex/code/05f950848084c1f7f23f10651248595bbe82afa2/pants/task/unpack_remote_sources_base.py:85: DeprecationWarning: Flags not at the start of the expression 'include\\/([^/]+/)*([' (truncated)
                     E   	  compiled_patterns.append(re.compile(fnmatch_translate_extended(p)))
                     E   	
                     E   	19:48:31 [WARN] /home/travis/.pex/code/05f950848084c1f7f23f10651248595bbe82afa2/pants/task/unpack_remote_sources_base.py:85: DeprecationWarning: Flags not at the start of the expression '\\.\\/([^/]+)\\.so\\Z(?m' (truncated)
                     E   	  compiled_patterns.append(re.compile(fnmatch_translate_extended(p)))
                     -------------- Captured stdout call --------------
                     logs/exceptions.2736.log +++ 
                     logs/exceptions.2736.log --- 
                     logs/exceptions.log +++ 
                     logs/exceptions.log --- 
                     - generated xml file: /home/travis/build/pantsbuild/pants/.pants.d/test/pytest/tests.python.pants_test.projects.testprojects_integration/junitxml/TEST-tests.python.pants_test.projects.testprojects_integration.xml -
                     ============ short test summary info =============
                     FAIL tests/python/pants_test/projects/test_testprojects_integration.py::TestProjectsIntegrationTest::test_shard_7
                     1 failed, 3 passed in 631.18 seconds

This impacts

      'examples/src/python/example/tensorflow_custom_op:tensorflow-zero-out-op',
      'examples/src/python/example/tensorflow_custom_op:tensorflow-zero-out-op-wrapper',
      'examples/src/python/example/tensorflow_custom_op:tensorflow_custom_op',
      'examples/tests/python/example_test/tensorflow_custom_op:tensorflow_custom_op',

@Eric-Arellano Eric-Arellano changed the title Linking native objects now flaky Failing to find `-ltensorflow_framework` Jun 19, 2019

@Eric-Arellano Eric-Arellano added 3rdparty and removed flaky-test labels Jun 19, 2019

Eric-Arellano added a commit that referenced this issue Jun 19, 2019

@wisechengyi

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.