Installation over pip fails to import with protobuf 2.6.1 #4

panmari · 2015-11-09T15:20:48Z

Pip freeze reports the following protobuf version

protobuf==2.6.1

However, upon importing I get the following error

Traceback (most recent call last):
  File "/home/panmari/PycharmProjects/tensor_stuff/stuff.py", line 150, in <module>
    import tensorflow as tf
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/__init__.py", line 4, in <module>
    from tensorflow.python import *
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/__init__.py", line 13, in <module>
    from tensorflow.core.framework.graph_pb2 import *
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/core/framework/graph_pb2.py", line 16, in <module>
    from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/core/framework/attr_value_pb2.py", line 16, in <module>
    from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/core/framework/tensor_pb2.py", line 16, in <module>
    from tensorflow.core.framework import tensor_shape_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__shape__pb2
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/core/framework/tensor_shape_pb2.py", line 22, in <module>
    serialized_pb=_b('\n,tensorflow/core/framework/tensor_shape.proto\x12\ntensorflow\"d\n\x10TensorShapeProto\x12-\n\x03\x64im\x18\x02 \x03(\x0b\x32 .tensorflow.TensorShapeProto.Dim\x1a!\n\x03\x44im\x12\x0c\n\x04size\x18\x01 \x01(\x03\x12\x0c\n\x04name\x18\x02 \x01(\tb\x06proto3')
TypeError: __init__() got an unexpected keyword argument 'syntax'

Do I need a newer version of the protobuf package?

The text was updated successfully, but these errors were encountered:

vrv · 2015-11-09T15:26:53Z

We definitely depend on protobuf 3.0 (it's what we use as a git submodule).

To help us debug, can you please try following the instructions to install within a virtualenv here: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/get_started/os_setup.md#virtualenv-based-installation- ?

panmari · 2015-11-09T15:30:16Z

Got it running by uninstalling the distribution-owned numpy package (?) and reinstalling it from pip. Guess they were interfering in some weird way. Thanks for your help!

pannous · 2015-12-09T10:35:27Z

got it running by explicitly installing
sudo pip install -Iv protobuf==3.0a3

Appache 2.0 license.

re-introduced entry guard for LIBXSMM convolutions

Stay up to date

It looks better for multiple subgraphs models. example) Your TFLite model has ‘3’ subgraph(s). In the subgraph description below, T# represents the Tensor numbers. For example, in Subgraph#0, the WHILE op takes tensor #1 and tensor #1 and tensor #0 as input and produces tensor #2 and tensor #3 and tensor #4 as output. Subgraph#0 main(T#0) -> [T#4, T#3] Op#0 WHILE(T#1, T#1, T#0) -> [T#2, T#3, T#4] Tensors of Subgraph#0 T#0(serving_default_inp:0) shape:[], type:FLOAT32 T#1(Const) shape:[], type:INT32 T#2(while) shape:[], type:INT32 T#3(PartitionedCall:0) shape:[], type:INT32 T#4(PartitionedCall:1) shape:[], type:FLOAT32 Subgraph#1 while_cond(T#0, T#1, T#2) -> [T#4] Op#0 LESS(T#1, T#3) -> [T#4] Tensors of Subgraph#1 T#0(arg0) shape:[], type:INT32 T#1(arg1) shape:[], type:INT32 T#2(arg2) shape:[], type:FLOAT32 T#3(while/Less/y) shape:[], type:INT32 T#4(while/Less) shape:[], type:BOOL Subgraph#2 while_body(T#0, T#1, T#2) -> [T#7, T#5, T#6] Op#0 ADD(T#1, T#3) -> [T#5] Op#1 MUL(T#2, T#4) -> [T#6] Op#2 ADD(T#0, T#3) -> [T#7] Tensors of Subgraph#2 T#0(arg0) shape:[], type:INT32 T#1(arg1) shape:[], type:INT32 T#2(arg2) shape:[], type:FLOAT32 T#3(while/Add/y) shape:[], type:INT32 T#4(while/scalar_mul/scalar) shape:[], type:FLOAT32 T#5(while/Add) shape:[], type:INT32 T#6(while/scalar_mul) shape:[], type:FLOAT32 T#7(while/add_1) shape:[], type:INT32 PiperOrigin-RevId: 390499926 Change-Id: If2dd3fc5b9f4404c83268d56f3eccb1bc1271d0f

It looks better for multiple subgraphs models. example) Your TFLite model has ‘3’ subgraph(s). In the subgraph description below, T# represents the Tensor numbers. For example, in Subgraph#0, the WHILE op takes tensor #1 and tensor #1 and tensor #0 as input and produces tensor #2 and tensor tensorflow#3 and tensor tensorflow#4 as output. Subgraph#0 main(T#0) -> [T#4, T#3] Op#0 WHILE(T#1, T#1, T#0) -> [T#2, T#3, T#4] Tensors of Subgraph#0 T#0(serving_default_inp:0) shape:[], type:FLOAT32 T#1(Const) shape:[], type:INT32 T#2(while) shape:[], type:INT32 T#3(PartitionedCall:0) shape:[], type:INT32 T#4(PartitionedCall:1) shape:[], type:FLOAT32 Subgraph#1 while_cond(T#0, T#1, T#2) -> [T#4] Op#0 LESS(T#1, T#3) -> [T#4] Tensors of Subgraph#1 T#0(arg0) shape:[], type:INT32 T#1(arg1) shape:[], type:INT32 T#2(arg2) shape:[], type:FLOAT32 T#3(while/Less/y) shape:[], type:INT32 T#4(while/Less) shape:[], type:BOOL Subgraph#2 while_body(T#0, T#1, T#2) -> [T#7, T#5, T#6] Op#0 ADD(T#1, T#3) -> [T#5] Op#1 MUL(T#2, T#4) -> [T#6] Op#2 ADD(T#0, T#3) -> [T#7] Tensors of Subgraph#2 T#0(arg0) shape:[], type:INT32 T#1(arg1) shape:[], type:INT32 T#2(arg2) shape:[], type:FLOAT32 T#3(while/Add/y) shape:[], type:INT32 T#4(while/scalar_mul/scalar) shape:[], type:FLOAT32 T#5(while/Add) shape:[], type:INT32 T#6(while/scalar_mul) shape:[], type:FLOAT32 T#7(while/add_1) shape:[], type:INT32 PiperOrigin-RevId: 390499926 Change-Id: If2dd3fc5b9f4404c83268d56f3eccb1bc1271d0f

On some CI nodes (typically those with higher CPU core counts 128/256), the `//tensorflow/c/eager:c_api_distributed_test_gpu` test fails on an intermitent basis. When it does fail, the failures manifests as segfault at the end of the test, with the stack dump shown at the end of this commit message. The stack dump points the finger to a routine within the MKLDNN implementation. This is further confirmed by the observation that disabling the MKLDNN based Eigen contraction kernels (for ROCm) seems to make the crash go away. related JIRA ticket - https://ontrack-internal.amd.com/browse/SWDEV-313684 A previous commit disabled the `//tensorflow/c/eager:c_api_distributed_test` unit-test only in the CPU unit-tests CI job (for the same reason). That comit cannot be reverted, because this commit disables MKLDNN based Eigen contraction kernels *only* for the ROCm build. ``` Thread 191 "c_api_distribut" received signal SIGSEGV, Segmentation fault. [Switching to thread 191 (Thread 0x7ffc777fe700 (LWP 159004))] 0x00007fff54530000 in ?? () (gdb) where #0 0x00007fff54530000 in ?? () #1 0x00007fffd5d15ae4 in dnnl::impl::cpu::x64::avx_gemm_f32::sgemm_nocopy_driver(char const*, char const*, long, long, long, float const*, float const*, long, float const*, long, float const*, float*, long, float const*, float*) () from /root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/c/eager/../../../_solib_local/libexternal_Smkl_Udnn_Uv1_Slibmkl_Udnn.so #2 0x00007fffd5d166e1 in dnnl::impl::cpu::x64::jit_avx_gemm_f32(int, char const*, char const*, long const*, long const*, long const*, float const*, float const*, long const*, float const*, long const*, float const*, float*, long const*, float const*) () from /root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/c/eager/../../../_solib_local/libexternal_Smkl_Udnn_Uv1_Slibmkl_Udnn.so #3 0x00007fffd5e277ed in dnnl_status_t dnnl::impl::cpu::x64::gemm_driver<float, float, float>(char const*, char const*, char const*, long const*, long const*, long const*, float const*, float const*, long const*, float const*, float const*, long const*, float const*, float const*, float*, long const*, float const*, bool, dnnl::impl::cpu::x64::pack_type, dnnl::impl::cpu::x64::gemm_pack_storage_t*, bool) () from /root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/c/eager/../../../_solib_local/libexternal_Smkl_Udnn_Uv1_Slibmkl_Udnn.so #4 0x00007fffd5665056 in dnnl::impl::cpu::extended_sgemm(char const*, char const*, long const*, long const*, long const*, float const*, float const*, long const*, float const*, long const*, float const*, float*, long const*, float const*, bool) () from /root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/c/eager/../../../_solib_local/libexternal_Smkl_Udnn_Uv1_Slibmkl_Udnn.so #5 0x00007fffd52fe983 in dnnl_sgemm () from /root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/c/eager/../../../_solib_local/libexternal_Smkl_Udnn_Uv1_Slibmkl_Udnn.so #6 0x0000555557187b0b in Eigen::internal::TensorContractionKernel<float, float, float, long, Eigen::internal::blas_data_mapper<float, long, 0, 0, 1>, Eigen::internal::TensorContractionInputMapper<float, long, 1, Eigen::TensorEvaluator<Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::ThreadPoolDevice>, Eigen::array<long, 1ul>, Eigen::array<long, 1ul>, 4, true, false, 0, Eigen::MakePointer>, Eigen::internal::TensorContractionInputMapper<float, long, 0, Eigen::TensorEvaluator<Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::ThreadPoolDevice>, Eigen::array<long, 1ul>, Eigen::array<long, 1ul>, 4, true, false, 0, Eigen::MakePointer> >::invoke(Eigen::internal::blas_data_mapper<float, long, 0, 0, 1> const&, Eigen::internal::ColMajorBlock<float, long> const&, Eigen::internal::ColMajorBlock<float, long> const&, long, long, long, float, float) () #7 0x000055555718dc76 in Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::NoOpOutputKernel const> const, Eigen::ThreadPoolDevice>::EvalParallelContext<Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::NoOpOutputKernel const> const, Eigen::ThreadPoolDevice>::NoCallback, true, true, false, 0>::kernel(long, long, long, bool) () #8 0x000055555718f327 in Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::NoOpOutputKernel const> const, Eigen::ThreadPoolDevice>::EvalParallelContext<Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::NoOpOutputKernel const> const, Eigen::ThreadPoolDevice>::NoCallback, true, true, false, 0>::signal_kernel(long, long, long, bool, bool) () #9 0x00005555571904cb in Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::NoOpOutputKernel const> const, Eigen::ThreadPoolDevice>::EvalParallelContext<Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::NoOpOutputKernel const> const, Eigen::ThreadPoolDevice>::NoCallback, true, true, false, 0>::pack_rhs(long, long) () #10 0x000055555718fd69 in Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::NoOpOutputKernel const> const, Eigen::ThreadPoolDevice>::EvalParallelContext<Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::NoOpOutputKernel const> const, Eigen::ThreadPoolDevice>::NoCallback, true, true, false, 0>::enqueue_packing_helper(long, long, long, bool) () #11 0x00007ffff6b607a1 in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/c/eager/../../../_solib_local/_U_S_Stensorflow_Sc_Seager_Cc_Uapi_Udistributed_Utest_Ugpu___Utensorflow/libtensorflow_framework.so.2 #12 0x00007ffff6b5de93 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/c/eager/../../../_solib_local/_U_S_Stensorflow_Sc_Seager_Cc_Uapi_Udistributed_Utest_Ugpu___Utensorflow/libtensorflow_framework.so.2 #13 0x00007ffff6b40107 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/c/eager/../../../_solib_local/_U_S_Stensorflow_Sc_Seager_Cc_Uapi_Udistributed_Utest_Ugpu___Utensorflow/libtensorflow_framework.so.2 #14 0x00007fffd1ca86db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #15 0x00007fffd00b471f in clone () from /lib/x86_64-linux-gnu/libc.so.6 ```

* change to manually copy all cutlass headers instead of using bazel import * try to add cutlass conv * add default grad for cus * change the padding dimension since padding is 2d in cutlass problem size * disable conv fused rewriter as fused conv is not implemented for cus * finish cutlass convolution forward for cus * finish cutlass conv backprop for cus

…d SavedObjectGraph. This commit only looks at the `concrete_functions` of the SavedObjectGraph, ignoring the `nodes`. RFC: tensorflow/community#415 PiperOrigin-RevId: 465475648

RFC: tensorflow/community#415 PiperOrigin-RevId: 465618721

Signed-off-by: Ritul Jasuja <ritul.jasuja@intel.com>

On some CI nodes (typically those with higher CPU core counts 128/256), the `//tensorflow/c/eager:c_api_distributed_test_gpu` test fails on an intermitent basis. When it does fail, the failures manifests as segfault at the end of the test, with the stack dump shown at the end of this commit message. The stack dump points the finger to a routine within the MKLDNN implementation. This is further confirmed by the observation that disabling the MKLDNN based Eigen contraction kernels (for ROCm) seems to make the crash go away. related JIRA ticket - https://ontrack-internal.amd.com/browse/SWDEV-313684 A previous commit disabled the `//tensorflow/c/eager:c_api_distributed_test` unit-test only in the CPU unit-tests CI job (for the same reason). That comit cannot be reverted, because this commit disables MKLDNN based Eigen contraction kernels *only* for the ROCm build. ``` Thread 191 "c_api_distribut" received signal SIGSEGV, Segmentation fault. [Switching to thread 191 (Thread 0x7ffc777fe700 (LWP 159004))] 0x00007fff54530000 in ?? () (gdb) where #0 0x00007fff54530000 in ?? () #1 0x00007fffd5d15ae4 in dnnl::impl::cpu::x64::avx_gemm_f32::sgemm_nocopy_driver(char const*, char const*, long, long, long, float const*, float const*, long, float const*, long, float const*, float*, long, float const*, float*) () from /root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/c/eager/../../../_solib_local/libexternal_Smkl_Udnn_Uv1_Slibmkl_Udnn.so tensorflow#2 0x00007fffd5d166e1 in dnnl::impl::cpu::x64::jit_avx_gemm_f32(int, char const*, char const*, long const*, long const*, long const*, float const*, float const*, long const*, float const*, long const*, float const*, float*, long const*, float const*) () from /root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/c/eager/../../../_solib_local/libexternal_Smkl_Udnn_Uv1_Slibmkl_Udnn.so tensorflow#3 0x00007fffd5e277ed in dnnl_status_t dnnl::impl::cpu::x64::gemm_driver<float, float, float>(char const*, char const*, char const*, long const*, long const*, long const*, float const*, float const*, long const*, float const*, float const*, long const*, float const*, float const*, float*, long const*, float const*, bool, dnnl::impl::cpu::x64::pack_type, dnnl::impl::cpu::x64::gemm_pack_storage_t*, bool) () from /root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/c/eager/../../../_solib_local/libexternal_Smkl_Udnn_Uv1_Slibmkl_Udnn.so tensorflow#4 0x00007fffd5665056 in dnnl::impl::cpu::extended_sgemm(char const*, char const*, long const*, long const*, long const*, float const*, float const*, long const*, float const*, long const*, float const*, float*, long const*, float const*, bool) () from /root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/c/eager/../../../_solib_local/libexternal_Smkl_Udnn_Uv1_Slibmkl_Udnn.so tensorflow#5 0x00007fffd52fe983 in dnnl_sgemm () from /root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/c/eager/../../../_solib_local/libexternal_Smkl_Udnn_Uv1_Slibmkl_Udnn.so tensorflow#6 0x0000555557187b0b in Eigen::internal::TensorContractionKernel<float, float, float, long, Eigen::internal::blas_data_mapper<float, long, 0, 0, 1>, Eigen::internal::TensorContractionInputMapper<float, long, 1, Eigen::TensorEvaluator<Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::ThreadPoolDevice>, Eigen::array<long, 1ul>, Eigen::array<long, 1ul>, 4, true, false, 0, Eigen::MakePointer>, Eigen::internal::TensorContractionInputMapper<float, long, 0, Eigen::TensorEvaluator<Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::ThreadPoolDevice>, Eigen::array<long, 1ul>, Eigen::array<long, 1ul>, 4, true, false, 0, Eigen::MakePointer> >::invoke(Eigen::internal::blas_data_mapper<float, long, 0, 0, 1> const&, Eigen::internal::ColMajorBlock<float, long> const&, Eigen::internal::ColMajorBlock<float, long> const&, long, long, long, float, float) () tensorflow#7 0x000055555718dc76 in Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::NoOpOutputKernel const> const, Eigen::ThreadPoolDevice>::EvalParallelContext<Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::NoOpOutputKernel const> const, Eigen::ThreadPoolDevice>::NoCallback, true, true, false, 0>::kernel(long, long, long, bool) () tensorflow#8 0x000055555718f327 in Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::NoOpOutputKernel const> const, Eigen::ThreadPoolDevice>::EvalParallelContext<Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::NoOpOutputKernel const> const, Eigen::ThreadPoolDevice>::NoCallback, true, true, false, 0>::signal_kernel(long, long, long, bool, bool) () tensorflow#9 0x00005555571904cb in Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::NoOpOutputKernel const> const, Eigen::ThreadPoolDevice>::EvalParallelContext<Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::NoOpOutputKernel const> const, Eigen::ThreadPoolDevice>::NoCallback, true, true, false, 0>::pack_rhs(long, long) () tensorflow#10 0x000055555718fd69 in Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::NoOpOutputKernel const> const, Eigen::ThreadPoolDevice>::EvalParallelContext<Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::NoOpOutputKernel const> const, Eigen::ThreadPoolDevice>::NoCallback, true, true, false, 0>::enqueue_packing_helper(long, long, long, bool) () tensorflow#11 0x00007ffff6b607a1 in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/c/eager/../../../_solib_local/_U_S_Stensorflow_Sc_Seager_Cc_Uapi_Udistributed_Utest_Ugpu___Utensorflow/libtensorflow_framework.so.2 tensorflow#12 0x00007ffff6b5de93 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/c/eager/../../../_solib_local/_U_S_Stensorflow_Sc_Seager_Cc_Uapi_Udistributed_Utest_Ugpu___Utensorflow/libtensorflow_framework.so.2 tensorflow#13 0x00007ffff6b40107 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/c/eager/../../../_solib_local/_U_S_Stensorflow_Sc_Seager_Cc_Uapi_Udistributed_Utest_Ugpu___Utensorflow/libtensorflow_framework.so.2 tensorflow#14 0x00007fffd1ca86db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 tensorflow#15 0x00007fffd00b471f in clone () from /lib/x86_64-linux-gnu/libc.so.6 ```

Addressed review comments Signed-off-by: Ritul Jasuja <ritul.jasuja@intel.com>

Currently we look for ptxas and nvlink in a few different places on the host machine, then we choose the first found binary without taking its version into account. If the chosen binary doesn't fulfill our version requirements we will later fail even if there was a suitable ptxas or nvlink in the search path in the first place. This change makes it take the version of each binary into account when going through the search path. Unsuitable binaries will be discarded right away and the search continues until we are out of locations to check. This should help with host environments that have multiple CUDA toolkits installed and should make ptxas and nvlink selection more robust. The concreate changes: 1. `FindCudaExecutable` now also takes a minimum version and a list of forbidden (think buggy) versions that are supposed to be skipped. 2. `WarnIfBadPtxAsVersion` has been removed. It was checking for ptxas < 11.1 which is way older than our minimum supported version of 11.8 and was not doing anything given the check described in #3. 3. There was another version check for `ptxas` in `NVPTXCompiler::ChooseLinkingMethod` which was checking for `version(ptxas)` < 11.8. This has also been removed/replace by the version check described in #4. 4. Version checking for `ptxas` and `nvlink` has been consolidated into 2 methods `FindPtxAsExectuable` and `FindNvLinkExecutable`. These methods hard code the current minimum version (and the list of excluded versions) of each tool in one place. It's still not great but at least less spaghetti-like. PiperOrigin-RevId: 612791713

Currently we look for ptxas and nvlink in a few different places on the host machine, then we choose the first found binary without taking its version into account. If the chosen binary doesn't fulfill our version requirements we will later fail even if there was a suitable ptxas or nvlink in the search path in the first place. This change makes it take the version of each binary into account when going through the search path. Unsuitable binaries will be discarded right away and the search continues until we are out of locations to check. This should help with host environments that have multiple CUDA toolkits installed and should make ptxas and nvlink selection more robust. The concreate changes: 1. `FindCudaExecutable` now also takes a minimum version and a list of forbidden (think buggy) versions that are supposed to be skipped. 2. `WarnIfBadPtxAsVersion` has been removed. It was checking for ptxas < 11.1 which is way older than our minimum supported version of 11.8 and was not doing anything given the check described in #3. 3. There was another version check for `ptxas` in `NVPTXCompiler::ChooseLinkingMethod` which was checking for `version(ptxas)` < 11.8. This has also been removed/replace by the version check described in #4. 4. Version checking for `ptxas` and `nvlink` has been consolidated into 2 methods `FindPtxAsExectuable` and `FindNvLinkExecutable`. These methods hard code the current minimum version (and the list of excluded versions) of each tool in one place. It's still not great but at least less spaghetti-like. PiperOrigin-RevId: 618797392

panmari closed this as completed Nov 9, 2015

panmari mentioned this issue Nov 23, 2015

Error: Executing cifar10_train.py on Ubuntu with Cuda #334

Closed

wchan mentioned this issue Jan 7, 2016

failed to query current context: CUDA_ERROR_DEINITIALIZED #713

Closed

anuragkr90 mentioned this issue Apr 8, 2016

Core Dump Error on running examples #1803

Closed

alphaf52 mentioned this issue Apr 19, 2016

recurrent layer on top of convolution layer fails with GPU #2015

Closed

wangyongliang mentioned this issue Apr 27, 2016

core dump when import tensorflow #2129

Closed

Mazecreator mentioned this issue Jun 27, 2016

Reinforcement Learning can be really, really slow in tensorflow. #2444

Closed

wagonhelm mentioned this issue Jun 30, 2016

Android Tensorflow Crashes w/ Custom Graph #3117

Closed

This was referenced Jul 10, 2016

Segmentation fault on tensorflow 0.9.0 #2773

Closed

Segmentation fault, import tensorflow, tensorflow 0.9, mac osx #3263

Closed

zheng-xq mentioned this issue Jul 18, 2016

tensor flow programs using gpu freeze instance #1947

Closed

pmousoul mentioned this issue Sep 11, 2016

Error malloc(): memory corruption #3442

Closed

zhouphd mentioned this issue Oct 24, 2016

[unexpected keyword argument 'syntax'] occurs on Ubuntu 16.04 with protobuf 3 of 4e8497d by installing from source #5168

Closed

benoitsteiner added a commit to benoitsteiner/tensorflow that referenced this issue Oct 28, 2016

Merge pull request tensorflow#4 from lukeiwanski/update/ComputeCpp

2291bb9

Appache 2.0 license.

danielgordon10 mentioned this issue Dec 10, 2016

Running summary call crashes sporadically #6195

Closed

aseuteurideu mentioned this issue Dec 13, 2016

in debug mode, got Assertion `cudaGetLastError() == cudaSuccess' failed #6285

Closed

beniroquai mentioned this issue Dec 18, 2016

[Android] Put large .pb files outside of Asset-folder? [ERROR]: Check failed: message->ParseFromZeroCopyStream(&lis) #6213

Closed

usmcamp0811 mentioned this issue Jan 9, 2017

tf.map_fn throws error...sometimes.... #6730

Closed

keotic mentioned this issue Jan 13, 2017

Variable "matching_filenames" @ tf.train.match_filenames_once() should be local? #6786

Closed

benoitsteiner pushed a commit to benoitsteiner/tensorflow that referenced this issue Jan 18, 2017

Merge pull request tensorflow#4 from alheinecke/master

d175d46

re-introduced entry guard for LIBXSMM convolutions

vahidk mentioned this issue Jan 27, 2017

Tensorflow freezes on iOS during Session::Run #7108

Closed

GeorgianaPetria mentioned this issue Jan 31, 2017

TensorFlow demo app crashes with my own model #4451

Closed

mkabra mentioned this issue Feb 14, 2017

Seg fault when using tf session with opencv 3 #7378

Closed

adammalpass added a commit to adammalpass/tensorflow that referenced this issue Feb 15, 2017

Complete upto just before Question tensorflow#4

c45fb6f

prabhant mentioned this issue Feb 17, 2017

After installation error #7621

Closed

bluezone2015 mentioned this issue Mar 8, 2017

import tensorflow Segmentation fault (core dumped) #8197

Closed

Panaetius mentioned this issue Mar 20, 2017

Floating Point Exception/SIGFPE in tf.map_fn over an empty tensor #8554

Closed

copybara-service bot pushed a commit that referenced this issue Apr 14, 2021

Merge pull request #4 from tensorflow/master

987d4be

Stay up to date

wang5566 mentioned this issue May 12, 2021

TensorRT Segmentation Fault During Conversion For Debug Mode #49139

Open

rsanthanam-amd mentioned this issue Jul 1, 2021

[ROCm] This change replaces the original assert for detecting multiple #49232

Closed

Koushik667 mentioned this issue Jul 8, 2021

Converting Boosted tree model to tflite #50667

Closed

DavidvSon1 mentioned this issue Oct 3, 2021

Segmentation fault when invoking TFLite interpreter on basic quantized model tensorflow/model-optimization#857

Open

Roshan1986 mentioned this issue Oct 9, 2021

"zoom_range" argument of "ImageDataGenerator" class show distortions around edges #52311

Closed

StewardH mentioned this issue Feb 13, 2022

A/libc: Fatal signal 11 (SIGSEGV), code 1, fault addr 0x70 in tid 28007 #34313

Closed

goddie1 mentioned this issue Mar 10, 2022

tensorflow core while do stdthread create #55186

Closed

copybara-service bot pushed a commit that referenced this issue Aug 5, 2022

[SavedModel Fingerprinting] Add python tests for fingerprint #4.

99d8d03

RFC: tensorflow/community#415 PiperOrigin-RevId: 465618721

rbaranchuk-capgemini mentioned this issue Jan 9, 2023

Fix memory leaks in xla::BufferDonationTest #59033

Closed

polarbbb mentioned this issue Mar 17, 2023

About Android calling TensorFlow Lite crash problem #60030

Closed

iammeizu mentioned this issue May 10, 2023

tensorflow1.7 hangs at LocalMaster::RunStep with tf.train.MonitoredTrainingSession in sync mode #24338

Closed

gbaned mentioned this issue Jun 12, 2023

[PluggableDevice] Always link kernels_experimental c api #60786

Merged

ivankxt mentioned this issue Jun 12, 2023

Get deadlock after Predict(cuda10.0, cudnn7.6.5, Tesla T4 GPU) #60841

Closed

Rjasuja added a commit to anikulk/tensorflow that referenced this issue Sep 14, 2023

[WIP]: Attempt to add Add Node Commit:tensorflow#4

5bc3c76

Signed-off-by: Ritul Jasuja <ritul.jasuja@intel.com>

lyz1005 mentioned this issue Oct 26, 2023

Interpreter run crash #62240

Closed

CarloWood mentioned this issue Nov 11, 2023

cuDNN, cuFFT, and cuBLAS Errors #62075

Open

spacycoder mentioned this issue Dec 11, 2023

Why does my full integer quantized tflite model crash when loaded? #62618

Closed

Rjasuja added a commit to Rjasuja/tensorflow that referenced this issue Dec 19, 2023

Refactor operations module tensorflow tensorflow#4

30d524f

Addressed review comments Signed-off-by: Ritul Jasuja <ritul.jasuja@intel.com>

copybara-service bot mentioned this issue Mar 22, 2024

Add version checks to FindCudaExecutable #64135

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Installation over pip fails to import with protobuf 2.6.1 #4

Installation over pip fails to import with protobuf 2.6.1 #4

panmari commented Nov 9, 2015

vrv commented Nov 9, 2015

panmari commented Nov 9, 2015

pannous commented Dec 9, 2015

Installation over pip fails to import with protobuf 2.6.1 #4

Installation over pip fails to import with protobuf 2.6.1 #4

Comments

panmari commented Nov 9, 2015

vrv commented Nov 9, 2015

panmari commented Nov 9, 2015

pannous commented Dec 9, 2015