With compile gcc 12.2.0 tensorflow::functor::CSRSparseMatrixTranspose missing #58881

perestoronin · 2022-12-14T04:54:44Z

Click to expand!

Issue Type

Build/Install

Source

source

Tensorflow Version

tf 2.11.0

Custom Code

Yes

OS Platform and Distribution

Linux Gentoo

Mobile device

No response

Python version

3.11.1

Bazel version

5.4.0

GCC/Compiler version

12.2.0

CUDA/cuDNN version

12.0.0/8.7.0.84

GPU model and memory

AMD Vega Frontier 16Gb

Current Behaviour?

Building TF 2.11.0 with use ROCM 5.4.1 with GCC 12.2.0 failed, but succesfull built with use ROCM 5.4.1 with GCC 11.3.0

Standalone code to reproduce the issue

ERROR: /var/tmp/portage/sci-libs/tensorflow-2.11.0-r12/work/tensorflow-2.11.0-python3_11/tensorflow/BUILD:1137:20: Linking tensorflow/libtensorflow.so.2.11.0 failed: (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command 
  (cd /var/tmp/portage/sci-libs/tensorflow-2.11.0-r12/work/tensorflow-2.11.0-python3_11-bazel-base/execroot/org_tensorflow && \
  exec env - \
    HOME=/var/tmp/portage/sci-libs/tensorflow-2.11.0-r12/homedir \
    KERAS_HOME=/var/tmp/portage/sci-libs/tensorflow-2.11.0-r12/temp/.keras \
    MLIR_CRASH_REPRODUCER_DIRECTORY=/var/tmp/portage/sci-libs/tensorflow-2.11.0-r12/temp/.crash \
    PATH=/var/tmp/portage/sci-libs/tensorflow-2.11.0-r12/temp/python3.11/bin:/usr/lib/portage/python3.11/ebuild-helpers/xattr:/usr/lib/portage/python3.11/ebuild-helpers:/usr/sbin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/bin:/usr/lib/llvm/15/bin:/usr/lib64/subversion/bin:/opt/cuda/bin \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/usr/bin/python3.11 \
    PYTHON_LIB_PATH=/usr/lib/python3.11/site-packages \
    ROCBLAS_TENSILE_LIBPATH=/usr/lib64/rocblas/library \
    ROCM_PATH=/usr \
    TF2_BEHAVIOR=1 \
    TF_SYSTEM_LIBS=absl_py,astor_archive,astunparse_archive,boringssl,com_github_googlecloudplatform_google_cloud_cpp,curl,cython,dill_archive,double_conversion,flatbuffers,functools32_archive,gast_archive,gif,hwloc,icu,libjpeg_turbo,lmdb,nasm,nsync,opt_einsum_archive,org_sqlite,pasta,png,pybind11,six_archive,snappy,tblib_archive,termcolor_archive,typing_extensions_archive,wrapt,zlib \
  external/local_config_rocm/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc @bazel-out/k8-opt/bin/tensorflow/libtensorflow.so.2.11.0-2.params)
# Configuration: 2f0fac33d35f1b8e1706c32b05ccaf2da62c75cfc452cc037134c3e368e32f12
# Execution platform: @local_execution_config_platform//:platform
bazel-out/k8-opt/bin/tensorflow/core/kernels/sparse/_objs/kernels/mat_mul_op.pic.o:mat_mul_op.cc:function tensorflow::CSRMatMulGPUOp<std::complex<double> >::Compute(tensorflow::OpKernelContext*): error: undefined reference to 'tensorflow::functor::CSRSparseMatrixTranspose<Eigen::GpuDevice, std::complex<double> >::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)'
bazel-out/k8-opt/bin/tensorflow/core/kernels/sparse/_objs/kernels/mat_mul_op.pic.o:mat_mul_op.cc:function tensorflow::CSRMatMulGPUOp<std::complex<float> >::Compute(tensorflow::OpKernelContext*): error: undefined reference to 'tensorflow::functor::CSRSparseMatrixTranspose<Eigen::GpuDevice, std::complex<float> >::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)'
bazel-out/k8-opt/bin/tensorflow/core/kernels/sparse/_objs/kernels/mat_mul_op.pic.o:mat_mul_op.cc:function tensorflow::CSRMatMulGPUOp<float>::Compute(tensorflow::OpKernelContext*): error: undefined reference to 'tensorflow::functor::CSRSparseMatrixTranspose<Eigen::GpuDevice, float>::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)'
bazel-out/k8-opt/bin/tensorflow/core/kernels/sparse/_objs/kernels/mat_mul_op.pic.o:mat_mul_op.cc:function tensorflow::CSRMatMulGPUOp<double>::Compute(tensorflow::OpKernelContext*): error: undefined reference to 'tensorflow::functor::CSRSparseMatrixTranspose<Eigen::GpuDevice, double>::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)'
bazel-out/k8-opt/bin/tensorflow/core/kernels/sparse/_objs/kernels/sparse_mat_mul_op.pic.o:sparse_mat_mul_op.cc:function tensorflow::CSRSparseMatMulGPUOp<Eigen::GpuDevice, float>::Compute(tensorflow::OpKernelContext*): error: undefined reference to 'tensorflow::functor::CSRSparseMatrixTranspose<Eigen::GpuDevice, float>::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)'
bazel-out/k8-opt/bin/tensorflow/core/kernels/sparse/_objs/kernels/sparse_mat_mul_op.pic.o:sparse_mat_mul_op.cc:function tensorflow::CSRSparseMatMulGPUOp<Eigen::GpuDevice, float>::Compute(tensorflow::OpKernelContext*): error: undefined reference to 'tensorflow::functor::CSRSparseMatrixTranspose<Eigen::GpuDevice, float>::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)'
bazel-out/k8-opt/bin/tensorflow/core/kernels/sparse/_objs/kernels/sparse_mat_mul_op.pic.o:sparse_mat_mul_op.cc:function tensorflow::CSRSparseMatMulGPUOp<Eigen::GpuDevice, double>::Compute(tensorflow::OpKernelContext*): error: undefined reference to 'tensorflow::functor::CSRSparseMatrixTranspose<Eigen::GpuDevice, double>::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)'
bazel-out/k8-opt/bin/tensorflow/core/kernels/sparse/_objs/kernels/sparse_mat_mul_op.pic.o:sparse_mat_mul_op.cc:function tensorflow::CSRSparseMatMulGPUOp<Eigen::GpuDevice, double>::Compute(tensorflow::OpKernelContext*): error: undefined reference to 'tensorflow::functor::CSRSparseMatrixTranspose<Eigen::GpuDevice, double>::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)'
bazel-out/k8-opt/bin/tensorflow/core/kernels/sparse/_objs/kernels/sparse_mat_mul_op.pic.o:sparse_mat_mul_op.cc:function tensorflow::CSRSparseMatMulGPUOp<Eigen::GpuDevice, std::complex<float> >::Compute(tensorflow::OpKernelContext*): error: undefined reference to 'tensorflow::functor::CSRSparseMatrixTranspose<Eigen::GpuDevice, std::complex<float> >::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)'
bazel-out/k8-opt/bin/tensorflow/core/kernels/sparse/_objs/kernels/sparse_mat_mul_op.pic.o:sparse_mat_mul_op.cc:function tensorflow::CSRSparseMatMulGPUOp<Eigen::GpuDevice, std::complex<float> >::Compute(tensorflow::OpKernelContext*): error: undefined reference to 'tensorflow::functor::CSRSparseMatrixTranspose<Eigen::GpuDevice, std::complex<float> >::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)'
bazel-out/k8-opt/bin/tensorflow/core/kernels/sparse/_objs/kernels/sparse_mat_mul_op.pic.o:sparse_mat_mul_op.cc:function tensorflow::CSRSparseMatMulGPUOp<Eigen::GpuDevice, std::complex<double> >::Compute(tensorflow::OpKernelContext*): error: undefined reference to 'tensorflow::functor::CSRSparseMatrixTranspose<Eigen::GpuDevice, std::complex<double> >::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)'
bazel-out/k8-opt/bin/tensorflow/core/kernels/sparse/_objs/kernels/sparse_mat_mul_op.pic.o:sparse_mat_mul_op.cc:function tensorflow::CSRSparseMatMulGPUOp<Eigen::GpuDevice, std::complex<double> >::Compute(tensorflow::OpKernelContext*): error: undefined reference to 'tensorflow::functor::CSRSparseMatrixTranspose<Eigen::GpuDevice, std::complex<double> >::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)'
collect2: error: ld returned 1 exit status

Relevant log output

log

sushreebarsa · 2022-12-15T05:41:45Z

@perestoronin Thank you for raising this issue.
In order to expedite the trouble-shooting process here,could you please fill the issue template, it would be more helpful for us to analyze this issue.
Thank you!

perestoronin · 2022-12-15T13:16:33Z

@perestoronin Thank you for raising this issue. In order to expedite the trouble-shooting process here,could you please fill the issue template, it would be more helpful for us to analyze this issue. Thank you!

I filled this issue based on template

perestoronin · 2022-12-21T07:38:24Z

Same issues about undefined reference to CSRSparseMatrixTranspose:

tf2.3 c++库编译失败 #43234
Build TF 2.9.1 with use ROCM 5.2.3 with GCC 12.2.0 failed, but succesfull build with use ROCM 5.2.3 with GCC 11.3.0 or with GCC 12.2.0 without use ROCM #57371

perestoronin · 2023-01-03T21:37:52Z

with gcc 11.3.0: nm -lC transpose_op.pic.o | wgetpaste
nm -lC transpose_op.pic.o | grep functor:

                 U tensorflow::functor::UnaryFunctor<Eigen::GpuDevice, tensorflow::functor::conj<std::complex<double> > >::operator()(Eigen::GpuDevice const&, Eigen::TensorMap<Eigen::Tensor<std::complex<double>, 1, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorMap<Eigen::Tensor<std::complex<double> const, 1, 1, long>, 16, Eigen::MakePointer>)      transpose_op.cc:0
                 U tensorflow::functor::UnaryFunctor<Eigen::GpuDevice, tensorflow::functor::conj<std::complex<float> > >::operator()(Eigen::GpuDevice const&, Eigen::TensorMap<Eigen::Tensor<std::complex<float>, 1, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorMap<Eigen::Tensor<std::complex<float> const, 1, 1, long>, 16, Eigen::MakePointer>) transpose_op.cc:0
                 U tensorflow::functor::SetZeroFunctor<Eigen::ThreadPoolDevice, int>::operator()(Eigen::ThreadPoolDevice const&, Eigen::TensorMap<Eigen::Tensor<int, 1, 1, long>, 16, Eigen::MakePointer>)      transpose_op.cc:0
                 U tensorflow::functor::SetZeroFunctor<Eigen::GpuDevice, int>::operator()(Eigen::GpuDevice const&, Eigen::TensorMap<Eigen::Tensor<int, 1, 1, long>, 16, Eigen::MakePointer>)    transpose_op.cc:0
0000000000000000 W tensorflow::functor::CSRSparseMatrixTranspose<Eigen::ThreadPoolDevice, std::complex<double> >::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)
0000000000000000 W tensorflow::functor::CSRSparseMatrixTranspose<Eigen::ThreadPoolDevice, std::complex<float> >::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)
0000000000000000 W tensorflow::functor::CSRSparseMatrixTranspose<Eigen::ThreadPoolDevice, double>::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)
0000000000000000 W tensorflow::functor::CSRSparseMatrixTranspose<Eigen::ThreadPoolDevice, float>::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)
0000000000000000 W tensorflow::functor::CSRSparseMatrixTranspose<Eigen::GpuDevice, std::complex<double> >::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)
0000000000000000 W tensorflow::functor::CSRSparseMatrixTranspose<Eigen::GpuDevice, std::complex<float> >::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)
0000000000000000 W tensorflow::functor::CSRSparseMatrixTranspose<Eigen::GpuDevice, double>::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)
0000000000000000 W tensorflow::functor::CSRSparseMatrixTranspose<Eigen::GpuDevice, float>::operator()(tensorflow::OpKernelContext*, bool, tensorflow::CSRSparseMatrix const&, tensorflow::CSRSparseMatrix*)

with gcc 12.2.0: nm -lC transpose_op.pic.o | wgetpaste
nm -lC transpose_op.pic.o | grep functor:

                 U tensorflow::functor::UnaryFunctor<Eigen::GpuDevice, tensorflow::functor::conj<std::complex<double> > >::operator()(Eigen::GpuDevice const&, Eigen::TensorMap<Eigen::Tensor<std::complex<double>, 1, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorMap<Eigen::Tensor<std::complex<double> const, 1, 1, long>, 16, Eigen::MakePointer>)      transpose_op.cc:0
                 U tensorflow::functor::UnaryFunctor<Eigen::GpuDevice, tensorflow::functor::conj<std::complex<float> > >::operator()(Eigen::GpuDevice const&, Eigen::TensorMap<Eigen::Tensor<std::complex<float>, 1, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorMap<Eigen::Tensor<std::complex<float> const, 1, 1, long>, 16, Eigen::MakePointer>) transpose_op.cc:0
                 U tensorflow::functor::SetZeroFunctor<Eigen::ThreadPoolDevice, int>::operator()(Eigen::ThreadPoolDevice const&, Eigen::TensorMap<Eigen::Tensor<int, 1, 1, long>, 16, Eigen::MakePointer>)      transpose_op.cc:0
                 U tensorflow::functor::SetZeroFunctor<Eigen::GpuDevice, int>::operator()(Eigen::GpuDevice const&, Eigen::TensorMap<Eigen::Tensor<int, 1, 1, long>, 16, Eigen::MakePointer>)    transpose_op.cc:0

Why with compile gcc 12.2.0 tensorflow::functor::CSRSparseMatrixTranspose missing in bazel-out/k8-opt/bin/tensorflow/core/kernels/sparse/_objs/kernels/transpose_op.pic.o ? @sushreebarsa

perestoronin · 2023-01-05T11:10:55Z

https://github.com/roelaaij/gentoo-overlay/blob/master/sci-libs/tensorflow/files/

perestoronin · 2023-01-05T13:39:58Z

This patch solved issue with gcc 12.2.0

# cat tf-sparse-transpose-op.patch 
--- a/tensorflow/core/kernels/sparse/transpose_op.cc
+++ b/tensorflow/core/kernels/sparse/transpose_op.cc
@@ -208,6 +208,11 @@
   return OkStatus();
 }
 
+template struct CSRSparseMatrixTranspose<GPUDevice, float>;
+template struct CSRSparseMatrixTranspose<GPUDevice, double>;
+template struct CSRSparseMatrixTranspose<GPUDevice, std::complex<float>>;
+template struct CSRSparseMatrixTranspose<GPUDevice, std::complex<double>>;
+
 // CPU kernel for transposing a single component of a CSR SparseMatrix.
 template <typename T>
 struct CSRSparseMatrixTransposeComponent<CPUDevice, T> {

For cuda also usefull patch

google-ml-butler · 2023-01-05T13:40:01Z

Are you satisfied with the resolution of your issue?
Yes
No

sushreebarsa · 2023-01-05T13:56:10Z

@perestoronin Thank you for the update!

svenstaro · 2023-02-19T13:47:20Z

This isn't upstreamed. Why is this closed? Current master branch still doesn't compile with gcc 12.

plopresti · 2023-07-25T01:14:11Z

@svenstaro Appears to be fixed on master by b8ac45d.

Still broken in r2.13.

The template CSRSparseMatrixTranspose, was used in a file where the template definition was unavailable. This requires the template to be explicitly instantiated, but it wasn't explicitly instantiated before. This issue has been around for a long time. I'm not sure why this wasn't failing before, but it's now failing for me. PiperOrigin-RevId: 533344851

google-ml-butler bot assigned tilakrayal Dec 14, 2022

sushreebarsa assigned sushreebarsa and unassigned tilakrayal Dec 14, 2022

sushreebarsa added type:build/install Build and install issues TF 2.11 Issues related to TF 2.11 labels Dec 15, 2022

sushreebarsa added the stat:awaiting response Status - Awaiting response from author label Dec 15, 2022

perestoronin mentioned this issue Dec 15, 2022

Build TF 2.11.0 with use ROCM 5.4.0 with GCC 12.2.0 failed, but succesfull build with use ROCM 5.4.0 with GCC 11.3.0 #58900

Closed

google-ml-butler bot removed the stat:awaiting response Status - Awaiting response from author label Dec 15, 2022

perestoronin changed the title ~~Build TF 2.11.0 with use ROCM 5.4.0 with GCC 12.2.0 failed, but succesfull build with use ROCM 5.4.0 with GCC 11.3.0~~ Build TF 2.11.0 with use ROCM 5.4.1 with GCC 12.2.0 failed, but succesfull build with use ROCM 5.4.1with GCC 11.3.0 Jan 3, 2023

perestoronin changed the title ~~Build TF 2.11.0 with use ROCM 5.4.1 with GCC 12.2.0 failed, but succesfull build with use ROCM 5.4.1with GCC 11.3.0~~ Build TF 2.11.0 with use ROCM 5.4.1 with GCC 12.2.0 failed, but succesfull build with use ROCM 5.4.1 with GCC 11.3.0 Jan 3, 2023

perestoronin changed the title ~~Build TF 2.11.0 with use ROCM 5.4.1 with GCC 12.2.0 failed, but succesfull build with use ROCM 5.4.1 with GCC 11.3.0~~ With compile gcc 12.2.0 tensorflow::functor::CSRSparseMatrixTranspose missing Jan 3, 2023

perestoronin closed this as completed Jan 5, 2023

linuxmetel mentioned this issue Mar 20, 2023

Add handling of TensorRT's new enumeration value nvinfer1::DataType::kFP8 #60046

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

With compile gcc 12.2.0 tensorflow::functor::CSRSparseMatrixTranspose missing #58881

With compile gcc 12.2.0 tensorflow::functor::CSRSparseMatrixTranspose missing #58881

perestoronin commented Dec 14, 2022 •

edited

Issue Type

Source

Tensorflow Version

Custom Code

OS Platform and Distribution

Mobile device

Python version

Bazel version

GCC/Compiler version

CUDA/cuDNN version

GPU model and memory

Current Behaviour?

Standalone code to reproduce the issue

Relevant log output

sushreebarsa commented Dec 15, 2022

perestoronin commented Dec 15, 2022

perestoronin commented Dec 21, 2022

perestoronin commented Jan 3, 2023

perestoronin commented Jan 5, 2023 •

edited

perestoronin commented Jan 5, 2023 •

edited

google-ml-butler bot commented Jan 5, 2023

sushreebarsa commented Jan 5, 2023

svenstaro commented Feb 19, 2023

plopresti commented Jul 25, 2023 •

edited

With compile gcc 12.2.0 tensorflow::functor::CSRSparseMatrixTranspose missing #58881

With compile gcc 12.2.0 tensorflow::functor::CSRSparseMatrixTranspose missing #58881

Comments

perestoronin commented Dec 14, 2022 • edited

Issue Type

Source

Tensorflow Version

Custom Code

OS Platform and Distribution

Mobile device

Python version

Bazel version

GCC/Compiler version

CUDA/cuDNN version

GPU model and memory

Current Behaviour?

Standalone code to reproduce the issue

Relevant log output

sushreebarsa commented Dec 15, 2022

perestoronin commented Dec 15, 2022

perestoronin commented Dec 21, 2022

perestoronin commented Jan 3, 2023

perestoronin commented Jan 5, 2023 • edited

perestoronin commented Jan 5, 2023 • edited

google-ml-butler bot commented Jan 5, 2023

sushreebarsa commented Jan 5, 2023

svenstaro commented Feb 19, 2023

plopresti commented Jul 25, 2023 • edited

perestoronin commented Dec 14, 2022 •

edited

perestoronin commented Jan 5, 2023 •

edited

perestoronin commented Jan 5, 2023 •

edited

plopresti commented Jul 25, 2023 •

edited