Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TF fails to build on CPU when compiled with "-march=skylake-avx512" flag #56731

Closed
bhavani-subramanian opened this issue Jul 11, 2022 · 5 comments
Closed

TF fails to build on CPU when compiled with "-march=skylake-avx512" flag #56731

bhavani-subramanian opened this issue Jul 11, 2022 · 5 comments
Assignees
@cantonios @tilakrayal
Labels
stat:awaiting tensorflower Status - Awaiting response from tensorflower TF 2.9 Issues found in the TF 2.9 release (or RCs) type:build/install Build and install issues

Comments

@bhavani-subramanian
Copy link
Contributor

bhavani-subramanian commented Jul 11, 2022 •
edited

Click to expand!

Issue Type

Build/Install

Source

source

Tensorflow Version

tf 2.9

Custom Code

No

OS Platform and Distribution

Linux Ubuntu 20.04

Mobile device

No response

Python version

3.9.12

Bazel version

5.1.1

GCC/Compiler version

9.3.0

CUDA/cuDNN version

No response

GPU model and memory

No response

Current Behaviour?

Starting from 74059203de9c21ca26b27ea7609d49b3c87f2c19 commit, TF fails to build when compiled with `--copt=-march=skylake=avx512` flag on Cascade Lake (CLX) CPU systems or higher. The compilation succeeds when this flag is not passed.

Standalone code to reproduce the issue

Command-line to trigger the build failure (on 74059203de9c21ca26b27ea7609d49b3c87f2c19 or newer commit):
bazel build --copt="-O3" --copt=-march=skylake-avx512 -c opt //tensorflow/tools/pip_package:build_pip_package

Relevant log output

external/eigen_archive/unsupported/Eigen/CXX11/src/Tensor/TensorContraction.h:250:5: error: ambiguous template instantiation for 'struct EigenForTFLite::internal::gemm_pack_rhs<float, long int, EigenForTFLite::internal::TensorContractionSubMapper<float, long int, 0, EigenForTFLite::TensorEvaluator<const EigenForTFLite::TensorReshapingOp<const EigenForTFLite::DSizes<long int, 2>, const EigenForTFLite::TensorImagePatchOp<-1, -1, const EigenForTFLite::TensorMap<EigenForTFLite::Tensor<const float, 4, 1, long int>, 16> > >, EigenForTFLite::ThreadPoolDevice>, std::array<long int, 1>, std::array<long int, 1>, 16, true, false, 0, EigenForTFLite::MakePointer>, 8, 0, false, false>' 
  250 |     RhsPacker()(*rhsBlock, data_mapper, depth, cols); 
      |     ^~~~~~~~~~~
@google-ml-butler google-ml-butler bot added the type:build/install Build and install issues label Jul 11, 2022
@bhavani-subramanian
Copy link
Contributor Author

Tagging @cantonios and @penpornk to take a look at this issue.

@cantonios
Copy link
Contributor

@bhavani-subramanian can you try hacking this at the top of tensorflow/core/kernels/eigen_spatial_convolutions.h, before including Eigen's Tensor header?

// Hack to disable breaking AVX512 special GemmKernel.
// There is a conflicting specialization there causing build breakages.
#define GEMM_KERNEL_H disabled

I think this should solve it temporarily to unblock you, but I'm having trouble actually building this on our end.

copybara-service bot pushed a commit that referenced this issue Jul 12, 2022
@cantonios @tensorflower-gardener
Disable Eigen's new AVX512 gemm kernels.
e782bef 
A recent addition from intel into Eigen ([!972](https://gitlab.com/libeigen/eigen/-/merge_requests/972))
added new specialized GEMM kernels for AVX512. Unfortunately, these conflict with and are
incompatible with specializations in `eigen_spatial_convolutions.h` in TensorFlow.
Here we put in a hack to disable them.

See issue #56731.

PiperOrigin-RevId: 460342434
@agramesh1
Copy link
Contributor

@cantonios thank for the quick response. We tried the change, it looks like it is still failing with ambiguous template instantiation error.

./tensorflow/core/kernels/eigen_contraction_kernel.h:665:9: error: ambiguous template instantiation for 'struct Eigen::internal::gemm_pack_rhs<float, long int, Eigen::internal::TensorContractionSubMapper<float, long int, 0, Eigen::TensorEvaluator<const Eigen::TensorReshapingOp<const Eigen::DSizes<long int, 2>, const Eigen::TensorImagePatchOp<-1, -1, const Eigen::TensorMap<Eigen::Tensor<const float, 4, 1, long int>, 16, Eigen::MakePointer> > >, Eigen::ThreadPoolDevice>, Eigen::array<long int, 1>, Eigen::array<long int, 1>, 16, true, true, 0, Eigen::MakePointer>, 8, 0, false, false>'
         EigenRhsPacker()(rhsBlock->packed_data, data_mapper, depth, cols);     \
         ^~~~~~~~~~~~~~~~

@tilakrayal tilakrayal added TF 2.9 Issues found in the TF 2.9 release (or RCs) stat:awaiting tensorflower Status - Awaiting response from tensorflower labels Jul 12, 2022
copybara-service bot pushed a commit that referenced this issue Jul 12, 2022
@cantonios @tensorflower-gardener
Disable Eigen's new AVX512 gemm kernels again.
a92b702 
A recent addition from intel into Eigen (!972)
added new specialized GEMM kernels for AVX512. Unfortunately, these conflict with and are
incompatible with specializations in `eigen_spatial_convolutions.h` in TensorFlow.
Here we put in a hack to disable them.

The last attempt to fix this failed, since other headers are sometimes included
before this the `eigen_spatial_convolutions.h` one, which end up including
the conflicting specialized gemm kernel before we have the chance to disable it.
The only option is to disable via `defines` in the BUILD file so that all dependent
targets inherit this option.

See issue #56731.

PiperOrigin-RevId: 460533762
@agramesh1
Copy link
Contributor

agramesh1 commented Jul 12, 2022 •
edited

@cantonios the commit a92b702 fixes the issue, TF builds with AVX512. Thanks.

@google-ml-butler
Copy link

Are you satisfied with the resolution of your issue?
Yes
No

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cantonios cantonios

@tilakrayal tilakrayal

Labels
stat:awaiting tensorflower Status - Awaiting response from tensorflower TF 2.9 Issues found in the TF 2.9 release (or RCs) type:build/install Build and install issues
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@cantonios @agramesh1 @bhavani-subramanian @tilakrayal