Unable to compile torchtext v0.13 #1797

cdeepali · 2022-06-20T15:07:35Z

🐛 Bug

Seeing the following error while building torchtext:

-- Found CUDA: /usr/local/cuda (found version "11.4")
-- The CUDA compiler identification is unknown
-- Detecting CUDA compiler ABI info

CMake Error in <myhome>/work/build/temp.linux-ppc64le-3.9/CMakeFiles/CMakeTmp/CMakeLists.txt:
  CUDA_ARCHITECTURES is empty for target "cmTC_94ffb".


CMake Error in <myhome>/work/build/temp.linux-ppc64le-3.9/CMakeFiles/CMakeTmp/CMakeLists.txt:
  CUDA_ARCHITECTURES is empty for target "cmTC_94ffb".


CMake Error at <myhome>/_build_env/share/cmake-3.19/Modules/CMakeDetermineCompilerABI.cmake:48 (try_compile):
  Failed to generate test project build system.

To Reproduce :

Build torchtext tag https://github.com/pytorch/text/releases/tag/v0.13.0-rc2

Expected behavior .
It should build successfully.

Environment

OS: Red Hat Enterprise Linux 8.5 (Ootpa) (ppc64le)
GCC version: Anaconda GCC 11.2
Python version: 3.9.12 (main, Apr 5 2022, 07:09:29)
CUDA runtime version: 11.4.152

The text was updated successfully, but these errors were encountered:

parmeet · 2022-06-20T15:50:55Z

Hi @cdeepali, torchtext does not depend on CUDA. Is there a reason you are building torchtext with FORCE_CUDA=1?

cdeepali · 2022-06-20T16:28:14Z

Yes thats correct. But my build is failing without it as well, so I had just tried passing FORCE_CUDA=1. But my build fails even if I don't pass it.

atalman · 2022-06-20T17:17:01Z

@cdeepali could you please post complete compile log with errors ?

cdeepali · 2022-06-21T05:20:17Z

I am actually trying to build torchtext when CUDA enabled pytorch is installed in the build environment. As a result cmake detects CUDA on its own and eventually fails:

full log:

++ python -V
++ cut -d ' ' -f 2
++ cut -d . -f 1,2
+ PYTHON_VERSION=3.9
+ PACKAGE_PATH=<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/
+ python setup.py install --single-version-externally-managed --record=record.txt
<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/torch/cuda/__init__.py:83: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at  <myhome>work/c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() > 0
<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
CMake Error in <myhome>work/build/temp.linux-ppc64le-3.9/CMakeFiles/CMakeTmp/CMakeLists.txt:
  CUDA_ARCHITECTURES is empty for target "cmTC_dffb8".


CMake Error in <myhome>work/build/temp.linux-ppc64le-3.9/CMakeFiles/CMakeTmp/CMakeLists.txt:
  CUDA_ARCHITECTURES is empty for target "cmTC_dffb8".


CMake Error at <myhome>_build_env/share/cmake-3.19/Modules/CMakeDetermineCompilerABI.cmake:48 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  <myhome>_build_env/share/cmake-3.19/Modules/CMakeTestCUDACompiler.cmake:19 (CMAKE_DETERMINE_COMPILER_ABI)
  <myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:47 (enable_language)
  <myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
  <myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:54 (find_package)
  
Traceback (most recent call last):
  File "<myhome>work/setup.py", line 109, in <module>
    setup(**setup_info)
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 148, in setup
    return run_commands(dist)
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
    dist.run_commands()
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/dist.py", line 1214, in run_command
    super().run_command(command)
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/command/install.py", line 68, in run
    return orig.install.run(self)
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/_distutils/command/install.py", line 664, in run
    self.run_command('build')
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/dist.py", line 1214, in run_command
    super().run_command(command)
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/dist.py", line 1214, in run_command
    super().run_command(command)
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "<myhome>work/build_tools/setup_helpers/extension.py", line 42, in run
    super().run()
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 79, in run
    _build_ext.run(self)
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 448, in build_extensions
    self._build_extensions_serial()
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 473, in _build_extensions_serial
    self.build_extension(ext)
  File "<myhome>work/build_tools/setup_helpers/extension.py", line 102, in build_extension
    subprocess.check_call(["cmake", str(_ROOT_DIR)] + cmake_args, cwd=self.build_temp)
  File "<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '<myhome>work', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_PREFIX_PATH=<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.9/site-packages/torch/share/cmake', '-DCMAKE_INSTALL_PREFIX=<myhome>work/build/lib.linux-ppc64le-3.9/torchtext/', '-DCMAKE_VERBOSE_MAKEFILE=ON', '-DPython_INCLUDE_DIR=<myhome>_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include/python3.9', '-DBUILD_TORCHTEXT_PYTHON_EXTENSION:BOOL=ON', '-DRE2_BUILD_TESTING:BOOL=OFF', '-DBUILD_TESTING:BOOL=OFF', '-DBUILD_SHARED_LIBS=OFF', '-DCMAKE_POLICY_DEFAULT_CMP0063=NEW', '-DSPM_ENABLE_SHARED=OFF']' returned non-zero exit status 1.

This issue was not seen until v0.12.

cdeepali · 2022-06-22T13:48:43Z

@atalman, I have now found a workaround for this issue. I have commented cmake code to look for torch https://github.com/pytorch/text/blob/v0.13.0-rc2/CMakeLists.txt#L54 and made local changes to link torchtext.so with required torch libraries in my env.

However, I would like to understand if we expect successful torchtext build in an environment where cudatoolkit is also installed? As prior to v0.13 this used to work.

parmeet · 2022-06-22T13:54:55Z

cc: @Nayef211

Nayef211 · 2022-06-22T20:41:15Z

@atalman, I have now found a workaround for this issue. I have commented cmake code to look for torch https://github.com/pytorch/text/blob/v0.13.0-rc2/CMakeLists.txt#L54 and made local changes to link torchtext.so with required torch libraries in my env.

However, I would like to understand if we expect successful torchtext build in an environment where cudatoolkit is also installed? As prior to v0.13 this used to work.

@cdeepali would you mind sharing the code you used to link torchtext.so with required torch libraries?

cc @mthrok do you know if torchaudio users have run into similar issues in the past?

mthrok · 2022-06-22T22:34:46Z

I think the issue is somehow related to the environment.
I get the impression that CMake's CUDA detection fails in ppc64le.

Searching for the same error, got this SO,
which suggests manually setting the CPU architecture is one possible workaround.
Though I am not sure what the right way to do. Maybe setting CMAKE_HOST_SYSTEM_PROCESSOR, might do

cdeepali · 2022-06-23T09:41:19Z

Although I reported this for ppc as I tried the build first on Power platform the issue is applicable to x86 as well. I observed same error on x86 in a trial I did today:

-- Found CUDA: /usr/local/cuda (found version "11.4")
-- The CUDA compiler identification is unknown
-- Detecting CUDA compiler ABI info

cdeepali · 2022-06-23T10:20:24Z

@Nayef211, I used the following patch to link torchtext with the required torch libraries:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 80e98f42..09e82e14 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -51,13 +51,15 @@ option(BUILD_TORCHTEXT_PYTHON_EXTENSION "Build Python extension" OFF)
 
 set(CMAKE_MODULE_PATH "${CMAKE_MODULE_PATH};${CMAKE_CURRENT_SOURCE_DIR}/cmake")
 
-find_package(Torch REQUIRED)
+set(TORCH_INSTALL_PREFIX $ENV{TORCH_INSTALL_PREFIX})
+find_library(TORCH_C10_LIBRARY c10 PATHS "${TORCH_INSTALL_PREFIX}/lib")
+find_library(TORCH_LIBRARY torch PATHS "${TORCH_INSTALL_PREFIX}/lib")
+find_library(TORCH_CPU_LIBRARY torch_cpu PATHS "${TORCH_INSTALL_PREFIX}/lib")
 
 if(MSVC)
   set(CMAKE_MSVC_RUNTIME_LIBRARY "MultiThreaded$<$<CONFIG:Debug>:Debug>")
 endif()
 
-
 # TORCH_CXX_FLAGS contains the same -D_GLIBCXX_USE_CXX11_ABI value as PyTorch
 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall ${TORCH_CXX_FLAGS}")
 
diff --git a/torchtext/csrc/CMakeLists.txt b/torchtext/csrc/CMakeLists.txt
index 93a6624f..642f5b03 100644
--- a/torchtext/csrc/CMakeLists.txt
+++ b/torchtext/csrc/CMakeLists.txt
@@ -23,6 +23,8 @@ set(
 
 set(
   LIBTORCHTEXT_INCLUDE_DIRS
+  ${TORCH_INSTALL_PREFIX}/include
+  ${TORCH_INSTALL_PREFIX}/include/torch/csrc/api/include
   ${PROJECT_SOURCE_DIR}
   ${PROJECT_SOURCE_DIR}/third_party/sentencepiece/src
   $<TARGET_PROPERTY:re2,INCLUDE_DIRECTORIES>
@@ -32,7 +34,9 @@ set(
 
 set(
   LIBTORCHTEXT_LINK_LIBRARIES
-  torch
+  ${TORCH_C10_LIBRARY}
+  ${TORCH_LIBRARY}
+  ${TORCH_CPU_LIBRARY}
   re2
   double-conversion
   sentencepiece
@@ -123,6 +127,8 @@ if (BUILD_TORCHTEXT_PYTHON_EXTENSION)
     $<TARGET_PROPERTY:re2,INCLUDE_DIRECTORIES>
     $<TARGET_PROPERTY:double-conversion,INCLUDE_DIRECTORIES>
     $<TARGET_PROPERTY:utf8proc,INCLUDE_DIRECTORIES>
+    ${TORCH_INSTALL_PREFIX}/include
+    ${TORCH_INSTALL_PREFIX}/include/torch/csrc/api/include
     )
 
   set(

Please let me know if you see any issue with this.

cdeepali · 2022-06-28T04:06:22Z

@Nayef211, pls let me know your view on the above patch.

Nayef211 · 2022-06-28T17:04:14Z

Hey @cdeepali. Let me create a PR later today incorporating some of your changes!

cdeepali · 2022-06-29T04:57:37Z

Sure @Nayef211.

Nayef211 · 2022-06-29T17:47:18Z

@cdeepali quick question, what is the value of your TORCH_INSTALL_PREFIX env variable? The build is erroring out for me locally on MacOS due to the env variable not being set.

cdeepali · 2022-06-30T04:52:25Z

I have it set to the following:

PACKAGE_PATH=$PREFIX/lib/python${PYTHON_VERSION}/site-packages/
export TORCH_INSTALL_PREFIX=$PACKAGE_PATH/torch/

and PYTHON_VERSION=3.9

wuxxin · 2022-07-07T15:40:30Z

torchtext 0.13 also fails to build on rocm, preliminary patch fixes build too.
See https://github.com/rocm-arch/python-pytorch-rocm/issues/46

Nayef211 · 2022-07-08T15:39:44Z

@cdeepali @wuxxin I've merged in #1814. Can you please verify if the build is fixed on your end before I close out this issue?

wuxxin · 2022-07-08T16:58:52Z

does not apply to released version (0.13), need to rebase patch for moved setup_helpers dir, will report once done

mthrok · 2022-07-08T17:07:21Z

I think PyTorch plans to do minor release 1.12.1 due to pytorch/pytorch#80569. So I think we can include #1814 in torchtext 0.13.1 release.

cdeepali · 2022-07-11T13:09:03Z

@Nayef211, I have verified with #1814 on my environment and it works. Thanks for adding the fix.

atalman self-assigned this Jun 20, 2022

cdeepali changed the title ~~Unable to compile torchtext v0.13 with CUDA~~ Unable to compile torchtext v0.13 Jun 20, 2022

Nayef211 mentioned this issue Jun 29, 2022

Fixing build when CUDA enabled torch is installed #1814

Merged

mthrok mentioned this issue Jul 8, 2022

TorchText patch for PyTorch 1.12.1 minor release pytorch/pytorch#81117

Closed

atalman assigned Nayef211 and unassigned atalman Jul 8, 2022

Nayef211 closed this as completed Jul 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to compile torchtext v0.13 #1797

Unable to compile torchtext v0.13 #1797

cdeepali commented Jun 20, 2022 •

edited

parmeet commented Jun 20, 2022

cdeepali commented Jun 20, 2022

atalman commented Jun 20, 2022

cdeepali commented Jun 21, 2022

cdeepali commented Jun 22, 2022 •

edited

parmeet commented Jun 22, 2022

Nayef211 commented Jun 22, 2022

mthrok commented Jun 22, 2022

cdeepali commented Jun 23, 2022

cdeepali commented Jun 23, 2022

cdeepali commented Jun 28, 2022

Nayef211 commented Jun 28, 2022

cdeepali commented Jun 29, 2022

Nayef211 commented Jun 29, 2022

cdeepali commented Jun 30, 2022

wuxxin commented Jul 7, 2022

Nayef211 commented Jul 8, 2022

wuxxin commented Jul 8, 2022

mthrok commented Jul 8, 2022

cdeepali commented Jul 11, 2022

Unable to compile torchtext v0.13 #1797

Unable to compile torchtext v0.13 #1797

Comments

cdeepali commented Jun 20, 2022 • edited

🐛 Bug

parmeet commented Jun 20, 2022

cdeepali commented Jun 20, 2022

atalman commented Jun 20, 2022

cdeepali commented Jun 21, 2022

cdeepali commented Jun 22, 2022 • edited

parmeet commented Jun 22, 2022

Nayef211 commented Jun 22, 2022

mthrok commented Jun 22, 2022

cdeepali commented Jun 23, 2022

cdeepali commented Jun 23, 2022

cdeepali commented Jun 28, 2022

Nayef211 commented Jun 28, 2022

cdeepali commented Jun 29, 2022

Nayef211 commented Jun 29, 2022

cdeepali commented Jun 30, 2022

wuxxin commented Jul 7, 2022

Nayef211 commented Jul 8, 2022

wuxxin commented Jul 8, 2022

mthrok commented Jul 8, 2022

cdeepali commented Jul 11, 2022

cdeepali commented Jun 20, 2022 •

edited

cdeepali commented Jun 22, 2022 •

edited