Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C++] Linker Error when building arrow bundled dependencies with MSVC #36117

Closed
GupAChup opened this issue Jun 15, 2023 · 11 comments · Fixed by #36160
Closed

[C++] Linker Error when building arrow bundled dependencies with MSVC #36117

GupAChup opened this issue Jun 15, 2023 · 11 comments · Fixed by #36160
Assignees
Milestone

Comments

@GupAChup
Copy link

Describe the bug, including details regarding any error messages, version, and platform.

I was trying to do a static build of arrow in my project and ran into a linker error. Here are the exact commands I ran:

git clone https://github.com/apache/arrow.git
cd arrow/cpp
mkdir build
cd build
cmake -DARROW_PARQUET=ON -DARROW_CSV=ON -DARROW_BUILD_STATIC=ON ..
cmake --build . --target install

After running those commands, during the build process, I get the following error.

  Performing install step for 'thrift_ep'
  -- thrift_ep install command succeeded.  See also C:/Users/guppy/CLionProjects/NewDataframes/libs/arrow/cpp/build/thrift_ep-prefix/src/thrift_ep-stamp/th
  rift_ep-install-*.log
  Completed 'thrift_ep'
  Bundling C:/Users/guppy/CLionProjects/NewDataframes/libs/arrow/cpp/build/release/arrow_bundled_dependencies.lib
LINK : fatal error LNK1370: failed to copy file 'C:\Users\guppy\CLionProjects\NewDataframes\libs\arrow\cpp\build\thrift_ep-install\bin\thriftmd.lib'; error
 code 3 [C:\Users\guppy\CLionProjects\NewDataframes\libs\arrow\cpp\build\src\arrow\arrow_bundled_dependencies_merge.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(247,5): error MSB8066: Custom build for 'C:\Users\guppy\CLionProjects\NewDataframes\libs\arrow\cpp\build\CMakeFiles\155e07f2be9d7893eea39b427b71fbbb\arrow_bundled_dependencies_merge.rule;C:\Users\guppy\CLionProjects\NewDataframes\libs\arrow\cpp\src\arrow\CMakeLists.txt' exited with code 1370. [C:\Users\guppy\CLionProjects\NewDataframes\libs\arrow\cpp\build\src\arrow\arrow_bundled_dependencies_merge.vcxproj]

Any help would be greatly appreciated.

Component(s)

C++

@kou
Copy link
Member

kou commented Jun 16, 2023

Could you specify -DCMAKE_BUILD_TYPE=Release explicitly?

git clone https://github.com/apache/arrow.git
cd arrow/cpp
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DARROW_PARQUET=ON -DARROW_CSV=ON -DARROW_BUILD_STATIC=ON ..
cmake --build . --config Release --target install

@GupAChup
Copy link
Author

I ran each of those commands specifically while explicitly stating the build type and It seems to produce the same error.

  Performing install step for 'thrift_ep'
  -- thrift_ep install command succeeded.  See also C:/Users/guppy/CLionProjects/NewDataframes/libs/arrow/arrow/cpp/build/thrift_ep-prefix/src/thrift_ep-stamp/thr
  ift_ep-install-*.log
  Completed 'thrift_ep'
  Bundling C:/Users/guppy/CLionProjects/NewDataframes/libs/arrow/arrow/cpp/build/release/arrow_bundled_dependencies.lib
LINK : fatal error LNK1370: failed to copy file 'C:\Users\guppy\CLionProjects\NewDataframes\libs\arrow\arrow\cpp\build\thrift_ep-install\bin\thriftmd.lib'; error
code 3 [C:\Users\guppy\CLionProjects\NewDataframes\libs\arrow\arrow\cpp\build\src\arrow\arrow_bundled_dependencies_merge.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(247,5): error MSB8066: Custom build for 'C:\Users\gu
ppy\CLionProjects\NewDataframes\libs\arrow\arrow\cpp\build\CMakeFiles\294c83af6aef239dd1ca2299792d917b\arrow_bundled_dependencies_merge.rule;C:\Users\guppy\CLionP
rojects\NewDataframes\libs\arrow\arrow\cpp\src\arrow\CMakeLists.txt' exited with code 1370. [C:\Users\guppy\CLionProjects\NewDataframes\libs\arrow\arrow\cpp\build
\src\arrow\arrow_bundled_dependencies_merge.vcxproj]

@kou
Copy link
Member

kou commented Jun 16, 2023

Could you show the outpuf of dir C:\Users\guppy\CLionProjects\NewDataframes\libs\arrow\arrow\cpp\build\thrift_ep-install\bin\?

@GupAChup
Copy link
Author

This is the output:

 Directory of C:\Users\guppy\CLionProjects\NewDataframes\libs\arrow\arrow\cpp\build\thrift_ep-install\bin

06/15/2023  05:35 PM    <DIR>          .
06/15/2023  05:35 PM    <DIR>          ..
06/15/2023  05:35 PM         4,586,458 thriftmd.lib
               1 File(s)      4,586,458 bytes
               2 Dir(s)  289,212,059,648 bytes free

@kou
Copy link
Member

kou commented Jun 16, 2023

Hmm... Could you add a debug message and run again?

diff --git a/cpp/cmake_modules/BuildUtils.cmake b/cpp/cmake_modules/BuildUtils.cmake
index aaf4b0bb5..3bf44c4d2 100644
--- a/cpp/cmake_modules/BuildUtils.cmake
+++ b/cpp/cmake_modules/BuildUtils.cmake
@@ -140,7 +140,7 @@ function(arrow_create_merged_static_lib output_target)
                     ${BUNDLE_COMMAND}
                     DEPENDS ${ARG_ROOT} ${ARG_TO_MERGE}
                     BYPRODUCTS ${output_lib_path}
-                    COMMENT "Bundling ${output_lib_path}"
+                    COMMENT "Bundling ${output_lib_path}: ${BUNDLE_COMMAND}"
                     VERBATIM)
 
   message(STATUS "Creating bundled static library target ${output_target} at ${output_lib_path}"

@GupAChup
Copy link
Author

Here's the output from that comment:

Performing install step for 'thrift_ep'
  -- thrift_ep install command succeeded.  See also C:/Users/guppy/CLionProjects/NewDataframes/libs/arrow/arrow/cpp/build/thrift_ep-prefix/src/thrift_ep-stamp/thr
  ift_ep-install-*.log
  Completed 'thrift_ep'
  Bundling C:/Users/guppy/CLionProjects/NewDataframes/libs/arrow/arrow/cpp/build/release/arrow_bundled_dependencies.lib: C:/Program Files/Microsoft Visual Studio/
  2022/Community/VC/Tools/MSVC/14.34.31933/bin/Hostx64/x64/lib.exe;/NOLOGO;/OUT:C:/Users/guppy/CLionProjects/NewDataframes/libs/arrow/arrow/cpp/build/release/arro
  w_bundled_dependencies.lib;C:/Users/guppy/CLionProjects/NewDataframes/libs/arrow/arrow/cpp/build/thrift_ep-install/bin/thriftmd.lib
LINK : fatal error LNK1370: failed to copy file 'C:\Users\guppy\CLionProjects\NewDataframes\libs\arrow\arrow\cpp\build\thrift_ep-install\bin\thriftmd.lib'; error
code 3 [C:\Users\guppy\CLionProjects\NewDataframes\libs\arrow\arrow\cpp\build\src\arrow\arrow_bundled_dependencies_merge.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(247,5): error MSB8066: Custom build for 'C:\Users\gu
ppy\CLionProjects\NewDataframes\libs\arrow\arrow\cpp\build\CMakeFiles\294c83af6aef239dd1ca2299792d917b\arrow_bundled_dependencies_merge.rule;C:\Users\guppy\CLionP
rojects\NewDataframes\libs\arrow\arrow\cpp\src\arrow\CMakeLists.txt' exited with code 1370. [C:\Users\guppy\CLionProjects\NewDataframes\libs\arrow\arrow\cpp\build
\src\arrow\arrow_bundled_dependencies_merge.vcxproj]

@kou
Copy link
Member

kou commented Jun 16, 2023

Could you run C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.34.31933/bin/Hostx64/x64/lib.exe /NOLOGO /OUT:C:/Users/guppy/CLionProjects/NewDataframes/libs/arrow/arrow/cpp/build/release/arrow_bundled_dependencies.lib C:/Users/guppy/CLionProjects/NewDataframes/libs/arrow/arrow/cpp/build/thrift_ep-install/bin/thriftmd.lib?

@GupAChup
Copy link
Author

Hopefully, I ran this correctly, but I'm uncertain because I got an error. I navigated to the folder

C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.34.31933\bin\Hostx64\x64>

and then ran the following command:

lib.exe /NOLOGO /OUT:C:/Users/guppy/CLionProjects/NewDataframes/libs/arrow/arrow/cpp/build/release/arrow_bundled_dependencies.lib C:/Users/guppy/CLionProjects/NewDataframes/libs/arrow/arrow/cpp/build/thrift_ep-install/bin/thriftmd.lib

and i got this error:

LINK : fatal error LNK1370: failed to copy file 'C:\Users\guppy\CLionProjects\NewDataframes\libs\arrow\arrow\cpp\build\thrift_ep-install\bin\thriftmd.lib'; error code 3

@kou
Copy link
Member

kou commented Jun 16, 2023

Oh... We can't get more information...
Sorry. I can't help you more without reproduced environment...
Could you debug the command line on your environment?

@Jenovesan
Copy link

@kou I got the same exact same error on a fresh windows install

@kou kou changed the title Linker Error when building arrow bundled dependencies [C++] Linker Error when building arrow bundled dependencies on Windows Jun 16, 2023
@kou kou changed the title [C++] Linker Error when building arrow bundled dependencies on Windows [C++] Linker Error when building arrow bundled dependencies with MSVC Jun 16, 2023
@kou
Copy link
Member

kou commented Jun 16, 2023

Does this work?

diff --git a/cpp/cmake_modules/BuildUtils.cmake b/cpp/cmake_modules/BuildUtils.cmake
index aaf4b0bb5..a9bdd0391 100644
--- a/cpp/cmake_modules/BuildUtils.cmake
+++ b/cpp/cmake_modules/BuildUtils.cmake
@@ -86,6 +86,7 @@ function(arrow_create_merged_static_lib output_target)
     message(SEND_ERROR "Error: unrecognized arguments: ${ARG_UNPARSED_ARGUMENTS}")
   endif()
 
+  file(MAKE_DIRECTORY ${BUILD_OUTPUT_ROOT_DIRECTORY})
   set(output_lib_path
       ${BUILD_OUTPUT_ROOT_DIRECTORY}${CMAKE_STATIC_LIBRARY_PREFIX}${ARG_NAME}${CMAKE_STATIC_LIBRARY_SUFFIX}
   )

kou added a commit to kou/arrow that referenced this issue Jun 19, 2023
If it doesn't exist when we build arrow_bundled_dependencies.lib,
"lib.exe ..." failed.

This also includes cleanup of lib.exe detection related codes but it's
not related of this fix. Sorry. "file(MAKE_DIRECTORY)" is only
important.
kou added a commit that referenced this issue Jun 21, 2023
### Rationale for this change

If it doesn't exist when we build arrow_bundled_dependencies.lib, "lib.exe ..." failed.

### What changes are included in this PR?

Ensure creating it in `cmake ...` phase not build phrase.

This also includes cleanup of lib.exe detection related codes but it's not related of this fix. Sorry. "file(MAKE_DIRECTORY)" is only important.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* Closes: #36117

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@kou kou added this to the 13.0.0 milestone Jun 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants