[C++] Support LTO for R

[asfimport]

The next version of R might enable LTO on Windows, i.e. R packages will be compiled with -flto by default. This works out of the box for most packages, but for arrow, the linker crashes as below.

 C:/rtools40/mingw64/bin/g++ -shared -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -flto -s -static-libgcc -o arrow.dll tmp.def array.o array_from_vector.o array_to_vector.o arraydata.o arrowExports.o buffer.o chunkedarray.o compression.o compute.o csv.o dataset.o datatype.o expression.o feather.o field.o filesystem.o imports.o io.o json.o memorypool.o message.o parquet.o py-to-r.o recordbatch.o recordbatchreader.o recordbatchwriter.o scalar.o schema.o symbols.o table.o threadpool.o -L../windows//lib-8.3.0/x64 -L../windows//lib/x64 -lparquet -larrow_dataset -larrow -lthrift -lsnappy -lz -lzstd -llz4 -lbcrypt -lpsapi -lcrypto -lcrypt32 -lws2_32 -LC:/PROGRA~1/R/R-devel/bin/x64 -lR
 lto1.exe: internal compiler error: in add_symbol_to_partition_1, at lto/lto-partition.c:153
 libbacktrace could not find executable to open
 Please submit a full bug report,
 with preprocessed source if appropriate.
 See <[https://github.com/r-windows]> for instructions.
 lto-wrapper.exe: fatal error: C:\rtools40\mingw64\bin\g++.exe returned 1 exit status
 compilation terminated.
 C:/rtools40/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/9.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: error: lto-wrapper failed

You can reproduce this in R on Windows for example like so:

dir.create("~/.R")
writeLines("CPPFLAGS=-flto", file = "~/.R/Makevars")
install.packages("arrow", type = 'source')

I am not sure if this is a bug in the toolchain, or in arrow. I tried with both gcc-8.3.0 and gcc-9.3.0, and the result is the same. I did find this issue in another project which suggests to enable INTERPROCEDURAL_OPTIMIZATION in cmake, when mixing lto code with non-lto code (which is the case when we only build the r bindings with lto, but not the c++ library).

Reporter: Jeroen / @jeroen

PRs and other links:

• GitHub Pull Request #10914
• GitHub Pull Request #8153

_{Note: This issue was originally created as ARROW-9616. Please see the migration documentation for further details.}

[asfimport]

Neal Richardson / @nealrichardson:
I started a branch to turn this on, and then to enable the cmake IPO as well: #8153

Even with that on in cmake, it fails: https://github.com/nealrichardson/arrow/runs/1088290997?check_suite_focus=true

C:/rtools40/mingw32/bin/g++ -shared -s -static-libgcc -o arrow.dll tmp.def array.o array_from_vector.o array_to_vector.o arraydata.o arrowExports.o buffer.o chunkedarray.o compression.o compute.o csv.o dataset.o datatype.o expression.o feather.o field.o filesystem.o imports.o io.o json.o memorypool.o message.o parquet.o py-to-r.o recordbatch.o recordbatchreader.o recordbatchwriter.o scalar.o schema.o symbols.o table.o threadpool.o -L../windows/arrow-1.0.1.9000/lib-8.3.0/i386 -L../windows/arrow-1.0.1.9000/lib/i386 -lparquet -larrow_dataset -larrow -larrow_bundled_dependencies -lthrift -lsnappy -lz -lzstd -llz4 -lbcrypt -lpsapi -lcrypto -lcrypt32 -laws-cpp-sdk-config -laws-cpp-sdk-transfer -laws-cpp-sdk-identity-management -laws-cpp-sdk-cognito-identity -laws-cpp-sdk-sts -laws-cpp-sdk-s3 -laws-cpp-sdk-core -laws-c-event-stream -laws-checksums -laws-c-common -lUserenv -lversion -lws2_32 -lBcrypt -lWininet -lwinhttp -LC:/R/bin/i386 -lR
lto1.exe: internal compiler error: in add_symbol_to_partition_1, at lto/lto-partition.c:155
libbacktrace could not find executable to open
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://github.com/r-windows> for instructions.
lto-wrapper.exe: fatal error: C:\rtools40\mingw32\bin\g++.exe returned 1 exit status
compilation terminated.
C:/rtools40/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.3.0/../../../../i686-w64-mingw32/bin/ld.exe: error: lto-wrapper failed
collect2.exe: error: ld returned 1 exit status
C:\rtools40\mingw64\bin\nm.exe: D:/a/arrow/arrow/check/arrow.Rcheck/00_pkg_src/arrow/src-i386/array.o: plugin needed to handle lto object
C:\rtools40\mingw64\bin\nm.exe: D:/a/arrow/arrow/check/arrow.Rcheck/00_pkg_src/arrow/src-i386/array_from_vector.o: plugin needed to handle lto object
...
C:\rtools40\mingw64\bin\nm.exe: D:/a/arrow/arrow/check/arrow.Rcheck/00_pkg_src/arrow/src-i386/threadpool.o: plugin needed to handle lto object
no DLL was created

Googling the "plugin needed to handle lto object" error message, it looks like there may be some necessary library missing from Rtools? https://stackoverflow.com/questions/32221221/mingw-x64-windows-plugin-needed-to-handle-lto-object/32461766#32461766

[asfimport]

Antoine Pitrou / @pitrou:
Can't we simply disable LTO? I doubt LTO would bring much to Arrow (and if it does, then I'd say it's a bug: we should structure our source code so that LTO is generally not useful).

[asfimport]

Neal Richardson / @nealrichardson:
If CRAN decides that it LTO's everything, then we wouldn't be able to turn that off. FWIW CRAN already has a LTO builder in its test setup (debian, I believe) and arrow is not failing that. So this is something in the Windows setup, and possibly not a problem in arrow at all.

[asfimport]

Antoine Pitrou / @pitrou:
An internal compiler error is certainly not a bug in Arrow, but we have to workaround the issue at some point, no?

[asfimport]

Antoine Pitrou / @pitrou:
Deciding to LTO everything sounds more ideological than pragmatic. LTO can be useful in some select cases, but I fail to understand why it would be mandatory. Also it will increase build times again.

[asfimport]

Uwe Korn / @xhochy:
LTO also has quite often the benefit of smaller binaries. Given that the Arrow libs are already large and have a lot of templated code that does similar things, it could be quite some benefit just for the binary size gains.

[asfimport]

Antoine Pitrou / @pitrou:
If the claimed benefit is binary size and/or performance, this should be evaluated on concrete terms.
(as well as the change in build times, btw :-))

[asfimport]

Uwe Korn / @xhochy:
Sure, will try that out in the next weeks on Linux and report back.

[asfimport]

Jeroen / @jeroen:
Testing this with the new GCC-10.3 UCRT toolchain, we get more specific linking errors when using LTO on Windows. Does this give a hint what might be wrong? I'm not seeing this for any other R package / C++ library.

 

g++ -std=gnu++11 -shared -s -static-libgcc -o arrow.dll tmp.def array.o array_to_vector.o arraydata.o arrowExports.o buffer.o chunkedarray.o compression.o compute.o csv.o dataset.o datatype.o expression.o feather.o field.o filesystem.o imports.o io.o json.o memorypool.o message.o parquet.o py-to-r.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o runtimeinfo.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L../windows/arrow-4.0.0/lib-10.2.0/x64 -L../windows/arrow-4.0.0/lib/x64-ucrt -lparquet -larrow_dataset -larrow -larrow_bundled_dependencies -lutf8proc -lthrift -lsnappy -lz -lzstd -llz4 -lbcrypt -lpsapi -lcrypto -lcrypt32 -lre2 -laws-cpp-sdk-config -laws-cpp-sdk-transfer -laws-cpp-sdk-identity-management -laws-cpp-sdk-cognito-identity -laws-cpp-sdk-sts -laws-cpp-sdk-s3 -laws-cpp-sdk-core -laws-c-event-stream -laws-checksums -laws-c-common -lUserenv -lversion -lws2_32 -lBcrypt -lWininet -lwinhttp -L/ucrt64/lib/x64 -L/ucrt64/lib -LC:/PROGRA~1/R/R-devel/bin/x64 -lR
`_ZTCN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE0_NS0_11InputStreamE' referenced in section `.rdata$_ZTTN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTTN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libparquet.a(schema.cc.obj): defined in discarded section `.gnu.linkonce.t._ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTCN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE0_NS0_11InputStreamE]' of io.o (symbol from plugin)
`_ZTCN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE0_NS0_11InputStreamE' referenced in section `.rdata$_ZTTN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTTN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libparquet.a(schema.cc.obj): defined in discarded section `.gnu.linkonce.t._ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTCN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE0_NS0_11InputStreamE]' of io.o (symbol from plugin)
`_ZTCN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE0_NS0_11InputStreamE' referenced in section `.rdata$_ZTTN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTTN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libparquet.a(schema.cc.obj): defined in discarded section `.gnu.linkonce.t._ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTCN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE0_NS0_11InputStreamE]' of io.o (symbol from plugin)
`_ZThn24_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4SeekEx' referenced in section `.rdata$_ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libparquet.a(schema.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4SeekEx[_ZThn24_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4SeekEx]' of io.o (symbol from plugin)
`_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv' referenced in section `.rdata$_ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libparquet.a(schema.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv[_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv]' of io.o (symbol from plugin)
`_ZTv0_n40_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv' referenced in section `.rdata$_ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libparquet.a(schema.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv[_ZTv0_n40_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv]' of io.o (symbol from plugin)
`_ZTv0_n48_NK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv' referenced in section `.rdata$_ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libparquet.a(schema.cc.obj): defined in discarded section `.gnu.linkonce.t._ZNK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv[_ZTv0_n48_NK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv]' of io.o (symbol from plugin)
`_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv' referenced in section `.text' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow_dataset.a(file_base.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv[_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv]' of io.o (symbol from plugin)
`_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv' referenced in section `.text' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow_dataset.a(file_base.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv[_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv]' of io.o (symbol from plugin)
`_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv' referenced in section `.rdata$_ZTCN5arrow2io12BufferReaderE0_NS0_8internal34RandomAccessFileConcurrencyWrapperIS1_EE[_ZTCN5arrow2io12BufferReaderE0_NS0_8internal34RandomAccessFileConcurrencyWrapperIS1_EE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow.a(memory.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv[_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv]' of io.o (symbol from plugin)
`_ZTv0_n40_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv' referenced in section `.rdata$_ZTCN5arrow2io12BufferReaderE0_NS0_8internal34RandomAccessFileConcurrencyWrapperIS1_EE[_ZTCN5arrow2io12BufferReaderE0_NS0_8internal34RandomAccessFileConcurrencyWrapperIS1_EE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow.a(memory.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv[_ZTv0_n40_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv]' of io.o (symbol from plugin)
`_ZTv0_n48_NK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv' referenced in section `.rdata$_ZTCN5arrow2io12BufferReaderE0_NS0_8internal34RandomAccessFileConcurrencyWrapperIS1_EE[_ZTCN5arrow2io12BufferReaderE0_NS0_8internal34RandomAccessFileConcurrencyWrapperIS1_EE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow.a(memory.cc.obj): defined in discarded section `.gnu.linkonce.t._ZNK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv[_ZTv0_n48_NK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv]' of io.o (symbol from plugin)
`_ZThn24_N5arrow2io12BufferReaderD1Ev' referenced in section `.rdata$_ZTVN5arrow2io12BufferReaderE[_ZTVN5arrow2io12BufferReaderE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow.a(memory.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io12BufferReaderD1Ev[_ZThn24_N5arrow2io12BufferReaderD1Ev]' of io.o (symbol from plugin)
`_ZThn24_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4SeekEx' referenced in section `.rdata$_ZTVN5arrow2io12BufferReaderE[_ZTVN5arrow2io12BufferReaderE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow.a(memory.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4SeekEx[_ZThn24_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4SeekEx]' of io.o (symbol from plugin)
`_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv' referenced in section `.rdata$_ZTVN5arrow2io12BufferReaderE[_ZTVN5arrow2io12BufferReaderE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow.a(memory.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv[_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv]' of io.o (symbol from plugin)
`_ZTv0_n40_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv' referenced in section `.rdata$_ZTVN5arrow2io12BufferReaderE[_ZTVN5arrow2io12BufferReaderE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow.a(memory.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv[_ZTv0_n40_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv]' of io.o (symbol from plugin)
`_ZTv0_n48_NK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv' referenced in section `.rdata$_ZTVN5arrow2io12BufferReaderE[_ZTVN5arrow2io12BufferReaderE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow.a(memory.cc.obj): defined in discarded section `.gnu.linkonce.t._ZNK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv[_ZTv0_n48_NK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv]' of io.o (symbol from plugin)
collect2.exe: error: ld returned 1 exit status
no DLL was created

 

[asfimport]

Antoine Pitrou / @pitrou:
@jeroen This seems to occur sometimes with LTO. arbor-sim/arbor#465 (comment) may give you a hint of how to fix/workaround it. Feel free to submit a PR if you manage to get it working.

(that said, if I were you, I wouldn't bother with LTO; Arrow C++ aims to be performant without relying on LTO)

[asfimport]

Jonathan Keane / @jonkeane:
https://cran.r-project.org/doc/manuals/r-release/R-admin.html#LTO-with-GCC notes something similar to what was suggested when addressing ARROW-13507: sounds like if LTO is enabled for the cmake build, our cmake setup needs to look for the lto plugin and possibly set a special AR/RANLIB. Though as of ARROW-13538, we have disabled LTO generally with UseLTO: false in the DESCRIPTION

[asfimport]

Jonathan Keane / @jonkeane:
We might need to use R CMD config AR and R CMD config RANLIB and set AR or CMAKE_CXX_COMPILER_AR so that both libarrow and the R C++ code use the same

[asfimport]

Neal Richardson / @nealrichardson:
Update: @jeroen got this link error compiling with gcc 10 and UCRT: https://gist.github.com/jeroen/fb83a191ebc90e7e867c5e1bb7465645

@bkietz suggested #10905 and #10914 as possible solutions, but with LTO enabled in our Rtools40 CI job (gcc 8?), it fails differently.

Note that this seems unrelated to the segfault on load on Linux under LTO, fixed in ARROW-13608 and referenced above here as well.