Opaque terminal messages during compilation using rtools40 and Eigen functions #321

James-Thorson-NOAA · 2020-08-08T16:01:19Z

Description:

I'm testing package VAST using R version 4.0.2, which uses rtools40 on Windows OS. Compiling any TMB CPP for package VAST using rtools40 (e.g., "VAST_v1_0_0.cpp" here) now results in a long series of compiler messages, where these messages did not occur using earlier R/rtools versions. Despite these messages, the compiler finishes successfully, and the compiled program passes all existing VAST integrated-tests (so it appears to be working correctly, except for this terminal output).

I have tried following Kasper's previous advice in eliminating needless compiler messages and therefore welcome any help eliminating these unnecessary and confusing messages.

Reproducible Steps:

To reproduce, please clone/download VAST, and compile "VAST_v1_0_0.cpp" here) using R 4.0.2 on a Windows machine (I assume it would work with other R 4.X.X. versions using rtools40).

Current Output:

A long series of compiler messages, e.g.

In file included from C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/Core:440,
                 from C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/Dense:1,
                 from C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/TMB/include/TMB.hpp:53,
                 from VAST_v12_0_0.cpp:2:
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/CoreEvaluators.h: In instantiation of 'struct Eigen::internal::evaluator<Eigen::Block<const Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, 1, -1, false> >':
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/CoreEvaluators.h:99:8:   required from 'struct Eigen::internal::evaluator<const Eigen::Block<const Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, 1, -1, false> >'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/CoreEvaluators.h:269:8:   required from 'struct Eigen::internal::unary_evaluator<Eigen::Transpose<const Eigen::Block<const Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, 1, -1, false> >, Eigen::internal::IndexBased, double>'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/CoreEvaluators.h:90:8:   required from 'struct Eigen::internal::evaluator<Eigen::Transpose<const Eigen::Block<const Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, 1, -1, false> > >'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/CoreEvaluators.h:99:8:   required from 'struct Eigen::internal::evaluator<const Eigen::Transpose<const Eigen::Block<const Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, 1, -1, false> > >'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/CoreEvaluators.h:681:51:   required from 'struct Eigen::internal::binary_evaluator<Eigen::CwiseBinaryOp<Eigen::internal::scalar_product_op<double, double>, const Eigen::Transpose<const Eigen::Block<const Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, 1, -1, false> >, const Eigen::Block<const Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, -1, 1, true> >, Eigen::internal::IndexBased, Eigen::internal::IndexBased, double, double>'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/CoreEvaluators.h:665:8:   [ skipping 18 instantiation contexts, use -ftemplate-backtrace-limit=0 to disable ]
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/AssignEvaluator.h:796:41:   required from 'void Eigen::internal::call_assignment(Dst&, const Src&, const Func&, typename Eigen::internal::enable_if<Eigen::internal::evaluator_assume_aliasing<Src>::value, void*>::type) [with Dst = Eigen::Map<Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >; Src = Eigen::Product<Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, 0>; Func = Eigen::internal::assign_op<double, double>; typename Eigen::internal::enable_if<Eigen::internal::evaluator_assume_aliasing<Src>::value, void*>::type = void*]'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/AssignEvaluator.h:782:18:   required from 'void Eigen::internal::call_assignment(Dst&, const Src&) [with Dst = Eigen::Map<Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >; Src = Eigen::Product<Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, 0>]'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/Assign.h:66:28:   required from 'Derived& Eigen::MatrixBase<Derived>::operator=(const Eigen::DenseBase<OtherDerived>&) [with OtherDerived = Eigen::Product<Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, 0>; Derived = Eigen::Map<Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >]'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/TMB/include/atomic_math.hpp:426:1:   required from 'void atomic::matmul(const CppAD::vector<Type>&, CppAD::vector<Type>&) [with Double = double]'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/TMB/include/atomic_math.hpp:426:1:   required from 'bool atomic::atomicmatmul<Type>::forward(size_t, size_t, const CppAD::vector<bool>&, CppAD::vector<bool>&, const CppAD::vector<Type>&, CppAD::vector<Type>&) [with Type = double; size_t = long long unsigned int]'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/TMB/include/atomic_math.hpp:426:1:   required from here
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/CoreEvaluators.h:960:8: warning: ignoring attributes on template argument 'Eigen::internal::packet_traits<double>::type' {aka '__vector(2) double'} [-Wignored-attributes]
   enum {
        ^
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/CoreEvaluators.h: In instantiation of 'struct Eigen::internal::evaluator<Eigen::Block<const Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, -1, 1, true> >':
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/CoreEvaluators.h:99:8:   required from 'struct Eigen::internal::evaluator<const Eigen::Block<const Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, -1, 1, true> >'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/CoreEvaluators.h:681:51:   required from 'struct Eigen::internal::binary_evaluator<Eigen::CwiseBinaryOp<Eigen::internal::scalar_product_op<double, double>, const Eigen::Transpose<const Eigen::Block<const Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, 1, -1, false> >, const Eigen::Block<const Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, -1, 1, true> >, Eigen::internal::IndexBased, Eigen::internal::IndexBased, double, double>'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/CoreEvaluators.h:665:8:   required from 'struct Eigen::internal::evaluator<Eigen::CwiseBinaryOp<Eigen::internal::scalar_product_op<double, double>, const Eigen::Transpose<const Eigen::Block<const Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, 1, -1, false> >, const Eigen::Block<const Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, -1, 1, true> > >'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/Redux.h:349:39:   required from 'class Eigen::internal::redux_evaluator<Eigen::CwiseBinaryOp<Eigen::internal::scalar_product_op<double, double>, const Eigen::Transpose<const Eigen::Block<const Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, 1, -1, false> >, const Eigen::Block<const Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, -1, 1, true> > >'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/Redux.h:416:17:   required from 'typename Eigen::internal::traits<T>::Scalar Eigen::DenseBase<Derived>::redux(const Func&) const [with BinaryOp = Eigen::internal::scalar_sum_op<double, double>; Derived = Eigen::CwiseBinaryOp<Eigen::internal::scalar_product_op<double, double>, const Eigen::Transpose<const Eigen::Block<const Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, 1, -1, false> >, const Eigen::Block<const Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, -1, 1, true> >; typename Eigen::internal::traits<T>::Scalar = double]'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/Redux.h:453:73:   [ skipping 15 instantiation contexts, use -ftemplate-backtrace-limit=0 to disable ]
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/AssignEvaluator.h:796:41:   required from 'void Eigen::internal::call_assignment(Dst&, const Src&, const Func&, typename Eigen::internal::enable_if<Eigen::internal::evaluator_assume_aliasing<Src>::value, void*>::type) [with Dst = Eigen::Map<Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >; Src = Eigen::Product<Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, 0>; Func = Eigen::internal::assign_op<double, double>; typename Eigen::internal::enable_if<Eigen::internal::evaluator_assume_aliasing<Src>::value, void*>::type = void*]'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/AssignEvaluator.h:782:18:   required from 'void Eigen::internal::call_assignment(Dst&, const Src&) [with Dst = Eigen::Map<Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >; Src = Eigen::Product<Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, 0>]'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/Assign.h:66:28:   required from 'Derived& Eigen::MatrixBase<Derived>::operator=(const Eigen::DenseBase<OtherDerived>&) [with OtherDerived = Eigen::Product<Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, Eigen::Map<const Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >, 0>; Derived = Eigen::Map<Eigen::Matrix<double, -1, -1>, 0, Eigen::Stride<0, 0> >]'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/TMB/include/atomic_math.hpp:426:1:   required from 'void atomic::matmul(const CppAD::vector<Type>&, CppAD::vector<Type>&) [with Double = double]'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/TMB/include/atomic_math.hpp:426:1:   required from 'bool atomic::atomicmatmul<Type>::forward(size_t, size_t, const CppAD::vector<bool>&, CppAD::vector<bool>&, const CppAD::vector<Type>&, CppAD::vector<Type>&) [with Type = double; size_t = long long unsigned int]'
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/TMB/include/atomic_math.hpp:426:1:   required from here
C:/Users/JAMES~1.THO/Work/R/R-40~1.2/library/RCPPEI~1/include/Eigen/src/Core/CoreEvaluators.h:960:8: warning: ignoring attributes on template argument 'Eigen::internal::packet_traits<double>::type' {aka '__vector(2) double'} [-Wignored-attributes]

but hundreds of lines longer.

Expected Output:

If applicable, the output you expected from TMB.

TMB Version:

1.7.18

R Version:

R version 4.0.2 (2020-06-22)

Operating System:

Windows 10.

kaskr · 2020-08-10T08:03:23Z

Majority of these warnings seems to be -Wignored-attributes.
Have you tried compiling with

compile("VAST_v1_0_0.cpp", CPPFLAGS="-Wno-ignored-attributes")

? (Add more -Wno-... if needed)

James-Thorson-NOAA · 2020-08-10T18:23:53Z

Thanks Kasper! That eliminated all of those opaque and extraneous messages, leaving only the terminal messages that I expected based on previous versions. I also confirmed that, if I introduced an intentional error, it still displayed that error appropriately in the terminal.

Case closed :)

Cole-Monnahan-NOAA · 2021-02-17T17:56:57Z

@kaskr I ran into the same wall of warnings so took your advice. However this causes pretty noticeable differences in the compilation, besides just suppressing output. The size of the .o and .dll files are different.

I have a simple spatial model where if I compile without the flag, both .o and .dll files are about 13MB. If I turn on the flag, the .o file is 34MB and it never creates the .dll -- at least after 15 or 20 minutes.

I was expecting this to simply suppress output but it's clearly doing more. Could you advise on what's going on here and whether there is another solution?

To reproduce I took this model

#include <TMB.hpp>
template<class Type>
Type objective_function<Type>::operator() ()
{
  DATA_VECTOR(y);
  PARAMETER(theta);      
  Type nll = -dnorm(y, theta, Type(1), true).sum();
  return nll;
}

and saved as both test1.cpp and test2.cpp (identical files). Then I compiled like

compile('test1.cpp')
compile('test2.cpp', flags='-Wno-ignored-attributes')

and the output files are

02/17/2021  09:47 AM                   208 test1.cpp
02/17/2021  09:49 AM        12,915,712 test1.dll
02/17/2021  09:49 AM        12,475,393 test1.o
02/17/2021  09:47 AM                   208 test2.cpp
02/17/2021  09:49 AM        14,000,640 test2.dll
02/17/2021  09:49 AM        18,362,462 test2.o

where the 3rd column gives disk size. Console output looks like this:

> "C:/rtools40/mingw64/bin/"g++ -std=gnu++11  -I"c:/PROGRA~1/R/R-40~1.3/include" -DNDEBUG -I"C:/Users/COLE~1.MON/RLibs/TMB/include" -I"C:/Users/COLE~1.MON/RLibs/RCPPEI~1/include"  -DTMB_SAFEBOUNDS -DLIB_UNLOAD=R_unload_test1  -DTMB_LIB_INIT=R_init_test1          -O2 -Wall  -mfpmath=sse -msse2 -mstackrealign -c test1.cpp -o test1.o
In file included from C:/Users/COLE~1.MON/RLibs/RCPPEI~1/include/Eigen/Core:397,
                 from C:/Users/COLE~1.MON/RLibs/RCPPEI~1/include/Eigen/Dense:1,
                 from C:/Users/COLE~1.MON/RLibs/TMB/include/TMB.hpp:53,
                 from test1.cpp:1:
C:/Users/COLE~1.MON/RLibs/RCPPEI~1/include/Eigen/src/Core/arch/SSE/PacketMath.h:60:39: warning: ignoring attributes on template argument '__m128' {aka '__vector(4) float'} [-Wignored-attributes]

[truncated]

C:/Users/COLE~1.MON/RLibs/TMB/include/expm.hpp:192:52:   required from here
C:/Users/COLE~1.MON/RLibs/RCPPEI~1/include/Eigen/src/Core/CoreEvaluators.h:960:8: warning: ignoring attributes on template argument 'Eigen::internal::packet_traits<double>::type' {aka '__vector(2) double'} [-Wignored-attributes]
   enum {
        ^
C:/rtools40/mingw64/bin/g++ -std=gnu++11 -shared -s -static-libgcc -o test1.dll tmp.def test1.o -Lc:/PROGRA~1/R/R-40~1.3/bin/x64 -lR
[1] 0
> "C:/rtools40/mingw64/bin/"g++ -std=gnu++11  -I"c:/PROGRA~1/R/R-40~1.3/include" -DNDEBUG -I"C:/Users/COLE~1.MON/RLibs/TMB/include" -I"C:/Users/COLE~1.MON/RLibs/RCPPEI~1/include"  -DTMB_SAFEBOUNDS -DLIB_UNLOAD=R_unload_test2  -DTMB_LIB_INIT=R_init_test2          -Wno-ignored-attributes -c test2.cpp -o test2.o
C:/rtools40/mingw64/bin/g++ -std=gnu++11 -shared -s -static-libgcc -o test2.dll tmp.def test2.o -Lc:/PROGRA~1/R/R-40~1.3/bin/x64 -lR
[1] 0

with

> sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19041)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252 
[2] LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] TMB_1.7.19

loaded via a namespace (and not attached):
[1] compiler_4.0.3  Matrix_1.3-2    tools_4.0.3     grid_4.0.3     
[5] lattice_0.20-41

bbolker · 2021-02-17T18:49:54Z

@Cole-Monnahan-NOAA that's crazy! Similar results with R-devel/Ubuntu 20.10. Can you pipe your output through grep -v ignore-attributes ?

32383640 Feb 17 13:39 test1.o
16258192 Feb 17 13:39 test1.so
 4817728 Feb 17 13:40 test2.o
 3351808 Feb 17 13:40 test2.so

Cole-Monnahan-NOAA · 2021-02-17T19:41:32Z

You mean like this?

>   [1] "C:/Users/COLE~1.MON/RLibs/RCPPEI~1/include/Eigen/src/Core/arch/SSE/PacketMath.h:60:39: warning: ignoring attributes on template argument '__m128' {aka '__vector(4) float'} [-Wignored-attributes]"                                                                     
  [2] "C:/Users/COLE~1.MON/RLibs/RCPPEI~1/include/Eigen/src/Core/arch/SSE/PacketMath.h:61:40: warning: ignoring attributes on template argument '__m128i' {aka '__vector(2) long long int'} [-Wignored-attributes]"                                                            
  [3] "C:/Users/COLE~1.MON/RLibs/RCPPEI~1/include/Eigen/src/Core/arch/SSE/PacketMath.h:62:40: warning: ignoring attributes on template argument '__m128d' {aka '__vector(2) double'} [-Wignored-attributes]"                                                                   
  [4] "C:/Users/COLE~1.MON/RLibs/RCPPEI~1/include/Eigen/src/Core/arch/SSE/PacketMath.h:161:43: warning: ignoring attributes on template argument 'Eigen::internal::Packet4f' {aka '__vector(4) float'} [-Wignored-attributes]"                                                 
  [5] "C:/Users/COLE~1.MON/RLibs/RCPPEI~1/include/Eigen/src/Core/arch/SSE/PacketMath.h:162:43: warning: ignoring attributes on template argument 'Eigen::internal::Packet2d' {aka '__vector(2) double'} [-Wignored-attributes]"                                                
  [6] "C:/Users/COLE~1.MON/RLibs/RCPPEI~1/include/Eigen/src/Core/arch/SSE/PacketMath.h:163:43: warning: ignoring attributes on template argument 'Eigen::internal::Packet4i' {aka '__vector(2) long long int'} [-Wignored-attributes]"                                         
  [7] "C:/Users/COLE~1.MON/RLibs/RCPPEI~1/include/Eigen/src/Core/arch/SSE/PacketMath.h:718:35: warning: ignoring attributes on template argument 'Eigen::internal::Packet4f' {aka '__vector(4) float'} [-Wignored-attributes]"                                                 

[truncated]

[141] "C:/Users/COLE~1.MON/RLibs/RCPPEI~1/include/Eigen/src/Core/CoreEvaluators.h:960:8: warning: ignoring attributes on template argument 'Eigen::internal::packet_traits<double>::type' {aka '__vector(2) double'} [-Wignored-attributes]"                                   
[142] "C:/Users/COLE~1.MON/RLibs/RCPPEI~1/include/Eigen/src/Core/DenseCoeffsBase.h:55:30: warning: ignoring attributes on template argument 'Eigen::internal::packet_traits<double>::type' {aka '__vector(2) double'} [-Wignored-attributes]"                                  
[143] "C:/Users/COLE~1.MON/RLibs/RCPPEI~1/include/Eigen/src/Core/CoreEvaluators.h:960:8: warning: ignoring attributes on template argument 'Eigen::internal::packet_traits<double>::type' {aka '__vector(2) double'} [-Wignored-attributes]"

bbolker · 2021-02-17T20:24:48Z

I was thinking about running compilation from the shell, where | grep -v "ignoring" would be available. This is harder to do within R (so far I haven't succeeded):

p <- pipe("egrep -v \"ignor|required\"")
open(p)
sink(p,type="message")
unlink("test1.so")
unlink("test1.o")
TMB::compile("test1.cpp")
sink(type="message")
close(p)

but I haven't got the incantation right to capture the output stream from system(). It might be necessary to hack TMB:::.shlib_internal() (which is where the system() call actually happens) to get this to work.

Cole-Monnahan-NOAA · 2021-02-17T20:57:56Z

@bbolker I also failed at pipe ninja'ing so just copied the R console output into a text file then readLines and grepped from there, resulting in what I pasted above. I just tested and there are no other "ignored" except those I pasted. I pasted the whole file below if that's helpful.

Very curious to see what Kasper says, or someone more familiar with C++ compilers.

output.txt
.

John-R-Wallace-NOAA · 2021-02-17T22:43:00Z

See my 17 Feb 2021 comments on Issue #277.

Cole-Monnahan-NOAA · 2021-02-18T00:30:23Z

@John-R-Wallace-NOAA comment is that when invoking that flag it changes other flags. I confirmed that if I run

compile('test3.cpp', flags="-Wno-ignored-attributes -O2 -mfpmath=sse -msse2 -mstackrealign")

it does indeed suppress the warnings but provides the same output files as without it. So this is a good solution for one-off cases like mine. Thanks and sharp eyes @John-R-Wallace-NOAA

@bbolker does that work for you as well?

Cole-Monnahan-NOAA · 2022-01-26T21:40:22Z

@kaskr Would you consider making this flag the default in TMB?

I recently taught some folks TMB and there were so many of these warnings in RStudio that the actual error messages were bumped off console completely and they couldn't see them. It worked to add this flag but seems unnecessary and is pretty opaque.

bbolker · 2022-01-26T23:19:59Z

Unfortunately this can't be made the default for the CRAN version of the package - CRAN simply won't accept these flags. One possibility, if @kaskr felt it was worth the trouble, would be to incorporate it in the GitHub version of the package but strip it out every time the package was uploaded to CRAN. That way you could tell users to remotes::install_github("kaskr/adcomp/TMB") instead of installing from CRAN.

Or TMB could include a helper function to add the appropriate flags to a user's .R/Makevars file, e.g. something like:

silence_warnings <- function() {
     p <- readline("Please confirm that it's OK to create and/or modify your ~/R/.Makevars file to suppress TMB compilation warnings (Y/n): ")
     if (!toupper(p) == "Y") return(invisible(NULL))
     if (!file.exists("~/.R/Makevars")) {
           message("creating ~/.R/Makevars")
           file.create("~/.R/Makevars")
     }
     cat([STUFF], "~/.R/Makevars", append = TRUE)
}

kaskr · 2022-01-27T11:08:29Z

The silence_warnings() is one way to solve the problem.

Alternatively, we could let TMB::compile add the flag by default. This won't violate CRAN polices, because TMB::compile is not used when building packages on CRAN.

James-Thorson-NOAA · 2022-01-27T14:33:19Z

personally, I think editing TMB::compile to use those flags by default makes sense to me. But of course I defer to any strong opinion

bbolker · 2022-01-27T14:54:30Z

No strong feelings either way, as long as modifying TMB::compile() won't lead to subtle/unforeseen CRAN problems for downstream packages that use TMB.

Cole-Monnahan-NOAA · 2022-01-27T16:21:20Z

If these are warnings users should never see then I I also vote for making it the default in TMB::compile. With maybe a note in the help file of why.

John-R-Wallace-NOAA · 2022-02-09T00:01:41Z

After finally fully understanding the cause of these excess warnings, I have eliminated them with a single line of code.
See: https://github.com/John-R-Wallace-NOAA/adcomp

kaskr · 2022-02-09T10:44:11Z

@John-R-Wallace-NOAA Seems you came across an important fact left out of this thread: The github version of TMB does not have the issue with warnings from Eigen - only the CRAN version has the problem.

Your idea about setting the flag depending on useRcppEigen is useful.
Also, the idea of adding to ppflags is important because this adds to PKG_CPPFLAGS which is appended to CPPFLAGS. This solves @Cole-Monnahan-NOAA's issue that CPPFLAGS by default contains important stuff that should not be overwritten.

Unfortunately, having solved the problem on Windows does not ensure general portability.
The flags to set are platform and compiler dependent. e.g. on ubuntu I need to set -Wno-deprecated-declarations -Wno-ignored-attributes -Wno-enum-compare -Wno-int-in-bool-context. These are for gcc and might not work for other compilers (clang, intel icc, oracle cc). Passing unsupported flags to a compiler usually results in a new warning - or sometimes even worse - in an error.
My point is that an attempt to solve the problem on some specific platforms can actually break things on others. This is the main reason why this issue hasn't already been fixed.

Cole-Monnahan-NOAA · 2022-02-09T16:10:16Z

@kaskr so do you suggest we adopt a silence_warnings approach like above instead? Or maybe something that could be done globally once on install?

I can see the headache of platform specific flags. But presumably most of the TMB user base installs from CRAN and uses Rtools4 on Windows. Being able to clearly see the error messages is a pretty important function. So I think trying to find a solution is still a good idea.

John-R-Wallace-NOAA · 2022-02-09T22:24:51Z

@kaskr Is the -Wno-ignored-attributes flag added for Ubuntu because it is broken otherwise? (I only have Cent-OS (ver 7) available to try, which I should have mentioned in my TMB fork.) What about adding -Wno-ignored-attributes in the same way for Windows/gcc combination, even though it is not strictly broken otherwise? The -Wno-ignored-attributes would then be added for both the GitHub and CRAN installations under Windows/gcc, even though it is only needed for the more common CRAN installation.
The help for compile() could even use -Wno-ignored-attributes as an example, stating that if a flag is used by default for a given OS and compiler, then its reverse (-Wignored-attributes) might be used in the ‘flags’ argument to see what is missing. But that for some OS’s, a crash may result. (If that is what happens for Ubuntu.)

kaskr · 2022-02-11T09:04:07Z

So I think trying to find a solution is still a good idea.

I agree

- Adds option to conditionally include DisableStupidWarnings.h - Copied recent version from Eigen main branch: Eigen/src/Core/util/DisableStupidWarnings.h - File renamed to by-pass a CRAN note - File excluded from TMB doxygen - cran-version modified to build 'simple' example without illegal flag

kaskr · 2022-02-11T13:27:23Z

Hopefully portable solution now on master branch. In case you feel like testing:

install TMB from master
Remove the file system.file("include/Eigen", package="TMB")
Compile a model and check you can enable/disable warnings using new flag ?compile

Cole-Monnahan-NOAA · 2022-02-11T20:09:40Z

Latest master branch pulled down and installed, deleted Eigen folder. Compiled simple.cpp with flag TRUE and FALSE. FALSE leads to the giant array of warnings as mentioned previously. TRUE results is below. Reduces maybe 95% of warnings but not all.

If I repeat this but do not delete the Eigen folder then TRUE works fine.

Output without Eigen folder:

> compile("simple.cpp", eigen.disable.warnings = TRUE)
"C:/rtools40/mingw64/bin/"g++ -std=gnu++11  -I"C:/PROGRA~1/R/R-41~1.2/include" -DNDEBUG -I"C:/Users/COLE~1.MON/Rlibs/TMB/include" -I"C:/Users/COLE~1.MON/Rlibs/RCPPEI~1/include"  -DTMB_SAFEBOUNDS -DTMB_EIGEN_DISABLE_WARNINGS -DLIB_UNLOAD=R_unload_simple  -DTMB_LIB_INIT=R_init_simple  -DCPPAD_FRAMEWORK         -O2 -Wall  -mfpmath=sse -msse2 -mstackrealign  -c simple.cpp -o simple.o
In file included from C:/Users/COLE~1.MON/Rlibs/RCPPEI~1/include/Eigen/Core:371,
                 from C:/Users/COLE~1.MON/Rlibs/RCPPEI~1/include/Eigen/Dense:1,
                 from C:/Users/COLE~1.MON/Rlibs/TMB/include/TMB.hpp:64,
                 from simple.cpp:2:
C:/Users/COLE~1.MON/Rlibs/RCPPEI~1/include/Eigen/src/Core/util/Memory.h: In member function 'void report_stack<Type>::push(Type, const char*) [with Type = CppAD::AD<double>]':
C:/Users/COLE~1.MON/Rlibs/RCPPEI~1/include/Eigen/src/Core/util/Memory.h:259:17: warning: iteration 1152921504606846976 invokes undefined behavior [-Waggressive-loop-optimizations]
     while(size) ptr[--size].~T();
                 ^~~
C:/Users/COLE~1.MON/Rlibs/RCPPEI~1/include/Eigen/src/Core/util/Memory.h:259:5: note: within this loop
     while(size) ptr[--size].~T();
     ^~~~~
C:/Users/COLE~1.MON/Rlibs/RCPPEI~1/include/Eigen/src/Core/util/Memory.h: In member function 'void report_stack<Type>::push(Type, const char*) [with Type = double]':
C:/Users/COLE~1.MON/Rlibs/RCPPEI~1/include/Eigen/src/Core/util/Memory.h:259:17: warning: iteration 1152921504606846976 invokes undefined behavior [-Waggressive-loop-optimizations]
     while(size) ptr[--size].~T();
                 ^~~
C:/Users/COLE~1.MON/Rlibs/RCPPEI~1/include/Eigen/src/Core/util/Memory.h:259:5: note: within this loop
     while(size) ptr[--size].~T();
     ^~~~~
C:/Users/COLE~1.MON/Rlibs/RCPPEI~1/include/Eigen/src/Core/util/Memory.h: In member function 'void report_stack<Type>::push(Type, const char*) [with Type = CppAD::AD<CppAD::AD<double> >]':
C:/Users/COLE~1.MON/Rlibs/RCPPEI~1/include/Eigen/src/Core/util/Memory.h:259:17: warning: iteration 1152921504606846976 invokes undefined behavior [-Waggressive-loop-optimizations]
     while(size) ptr[--size].~T();
                 ^~~
C:/Users/COLE~1.MON/Rlibs/RCPPEI~1/include/Eigen/src/Core/util/Memory.h:259:5: note: within this loop
     while(size) ptr[--size].~T();
     ^~~~~
C:/Users/COLE~1.MON/Rlibs/RCPPEI~1/include/Eigen/src/Core/util/Memory.h: In member function 'void report_stack<Type>::push(Type, const char*) [with Type = CppAD::AD<CppAD::AD<CppAD::AD<double> > >]':
C:/Users/COLE~1.MON/Rlibs/RCPPEI~1/include/Eigen/src/Core/util/Memory.h:259:17: warning: iteration 1152921504606846976 invokes undefined behavior [-Waggressive-loop-optimizations]
     while(size) ptr[--size].~T();
                 ^~~
C:/Users/COLE~1.MON/Rlibs/RCPPEI~1/include/Eigen/src/Core/util/Memory.h:259:5: note: within this loop
     while(size) ptr[--size].~T();
     ^~~~~
C:/rtools40/mingw64/bin/g++ -std=gnu++11 -shared -s -static-libgcc -o simple.dll tmp.def simple.o -LC:/PROGRA~1/R/R-41~1.2/bin/x64 -lR

Output with Eigen folder:

> compile("simple.cpp", eigen.disable.warnings = TRUE)
"C:/rtools40/mingw64/bin/"g++ -std=gnu++11  -I"C:/PROGRA~1/R/R-41~1.2/include" -DNDEBUG -I"C:/Users/COLE~1.MON/Rlibs/TMB/include"   -DTMB_SAFEBOUNDS -DTMB_EIGEN_DISABLE_WARNINGS -DLIB_UNLOAD=R_unload_simple  -DTMB_LIB_INIT=R_init_simple  -DCPPAD_FRAMEWORK         -O2 -Wall  -mfpmath=sse -msse2 -mstackrealign  -c simple.cpp -o simple.o
C:/rtools40/mingw64/bin/g++ -std=gnu++11 -shared -s -static-libgcc -o simple.dll tmp.def simple.o -LC:/PROGRA~1/R/R-41~1.2/bin/x64 -lR
[1] 0

John-R-Wallace-NOAA · 2022-02-12T02:05:44Z

@Cole-Monnahan-NOAA I have only seen the '-Waggressive-loop-optimizations' warning when compiling 'simple.cpp', presumably because it is too simple. Use '-Wno-aggressive-loop-optimizations' in flags to remove:

compile('simple.cpp', flags = "-Wno-aggressive-loop-optimizations", eigen.disable.warnings = TRUE)
"C:/rtools40/mingw64/bin/"g++ -std=gnu++11  -I"W:/R/R-4.1.2/include" -DNDEBUG -I"W:/R/R-4.1.2/library/TMB/include" -I"W:/R/R-4.1.2/library/RcppEigen/include"  -DTMB_SAFEBOUNDS -DTMB_EIGEN_DISABLE_WARNINGS -DLIB_UNLOAD=R_unload_simple  -DTMB_LIB_INIT=R_init_simple  -DCPPAD_FRAMEWORK         -Wno-aggressive-loop-optimizations -c simple.cpp -o simple.o
C:/rtools40/mingw64/bin/g++ -std=gnu++11 -shared -s -static-libgcc -o simple.dll tmp.def simple.o -LW:/R/R-4.1.2/bin/x64 -lR
[1] 0

Otherwise the new 'eigen.disable.warnings' argument is working fine (here with 'linreg_parallel.cpp'):

TMB::compile('linreg_parallel.cpp', eigen.disable.warnings = TRUE)  # Renamed Eigen folder
"C:/rtools40/mingw64/bin/"g++ -std=gnu++11  -I"W:/R/R-4.1.2/include" -DNDEBUG -I"W:/R/R-4.1.2/library/TMB/include" -I"W:/R/R-4.1.2/library/RcppEigen/include"  -DTMB_SAFEBOUNDS -DTMB_EIGEN_DISABLE_WARNINGS -DLIB_UNLOAD=R_unload_linreg_parallel  -DTMB_LIB_INIT=R_init_linreg_parallel  -DCPPAD_FRAMEWORK      -fopenmp   -O2 -Wall  -mfpmath=sse -msse2 -mstackrealign  -c linreg_parallel.cpp -o linreg_parallel.o
C:/rtools40/mingw64/bin/g++ -std=gnu++11 -shared -s -static-libgcc -o linreg_parallel.dll tmp.def linreg_parallel.o -fopenmp -LW:/R/R-4.1.2/bin/x64 -lR
[1] 0

Thanks for the new flag @kaskr. Are there differences between using the Eigen library versus the RcppEigen package?

Since the compile()'s help says that this new flag is not allowed for code to be compiled on CRAN, I will repeat below the information that is on my TMB fork related to this issue.

Using the '-w' flag (note the lower case) in the flags argument of compile() will remove all warnings, e.g.:

compile('simple.cpp', '-w')

compile('simple.cpp', "-w -O2 -mfpmath=sse -msse2")  # R's global Makeconf CXXFLAGS added back in ('-Wall' removed).

Of course, such a global removal of warnings needs to be used with caution.

kaskr · 2022-02-12T10:51:41Z

@John-R-Wallace-NOAA and @Cole-Monnahan-NOAA thanks for testing - I can actually reproduce. I'll have to leave this for now but will get back to it.

Are there differences between using the Eigen library versus the RcppEigen package?

Should only be a matter of the Eigen version. The version shipping with TMB github is eigen-3.4. The current version with RcppEigen is eigen-3.3 (I think). That might explain the unexpected extra warnings with the simple example, because one could hope that these warnings have been fixed in 3.4 in a cleaner way than by disabling.

- Eigen includes 'ReenableStupidWarnings.h' after each include - Workaround this by setting EIGEN_PERMANENTLY_DISABLE_... - See here #321

- std::vector push_back() simpler and asymp faster than conservativeResize() - Should solve the 'iteration ... undefined behaviour' warnings which seem to be triggered by conservativeResize() - See here #344 and here #321

kaskr · 2022-03-04T15:42:29Z

@John-R-Wallace-NOAA @Cole-Monnahan-NOAA It seems to be working now. I'll probably submit to CRAN sometime next week.

Cole-Monnahan-NOAA · 2022-03-04T16:17:56Z

@kaskr Looking forward to seeing that. Let me know if I can help test again.

kaskr · 2022-03-04T17:01:04Z

@Cole-Monnahan-NOAA Would be great if you could run the same tests again and hopefully confirm that it works on Windows!

John-R-Wallace-NOAA · 2022-03-04T19:47:09Z

I will also test again.

Cole-Monnahan-NOAA · 2022-03-04T21:28:39Z

Confirmed this works on Win10 using latest master branch pull. Default to compile the simple example produced only the expected output.

This is great thanks @kaskr

John-R-Wallace-NOAA · 2022-03-05T02:37:54Z

I renamed the '...\include\Eigen' folder and saw the RcppEigen package is being used with no excess warnings.

@kaskr, I see the added flag '-DTMB_EIGEN_DISABLE_WARNINGS' has been added which I assume works with 'DisableStupidWarnings.h', but how are you eliminating the aggressive loop optimizations warnings in the simple example?

kaskr · 2022-03-07T18:44:59Z

@John-R-Wallace-NOAA and @Cole-Monnahan-NOAA thanks for testing. This is now on CRAN.

but how are you eliminating the aggressive loop optimizations warnings in the simple example?

was a bit more tricky - see commit f08ff4f

James-Thorson-NOAA closed this as completed Aug 10, 2020

James-Thorson-NOAA mentioned this issue Aug 10, 2020

Error in TMB:compile Compilation failed 64bit Windows10 R4.0 Rtool40 James-Thorson-NOAA/VAST#242

Closed

kaskr reopened this Feb 11, 2022

kaskr added a commit that referenced this issue Feb 11, 2022

compile: Add option 'eigen.disable.warnings' #321

7a59476

Lewis-Barnett-NOAA mentioned this issue Feb 16, 2022

TMB fails to compile on MRAN afsc-gap-products/model-based-indices#24

Closed

kaskr added a commit that referenced this issue Mar 4, 2022

Fix b48a098

f46094d

- Eigen includes 'ReenableStupidWarnings.h' after each include - Workaround this by setting EIGEN_PERMANENTLY_DISABLE_... - See here #321

kaskr closed this as completed Mar 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Opaque terminal messages during compilation using rtools40 and Eigen functions #321

Opaque terminal messages during compilation using rtools40 and Eigen functions #321

James-Thorson-NOAA commented Aug 8, 2020

kaskr commented Aug 10, 2020

James-Thorson-NOAA commented Aug 10, 2020

Cole-Monnahan-NOAA commented Feb 17, 2021

bbolker commented Feb 17, 2021

Cole-Monnahan-NOAA commented Feb 17, 2021

bbolker commented Feb 17, 2021 •

edited

Cole-Monnahan-NOAA commented Feb 17, 2021

John-R-Wallace-NOAA commented Feb 17, 2021

Cole-Monnahan-NOAA commented Feb 18, 2021

Cole-Monnahan-NOAA commented Jan 26, 2022

bbolker commented Jan 26, 2022

kaskr commented Jan 27, 2022

James-Thorson-NOAA commented Jan 27, 2022

bbolker commented Jan 27, 2022

Cole-Monnahan-NOAA commented Jan 27, 2022

John-R-Wallace-NOAA commented Feb 9, 2022

kaskr commented Feb 9, 2022

Cole-Monnahan-NOAA commented Feb 9, 2022

John-R-Wallace-NOAA commented Feb 9, 2022

kaskr commented Feb 11, 2022

kaskr commented Feb 11, 2022

Cole-Monnahan-NOAA commented Feb 11, 2022

John-R-Wallace-NOAA commented Feb 12, 2022

kaskr commented Feb 12, 2022

kaskr commented Mar 4, 2022

Cole-Monnahan-NOAA commented Mar 4, 2022

kaskr commented Mar 4, 2022

John-R-Wallace-NOAA commented Mar 4, 2022

Cole-Monnahan-NOAA commented Mar 4, 2022

John-R-Wallace-NOAA commented Mar 5, 2022

kaskr commented Mar 7, 2022

Opaque terminal messages during compilation using rtools40 and Eigen functions #321

Opaque terminal messages during compilation using rtools40 and Eigen functions #321

Comments

James-Thorson-NOAA commented Aug 8, 2020

Description:

Reproducible Steps:

Current Output:

Expected Output:

TMB Version:

R Version:

Operating System:

kaskr commented Aug 10, 2020

James-Thorson-NOAA commented Aug 10, 2020

Cole-Monnahan-NOAA commented Feb 17, 2021

bbolker commented Feb 17, 2021

Cole-Monnahan-NOAA commented Feb 17, 2021

bbolker commented Feb 17, 2021 • edited

Cole-Monnahan-NOAA commented Feb 17, 2021

John-R-Wallace-NOAA commented Feb 17, 2021

Cole-Monnahan-NOAA commented Feb 18, 2021

Cole-Monnahan-NOAA commented Jan 26, 2022

bbolker commented Jan 26, 2022

kaskr commented Jan 27, 2022

James-Thorson-NOAA commented Jan 27, 2022

bbolker commented Jan 27, 2022

Cole-Monnahan-NOAA commented Jan 27, 2022

John-R-Wallace-NOAA commented Feb 9, 2022

kaskr commented Feb 9, 2022

Cole-Monnahan-NOAA commented Feb 9, 2022

John-R-Wallace-NOAA commented Feb 9, 2022

kaskr commented Feb 11, 2022

kaskr commented Feb 11, 2022

Cole-Monnahan-NOAA commented Feb 11, 2022

John-R-Wallace-NOAA commented Feb 12, 2022

kaskr commented Feb 12, 2022

kaskr commented Mar 4, 2022

Cole-Monnahan-NOAA commented Mar 4, 2022

kaskr commented Mar 4, 2022

John-R-Wallace-NOAA commented Mar 4, 2022

Cole-Monnahan-NOAA commented Mar 4, 2022

John-R-Wallace-NOAA commented Mar 5, 2022

kaskr commented Mar 7, 2022

bbolker commented Feb 17, 2021 •

edited