Travis clang tests timing out due to infinite compile loop

[rok-cesnovar]

Summary:

Travis tests are failing since 2.20 for clang. It seems that clang gets stuck in an infinite loop of compiling the same files over and over again.

Looking at the the build history https://travis-ci.org/stan-dev/cmdstan/builds the first PR that did not pass the clang tests on Travis is https://github.com/stan-dev/cmdstan/pull/723. I am not pointing to that PR as the culprit, I can't see why that would interfere with compiling on Travis.

g++ tests run fine, but the clang tests timeout after 50 minutes. On 2.20 and before that they ran for 10-20 minutes.

Description:

It seems that test on Travis try to compiler the same files over and over again until the job is terminated for running too long.

See the following example (this is just a part of the whole output). You can repeats of the exact same compile commands:

clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/ast_def.o -MT stan/src/stan/lang/ast_def.d -MM -E -MG -MP -MF stan/src/stan/lang/ast_def.d stan/src/stan/lang/ast_def.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/semantic_actions_def.o -MT stan/src/stan/lang/grammars/semantic_actions_def.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/semantic_actions_def.d stan/src/stan/lang/grammars/semantic_actions_def.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/block_var_decls_grammar_inst.o -MT stan/src/stan/lang/grammars/block_var_decls_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/block_var_decls_grammar_inst.d stan/src/stan/lang/grammars/block_var_decls_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/statement_grammar_inst.o -MT stan/src/stan/lang/grammars/statement_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/statement_grammar_inst.d stan/src/stan/lang/grammars/statement_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/local_var_decls_grammar_inst.o -MT stan/src/stan/lang/grammars/local_var_decls_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/local_var_decls_grammar_inst.d stan/src/stan/lang/grammars/local_var_decls_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/functions_grammar_inst.o -MT stan/src/stan/lang/grammars/functions_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/functions_grammar_inst.d stan/src/stan/lang/grammars/functions_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/statement_2_grammar_inst.o -MT stan/src/stan/lang/grammars/statement_2_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/statement_2_grammar_inst.d stan/src/stan/lang/grammars/statement_2_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/indexes_grammar_inst.o -MT stan/src/stan/lang/grammars/indexes_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/indexes_grammar_inst.d stan/src/stan/lang/grammars/indexes_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/term_grammar_inst.o -MT stan/src/stan/lang/grammars/term_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/term_grammar_inst.d stan/src/stan/lang/grammars/term_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/expression07_grammar_inst.o -MT stan/src/stan/lang/grammars/expression07_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/expression07_grammar_inst.d stan/src/stan/lang/grammars/expression07_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/program_grammar_inst.o -MT stan/src/stan/lang/grammars/program_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/program_grammar_inst.d stan/src/stan/lang/grammars/program_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/whitespace_grammar_inst.o -MT stan/src/stan/lang/grammars/whitespace_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/whitespace_grammar_inst.d stan/src/stan/lang/grammars/whitespace_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/bare_type_grammar_inst.o -MT stan/src/stan/lang/grammars/bare_type_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/bare_type_grammar_inst.d stan/src/stan/lang/grammars/bare_type_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/expression_grammar_inst.o -MT stan/src/stan/lang/grammars/expression_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/expression_grammar_inst.d stan/src/stan/lang/grammars/expression_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/ast_def.o -MT stan/src/stan/lang/ast_def.d -MM -E -MG -MP -MF stan/src/stan/lang/ast_def.d stan/src/stan/lang/ast_def.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/semantic_actions_def.o -MT stan/src/stan/lang/grammars/semantic_actions_def.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/semantic_actions_def.d stan/src/stan/lang/grammars/semantic_actions_def.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/block_var_decls_grammar_inst.o -MT stan/src/stan/lang/grammars/block_var_decls_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/block_var_decls_grammar_inst.d stan/src/stan/lang/grammars/block_var_decls_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/statement_grammar_inst.o -MT stan/src/stan/lang/grammars/statement_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/statement_grammar_inst.d stan/src/stan/lang/grammars/statement_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/local_var_decls_grammar_inst.o -MT stan/src/stan/lang/grammars/local_var_decls_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/local_var_decls_grammar_inst.d stan/src/stan/lang/grammars/local_var_decls_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/functions_grammar_inst.o -MT stan/src/stan/lang/grammars/functions_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/functions_grammar_inst.d stan/src/stan/lang/grammars/functions_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/statement_2_grammar_inst.o -MT stan/src/stan/lang/grammars/statement_2_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/statement_2_grammar_inst.d stan/src/stan/lang/grammars/statement_2_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/indexes_grammar_inst.o -MT stan/src/stan/lang/grammars/indexes_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/indexes_grammar_inst.d stan/src/stan/lang/grammars/indexes_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/term_grammar_inst.o -MT stan/src/stan/lang/grammars/term_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/term_grammar_inst.d stan/src/stan/lang/grammars/term_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/expression07_grammar_inst.o -MT stan/src/stan/lang/grammars/expression07_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/expression07_grammar_inst.d stan/src/stan/lang/grammars/expression07_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/program_grammar_inst.o -MT stan/src/stan/lang/grammars/program_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/program_grammar_inst.d stan/src/stan/lang/grammars/program_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/whitespace_grammar_inst.o -MT stan/src/stan/lang/grammars/whitespace_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/whitespace_grammar_inst.d stan/src/stan/lang/grammars/whitespace_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/bare_type_grammar_inst.o -MT stan/src/stan/lang/grammars/bare_type_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/bare_type_grammar_inst.d stan/src/stan/lang/grammars/bare_type_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/grammars/expression_grammar_inst.o -MT stan/src/stan/lang/grammars/expression_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/expression_grammar_inst.d stan/src/stan/lang/grammars/expression_grammar_inst.cpp
clang++-3.8 -stdlib=libc++ -stdlib=libc++ -std=c++1y  -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS     -c -MT bin/cmdstan/lang/ast_def.o -MT stan/src/stan/lang/ast_def.d -MM -E -MG -MP -MF stan/src/stan/lang/ast_def.d stan/src/stan/lang/ast_def.cpp

Reproducible Steps:

See Travis.

Current Output:

Infinite loop of compiling the same files over and over.

Expected Output:

Travis tests should pass.

Additional Information:

/

Current Version:

v2.20.0

[seantalts]

@serban-nicusor-toptal whatever happened to the attempt to get CmdStan running the one-compiler-per-OS set of tests on Jenkins? Did that go in or is that tracked somewhere? Once that's in we can delete the Travis file entirely.

[serban-nicusor-toptal]

Hey @seantalts, are we talking about this one ? #706

[seantalts]

Yes, thank you! So I think we can delete the Travis tests now as all relevant OS compiler setups should be tested under Jenkins :)

…

On Tue, Sep 17, 2019 at 16:05 Nicusor Serban ***@***.***> wrote: Hey @seantalts <https://github.com/seantalts>, are we talking about this one ? #706 <#706> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#728?email_source=notifications&email_token=AAGET3EV4XMNEXFSBOBCL2DQKE2A3A5CNFSM4ISWD3V2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD65XUYI#issuecomment-532380257>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGET3FQRPWHKNVYJFRY2XDQKE2A3ANCNFSM4ISWD3VQ> .

[rok-cesnovar]

Closed via #731