New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ehlers Damage test failing sporadically #1881

Closed
chleh opened this Issue Jul 24, 2017 · 9 comments

Comments

Projects
None yet
3 participants
@chleh
Contributor

chleh commented Jul 24, 2017

Symptom: ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension sometimes falies (roughly 50% of the time)

[8] lehmannc@envinf141l:build-release-eigenlis$ OMP_NUM_THREADS=1 ctest -R ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension
Test project /home/lehmannc/prog/ogs6/github-chleh-PRs/build-release-eigenlis
    Start 210: ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension
1/2 Test #210: ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension ...........***Failed    9.26 sec
    Start 211: ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension-vtkdiff
2/2 Test #211: ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension-vtkdiff ...***Failed    0.05 sec

0% tests passed, 2 tests failed out of 2

Total Test time (real) =   9.33 sec

The following tests FAILED:
	210 - ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension (Failed)
	211 - ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension-vtkdiff (Failed)
Errors while running CTest
[8] lehmannc@envinf141l:build-release-eigenlis$ OMP_NUM_THREADS=1 ctest -R ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension
Test project /home/lehmannc/prog/ogs6/github-chleh-PRs/build-release-eigenlis
    Start 210: ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension
1/2 Test #210: ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension ...........***Failed    9.42 sec
    Start 211: ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension-vtkdiff
2/2 Test #211: ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension-vtkdiff ...***Failed    0.05 sec

0% tests passed, 2 tests failed out of 2

Total Test time (real) =   9.50 sec

The following tests FAILED:
	210 - ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension (Failed)
	211 - ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension-vtkdiff (Failed)
Errors while running CTest
[8] lehmannc@envinf141l:build-release-eigenlis$ OMP_NUM_THREADS=1 ctest -R ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension
Test project /home/lehmannc/prog/ogs6/github-chleh-PRs/build-release-eigenlis
    Start 210: ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension
1/2 Test #210: ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension ...........***Failed    0.38 sec
    Start 211: ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension-vtkdiff
2/2 Test #211: ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension-vtkdiff ...***Failed    0.03 sec

0% tests passed, 2 tests failed out of 2

Total Test time (real) =   0.43 sec

The following tests FAILED:
	210 - ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension (Failed)
	211 - ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension-vtkdiff (Failed)
Errors while running CTest
[8] lehmannc@envinf141l:build-release-eigenlis$ OMP_NUM_THREADS=1 ctest -R ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension
Test project /home/lehmannc/prog/ogs6/github-chleh-PRs/build-release-eigenlis
    Start 210: ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension
1/2 Test #210: ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension ...........   Passed   10.72 sec
    Start 211: ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension-vtkdiff
2/2 Test #211: ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension-vtkdiff ...   Passed    0.05 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =  10.79 sec

Probably this is linked to the following compiler warnings: (gcc 7.1.1, dynamic shape matrices):

[ 63%] Building CXX object MaterialLib/CMakeFiles/MaterialLib.dir/SolidModels/Ehlers.cpp.o
In file included from /home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Lubby2.cpp:10:0:
/home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Lubby2.h: In instantiation of ‘std::unique_ptr<typename MaterialLib::Solids::MechanicsBase<DisplacementDim>::MaterialStateVariables> MaterialLib::Solids::Lubby2::Lubby2<DisplacementDim>::createMaterialStateVariables() [with int DisplacementDim = 2; typename MaterialLib::Solids::MechanicsBase<DisplacementDim>::MaterialStateVariables = MaterialLib::Solids::MechanicsBase<2>::MaterialStateVariables]’:
/home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Lubby2.cpp:19:16:   required from here
/home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Lubby2.h:160:13: warning: ‘new’ of type ‘MaterialLib::Solids::Lubby2::Lubby2<2>::MaterialStateVariables’ with extended alignment 32 [-Waligned-new=]
             new MaterialStateVariables};
             ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Lubby2.h:160:13: note: uses ‘void* operator new(std::size_t)’, which does not have an alignment parameter
/home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Lubby2.h:160:13: note: use ‘-faligned-new’ to enable C++17 over-aligned new support
In file included from /home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Lubby2.cpp:11:0:
/home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Lubby2-impl.h: In instantiation of ‘boost::optional<std::tuple<Eigen::Matrix<double, ProcessLib::KelvinVectorDimensions<DisplacementDim>::value, 1, 0, ProcessLib::KelvinVectorDimensions<DisplacementDim>::value, 1>, std::unique_ptr<typename MaterialLib::Solids::MechanicsBase<DisplacementDim>::MaterialStateVariables, std::default_delete<typename MaterialLib::Solids::MechanicsBase<DisplacementDim>::MaterialStateVariables> >, Eigen::Matrix<double, ProcessLib::KelvinVectorDimensions<DisplacementDim>::value, ProcessLib::KelvinVectorDimensions<DisplacementDim>::value, 1, ProcessLib::KelvinVectorDimensions<DisplacementDim>::value, ProcessLib::KelvinVectorDimensions<DisplacementDim>::value> > > MaterialLib::Solids::Lubby2::Lubby2<DisplacementDim>::integrateStress(double, const ProcessLib::SpatialPosition&, double, const KelvinVector&, const KelvinVector&, const KelvinVector&, const typename MaterialLib::Solids::MechanicsBase<DisplacementDim>::MaterialStateVariables&) [with int DisplacementDim = 2; typename MaterialLib::Solids::MechanicsBase<DisplacementDim>::MaterialStateVariables = MaterialLib::Solids::MechanicsBase<2>::MaterialStateVariables; MaterialLib::Solids::Lubby2::Lubby2<DisplacementDim>::KelvinVector = Eigen::Matrix<double, 4, 1>]’:
/home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Lubby2.cpp:19:16:   required from here
/home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Lubby2-impl.h:191:13: warning: ‘new’ of type ‘MaterialLib::Solids::Lubby2::Lubby2<2>::MaterialStateVariables’ with extended alignment 32 [-Waligned-new=]
             new MaterialStateVariables{state}},
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Lubby2-impl.h:191:13: note: uses ‘void* operator new(std::size_t)’, which does not have an alignment parameter
/home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Lubby2-impl.h:191:13: note: use ‘-faligned-new’ to enable C++17 over-aligned new support
In file included from /home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Ehlers.cpp:11:0:
/home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Ehlers-impl.h: In instantiation of ‘boost::optional<std::tuple<Eigen::Matrix<double, ProcessLib::KelvinVectorDimensions<DisplacementDim>::value, 1, 0, ProcessLib::KelvinVectorDimensions<DisplacementDim>::value, 1>, std::unique_ptr<typename MaterialLib::Solids::MechanicsBase<DisplacementDim>::MaterialStateVariables, std::default_delete<typename MaterialLib::Solids::MechanicsBase<DisplacementDim>::MaterialStateVariables> >, Eigen::Matrix<double, ProcessLib::KelvinVectorDimensions<DisplacementDim>::value, ProcessLib::KelvinVectorDimensions<DisplacementDim>::value, 1, ProcessLib::KelvinVectorDimensions<DisplacementDim>::value, ProcessLib::KelvinVectorDimensions<DisplacementDim>::value> > > MaterialLib::Solids::Ehlers::SolidEhlers<DisplacementDim>::integrateStress(double, const ProcessLib::SpatialPosition&, double, const KelvinVector&, const KelvinVector&, const KelvinVector&, const typename MaterialLib::Solids::MechanicsBase<DisplacementDim>::MaterialStateVariables&) [with int DisplacementDim = 2; typename MaterialLib::Solids::MechanicsBase<DisplacementDim>::MaterialStateVariables = MaterialLib::Solids::MechanicsBase<2>::MaterialStateVariables; MaterialLib::Solids::Ehlers::SolidEhlers<DisplacementDim>::KelvinVector = Eigen::Matrix<double, 4, 1>]’:
/home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Ehlers.cpp:19:16:   required from here
/home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Ehlers-impl.h:738:13: warning: ‘new’ of type ‘MaterialLib::Solids::Ehlers::StateVariables<2>’ with extended alignment 32 [-Waligned-new=]
             new StateVariables<DisplacementDim>{
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                 static_cast<StateVariables<DisplacementDim> const&>(state)}},
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Ehlers-impl.h:738:13: note: uses ‘void* operator new(std::size_t)’, which does not have an alignment parameter
/home/lehmannc/prog/ogs6/github-chleh-PRs/src/MaterialLib/SolidModels/Ehlers-impl.h:738:13: note: use ‘-faligned-new’ to enable C++17 over-aligned new support

Because adding add_definitions(-DEIGEN_DONT_VECTORIZE) to the main CMakeLists.txt fixes this problem: Now $ OMP_NUM_THREADS=1 ctest --repeat-until-fail 50 -j 3 -R ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension just runs fine.

@endJunction noticed that in some classes EIGEN_MAKE_ALIGNED_OPERATOR_NEW is missing.

This problem adds to a set of seven issues or pull requests regarding Eigen alignment: #743, #1221, #1250, #1323, #1483, #1684, #1760. Maybe we should consider disabling Eigen alignment entirely (at least by default).
Additionally, the warnings above are generated only with very recent compiler versions (cf. https://gcc.gnu.org/gcc-7/changes.html). So the average user might not get notified of the indicated problem.

@chleh chleh added the bug 🐛 label Jul 24, 2017

@TomFischer

This comment has been minimized.

Member

TomFischer commented Jul 27, 2017

I tested it without lis on my laptop. I compiled ogs with gcc 7.1.1 and disabled dynamic shape matrices. The tests passed ($ OMP_NUM_THREADS=1 ctest --repeat-until-fail 50 -j 3 -R ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension):

100% tests passed, 0 tests failed out of 2     

Total Test time (real) = 318.80 sec
@TomFischer

This comment has been minimized.

Member

TomFischer commented Jul 27, 2017

Now, I tested it also with lis and obtained the same results: 100% tests passed. I cannot reproduce the behaviour.

@chleh

This comment has been minimized.

Contributor

chleh commented Jul 27, 2017

Could you please test it with dynamically allocated shape matrices. Indeed for fixed ones tests seem OK for me, but for the dynamic ones they fail.

@chleh

This comment has been minimized.

Contributor

chleh commented Aug 10, 2017

Some HML tests were failing, too, fixed in ufz/ogs-data#69.

@chleh

This comment has been minimized.

Contributor

chleh commented Aug 10, 2017

Just for the recors: Even after the Eigen alginment corrections #1898 the following tests still fail sporadically:

	222 - ogs-Mechanics_PlasticModel_SDE-DP_Ehlers_SpecialCase_DruckerPrager (Failed)
	226 - ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension (Failed)
	228 - ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialCompression (Failed)
	230 - ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_TriaxialCompression (Failed)

Some OGS build info (generated using this script):

== info from CMakeCache.txt ==
CMAKE_BUILD_TYPE:STRING=Release
CMAKE_CXX_COMPILER:STRING=/usr/bin/g++
CMAKE_CXX_FLAGS_DEBUG:STRING=-g
CMAKE_CXX_FLAGS_MINSIZEREL:STRING=-Os -DNDEBUG
CMAKE_CXX_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
CMAKE_CXX_FLAGS_RELWITHDEBINFO:STRING=-O2 -g -DNDEBUG
CMAKE_CXX_FLAGS:STRING=
EIGEN3_INCLUDE_DIR:PATH=/usr/include/eigen3
EIGEN_NO_DEBUG:BOOL=OFF
FIND_PACKAGE_MESSAGE_DETAILS_Eigen3:INTERNAL=[/usr/include/eigen3][TRUE][v3.3.4(3.2.9)]
OGS_CPU_ARCHITECTURE:STRING=native
OGS_CXX_FLAGS:STRING=
OGS_EIGEN_DYNAMIC_SHAPE_MATRICES:BOOL=ON
OGS_LIB_EIGEN:STRING=Default
OGS_USE_EIGEN:BOOL=ON
OGS_USE_EIGEN_UNSUPPORTED:BOOL=ON

== ogs info ==
$ bin/ogs --version

bin/ogs  version: 6.0.8-1165-g6590244a9

== compiler info ==
$ LANG=C "/usr/bin/g++" "-march=native" -E -v - </dev/null 2>&1
Using built-in specs.
COLLECT_GCC=/usr/bin/g++
Target: x86_64-pc-linux-gnu
Configured with: /build/gcc-multilib/src/gcc/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++ --enable-shared --enable-threads=posix --enable-libmpx --with-system-zlib --with-isl --enable-__cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --disable-libssp --enable-gnu-unique-object --enable-linker-build-id --enable-lto --enable-plugin --enable-install-libiberty --with-linker-hash-style=gnu --enable-gnu-indirect-function --enable-multilib --disable-werror --enable-checking=release --enable-default-pie --enable-default-ssp
Thread model: posix
gcc version 7.1.1 20170630 (GCC) 
COLLECT_GCC_OPTIONS='-march=native' '-E' '-v' '-shared-libgcc'
 /usr/lib/gcc/x86_64-pc-linux-gnu/7.1.1/cc1 -E -quiet -v - -march=haswell -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3 -mno-sse4a -mcx16 -msahf -mmovbe -maes -mno-sha -mpclmul -mpopcnt -mabm -mno-lwp -mfma -mno-fma4 -mno-xop -mbmi -mno-sgx -mbmi2 -mno-tbm -mavx -mavx2 -msse4.2 -msse4.1 -mlzcnt -mno-rtm -mno-hle -mrdrnd -mf16c -mfsgsbase -mno-rdseed -mno-prfchw -mno-adx -mfxsr -mxsave -mxsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1 -mno-clflushopt -mno-xsavec -mno-xsaves -mno-avx512dq -mno-avx512bw -mno-avx512vl -mno-avx512ifma -mno-avx512vbmi -mno-avx5124fmaps -mno-avx5124vnniw -mno-clwb -mno-mwaitx -mno-clzero -mno-pku -mno-rdpid --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=3072 -mtune=haswell
ignoring nonexistent directory "/usr/lib/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../x86_64-pc-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-pc-linux-gnu/7.1.1/include
 /usr/local/include
 /usr/lib/gcc/x86_64-pc-linux-gnu/7.1.1/include-fixed
 /usr/include
End of search list.
# 1 "<stdin>"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "<stdin>"
COMPILER_PATH=/usr/lib/gcc/x86_64-pc-linux-gnu/7.1.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/7.1.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/:/usr/lib/gcc/x86_64-pc-linux-gnu/7.1.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-pc-linux-gnu/7.1.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../lib/:/lib/../lib/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-pc-linux-gnu/7.1.1/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-march=native' '-E' '-v' '-shared-libgcc'

== last ten git commits ==
$ git -C "$srcdir" log --oneline --decorate -n 10
902e8645b (HEAD -> master, upstream/master, origin/master, origin/HEAD, eigen-init-zero) Re-added PNG as git lfs file.
b27794239 Re-added PNGs as git lfs files.
ac95d3c55 Update Tests/Data pointer.
8ca1e68c7 Updated DOI link.
4be27f55e Typo in editorconfig.
47acf05aa Tests/Data set to master
c9e9e18a0 Merge pull request #1875 from chleh/internal-variables-output
ba7e3719c (origin/internal-variables-output, internal-variables-output) [PL] number of points instead of ip_data size
ec1359985 updated Tests/Data
d28b746dc [PL] added vectorial output for small deformation sigma, eps
@chleh

This comment has been minimized.

Contributor

chleh commented Aug 10, 2017

Still the add_definitions(-DEIGEN_DONT_VECTORIZE) catchall makes the mentioned tests run fine. Maybe the following could be causing the error:

  • -O3 optimizations
  • some bug in the Eigen 3.3 series, because as far as I remember, in Eigen 3.3 they introduced a major rewrite of their core expression evaluators.

@endJunction or @TomFischer, could you please test on your machines:
OMP_NUM_THREADS=1 ctest -j 3 -R Ehlers --repeat-until-fail 50
with OGS_EIGEN_DYNAMIC_SHAPE_MATRICES=ON

@chleh

This comment has been minimized.

Contributor

chleh commented Aug 11, 2017

@endJunction told me yesterday afternoon that he observed the same behaviour (sproradic fails) on his machine with dynamic shape matrices. Therefore this bug is reproducible. Correct me if I'm wrong.

@endJunction

This comment has been minimized.

Member

endJunction commented Aug 14, 2017

Correct. ogs-Mechanics_PlasticModel_SDED_Ehlers_Damage_UniaxialTension is failing sometimes, but only that (of the Ehlers selection).

@chleh

This comment has been minimized.

Contributor

chleh commented Aug 15, 2017

So should we disable vectorization for Eigen >= 3.3 && DynamicMatrices? (Eigen>=3.3 to be checked).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment