quick_mpi test not working for deal.ii9.0pre

[MFraters]

I noticed that the quick_mpi test is not properly functioning on my (unbutu 16) system when coupled to deal.ii 9.0pre. When I use deal.ii 8.5.1 it works fine. To be sure, I just reinstalled all the dependencies of deal.ii with the candi script and used the newest deal.ii and aspect master branches, but that doesn't help. The error message I get is the following:

$ ctest -VV
UpdateCTestConfiguration  from :/home/fraters/aspect/aspect_2017-09-26/aspect/build/DartConfiguration.tcl
 Add coverage exclude regular expressions.
SetCTestConfiguration:CMakeCommand:/usr/bin/cmake
UpdateCTestConfiguration  from :/home/fraters/aspect/aspect_2017-09-26/aspect/build/DartConfiguration.tcl
Test project /home/fraters/aspect/aspect_2017-09-26/aspect/build
Constructing a list of tests
Done constructing a list of tests
Checking test dependency graph...
Checking test dependency graph end
test 1
    Start 1: quick_mpi

1: Test command: /usr/bin/cmake "-DBINARY_DIR=/home/fraters/aspect/aspect_2017-09-26/aspect/build/tests" "-DTESTNAME=tests.quick_mpi" "-DERROR="Test quick_mpi failed"" "-P" "/home/fraters/aspect/aspect_2017-09-26/aspect/tests/run_test.cmake"
1: Test timeout computed to be: 600
1: 
1: --------------------------------------------------------
1: An error occurred in line <603> of file </home/fraters/deal.ii/deal.ii_2017-09-26/dealii/source/lac/trilinos_vector.cc> in function
1:     dealii::TrilinosScalar dealii::TrilinosWrappers::MPI::Vector::operator()(dealii::TrilinosWrappers::MPI::Vector::size_type) const
1: The violated condition was: 
1:     false
1: Additional information: 
1:     You tried to access element 30 of a distributed vector, but this element is not stored on the current processor. Note: There are 72 elements stored on the current processor from within the range 90 through 161 but Trilinos vectors need not store contiguous ranges on each processor, and not every element in this range may in fact be stored locally.
1: 
1: Stacktrace:
1: -----------
1: #0  /home/fraters/deal.ii/deal.ii_2017-09-26/dealii/build/lib/libdeal_II.g.so.9.0.0-pre: dealii::TrilinosWrappers::MPI::Vector::operator()(unsigned int) const
1: #1  /home/fraters/deal.ii/deal.ii_2017-09-26/dealii/build/lib/libdeal_II.g.so.9.0.0-pre: void dealii::VectorTools::interpolate<2, 2, dealii::TrilinosWrappers::MPI::BlockVector, dealii::DoFHandler>(dealii::Mapping<2, 2> const&, dealii::DoFHandler<2, 2> const&, dealii::Function<2, dealii::TrilinosWrappers::MPI::BlockVector::value_type> const&, dealii::TrilinosWrappers::MPI::BlockVector&, dealii::ComponentMask const&)
1: #2  /home/fraters/aspect/aspect_2017-09-26/aspect/build/aspect: aspect::Simulator<2>::set_initial_temperature_and_compositional_fields()
1: #3  /home/fraters/aspect/aspect_2017-09-26/aspect/build/aspect: aspect::Simulator<2>::run()
1: #4  /home/fraters/aspect/aspect_2017-09-26/aspect/build/aspect: main
1: --------------------------------------------------------
1: 
1: Calling MPI_Abort now.
1: To break execution in a GDB session, execute 'break MPI_Abort' before running. You can also put the following into your ~/.gdbinit:
1:   set breakpoint pending on
1:   break MPI_Abort
1:   set breakpoint pending auto
1: make[3]: *** [output-quick_mpi/screen-output] Error 255
1: make[2]: *** [CMakeFiles/quick_mpi.screen-output.diff.dir/all] Error 2
1: make[1]: *** [CMakeFiles/tests.quick_mpi.dir/rule] Error 2
1: make: *** [tests.quick_mpi] Error 2
1: *** "Test quick_mpi failed": ***
1: [ 22%] Built target quick_mpi
1: [ 33%] Generating output-quick_mpi/screen-output
1: CMakeFiles/quick_mpi.screen-output.diff.dir/build.make:76: recipe for target 'output-quick_mpi/screen-output' failed
1: CMakeFiles/Makefile2:164: recipe for target 'CMakeFiles/quick_mpi.screen-output.diff.dir/all' failed
1: CMakeFiles/Makefile2:107: recipe for target 'CMakeFiles/tests.quick_mpi.dir/rule' failed
1: Makefile:142: recipe for target 'tests.quick_mpi' failed
1: 
1: CMake Error at /home/fraters/aspect/aspect_2017-09-26/aspect/tests/run_test.cmake:14 (MESSAGE):
1:   *** Test aborted.
1: 
1: 
1/1 Test #1: quick_mpi ........................***Failed    0.43 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.43 sec

The following tests FAILED:
	  1 - quick_mpi (Failed)

Do other people have this problem, or have any idea's what is going on here?

[gassmoeller]

I have the same issue. I am going to investigate the deal.II dev history to find out if something introduced the error in there.

[gassmoeller]

It seems to me like this issue was introduced in dealii/dealii#4857. It is not just quick_mpi though. A number of other tests fail as well, in particular the ones using MPI (I have not yet seen a serial test fail, but I did not check all of them). We will need to fix this in deal.II. For all running into trouble with this issue: For me a workaround is to use deal.II older than commit 60bfd3170a465002ac1404dc36a5ba9671254b71.

[gassmoeller]

For reference: A different triggered assertion I see in some tests (in the same function) is:

--------------------------------------------------------
An error occurred in line <2639> of file </raid/gassmoel/software/dealii/source/fe/fe_values.cc> in function
    dealii::FEValuesBase<dim, spacedim>::FEValuesBase(unsigned int, unsigned int, dealii::UpdateFlags, const dealii::Mapping<dim, spacedim>&, const dealii::FiniteElement<dim, spacedim>&) [with int dim = 2; int spacedim = 2]
The violated condition was: 
    n_q_points > 0
Additional information: 
    There is nothing useful you can do with an FEValues object when using a quadrature formula with zero quadrature points!

Stacktrace:
-----------
#0  /raid/gassmoel/software/deal.II-dev/lib/libdeal_II.g.so.9.0.0-pre: dealii::FEValuesBase<2, 2>::FEValuesBase(unsigned int, unsigned int, dealii::UpdateFlags, dealii::Mapping<2, 2> const&, dealii::FiniteElement<2, 2> const&)
#1  /raid/gassmoel/software/deal.II-dev/lib/libdeal_II.g.so.9.0.0-pre: dealii::FEValues<2, 2>::FEValues(dealii::Mapping<2, 2> const&, dealii::FiniteElement<2, 2> const&, dealii::Quadrature<2> const&, dealii::UpdateFlags)
#2  /raid/gassmoel/software/deal.II-dev/lib/libdeal_II.g.so.9.0.0-pre: dealii::internal::hp::FEValuesBase<2, 2, dealii::FEValues<2, 2> >::select_fe_values(unsigned int, unsigned int, unsigned int)
#3  /raid/gassmoel/software/deal.II-dev/lib/libdeal_II.g.so.9.0.0-pre: void dealii::hp::FEValues<2, 2>::reinit<dealii::DoFHandler<2, 2>, false>(dealii::TriaIterator<dealii::DoFCellAccessor<dealii::DoFHandler<2, 2>, false> >, unsigned int, unsigned int, unsigned int)
#4  /raid/gassmoel/software/deal.II-dev/lib/libdeal_II.g.so.9.0.0-pre: void dealii::VectorTools::interpolate<2, 2, dealii::TrilinosWrappers::MPI::BlockVector, dealii::DoFHandler>(dealii::Mapping<2, 2> const&, dealii::DoFHandler<2, 2> const&, dealii::Function<2, dealii::TrilinosWrappers::MPI::BlockVector::value_type> const&, dealii::TrilinosWrappers::MPI::BlockVector&, dealii::ComponentMask const&)
#5  /scratch/gassmoel/aspect-debug/aspect: aspect::Simulator<2>::set_initial_temperature_and_compositional_fields()
#6  /scratch/gassmoel/aspect-debug/aspect: aspect::Simulator<2>::run()
#7  /scratch/gassmoel/aspect-debug/aspect: main
--------------------------------------------------------

[MFraters]

Thanks for looking into this! It somehow always feels better when you are not the only one having the problem ;)

[gassmoeller]

@bangerth: This issue in deal.II is still not resolved. Is there anything we can do to speed up the fix? If not we should probably send around an email on the mailing list warning people that deal.II-dev does currently not work with aspect.
I am asking, because it now affects my development in dealii/dealii#5125 as well. I have a branch of aspect that compiles with the deal.II particle code, but I can not test it, because this issue prevents ASPECT from running with deal.II-dev.

[bangerth]

@tamiko -- can you comment? The fact that a function that's used in ASPECT no longer works is a real problem!

[gassmoeller]

I should probably add that the error message has changed slightly since the original post, although it still originates in VectorTools::interpolate. If you need further debug information let me know, I have a lot of failing example tests 😄

--------------------------------------------------------
An error occurred in line <965> of file </raid/gassmoel/software/dealii/include/deal.II/base/tensor.h> in function
    dealii::Tensor<rank, dim, T>& dealii::Tensor<rank_, dim, Number>::operator=(const Number&) [with int rank_ = 1; int dim = 2; Number = double]
The violated condition was: 
    d == internal::NumberType<Number>::value(0.0)
Additional information: 
    Only assignment with zero is allowed

Stacktrace:
-----------
#0  /raid/gassmoel/software/deal.II-dev/lib/libdeal_II.g.so.9.0.0-pre: dealii::MappingCartesian<2, 2>::fill_fe_values(dealii::TriaIterator<dealii::CellAccessor<2, 2> > const&, dealii::CellSimilarity::Similarity, dealii::Quadrature<2> const&, dealii::Mapping<2, 2>::InternalDataBase const&, dealii::internal::FEValues::MappingRelatedData<2, 2>&) const
#1  /raid/gassmoel/software/deal.II-dev/lib/libdeal_II.g.so.9.0.0-pre: dealii::FEValues<2, 2>::do_reinit()
#2  /raid/gassmoel/software/deal.II-dev/lib/libdeal_II.g.so.9.0.0-pre: void dealii::FEValues<2, 2>::reinit<dealii::DoFHandler, false>(dealii::TriaIterator<dealii::DoFCellAccessor<dealii::DoFHandler<2, 2>, false> > const&)
#3  /raid/gassmoel/software/deal.II-dev/lib/libdeal_II.g.so.9.0.0-pre: void dealii::hp::FEValues<2, 2>::reinit<dealii::DoFHandler<2, 2>, false>(dealii::TriaIterator<dealii::DoFCellAccessor<dealii::DoFHandler<2, 2>, false> >, unsigned int, unsigned int, unsigned int)
#4  /raid/gassmoel/software/deal.II-dev/lib/libdeal_II.g.so.9.0.0-pre: 
#5  /raid/gassmoel/software/deal.II-dev/lib/libdeal_II.g.so.9.0.0-pre: void dealii::VectorTools::interpolate<2, 2, dealii::TrilinosWrappers::MPI::BlockVector, dealii::DoFHandler>(dealii::Mapping<2, 2> const&, dealii::DoFHandler<2, 2> const&, dealii::Function<2, dealii::TrilinosWrappers::MPI::BlockVector::value_type> const&, dealii::TrilinosWrappers::MPI::BlockVector&, dealii::ComponentMask const&)
#6  ../aspect: aspect::Simulator<2>::set_initial_temperature_and_compositional_fields()
#7  ../aspect: aspect::Simulator<2>::run()
#8  ../aspect: main
--------------------------------------------------------

[gassmoeller]

This was resolved by the above linked pull requests in deal.II. If someone still encounters problems feel free to reopen.