particle problem with deal.ii 9.2.0-pre but not 9.0.1

[MFraters]

When I run the prm file below (one of the tests but with 1 particle and 1 adaptive mesh refinement) with deal.ii v9.0.1 it runs fine, but when I run it with deal.ii 9.2.0-pre I get the error below.

The problem goes away when I use no mesh refinement and/or 2 or more particles. Any idea's on what the problem could be?

-----------------------------------------------------------------------------
-- This is ASPECT, the Advanced Solver for Problems in Earth's ConvecTion.
--     . version 2.2.0-pre (master, 750e62800)
--     . using deal.II 9.2.0-pre (fix_particle_probability_density_mpi_problem, 76bdd05a84)
--     .       with 32 bit indices and vectorization level 1 (128 bits)
--     . using Trilinos 12.10.1
--     . using p4est 2.2.0
--     . running in DEBUG mode
--     . running with 2 MPI processes
-----------------------------------------------------------------------------

-----------------------------------------------------------------------------
-- For information on how to cite ASPECT, see:
--   https://aspect.geodynamics.org/citing.html?ver=2.2.0-pre&particles=1&sha=750e62800&src=code
-----------------------------------------------------------------------------
Number of active cells: 16 (on 3 levels)
Number of degrees of freedom: 268 (162+25+81)

*** Timestep 0:  t=0 seconds
   Solving temperature system... 0 iterations.
   Rebuilding Stokes preconditioner...
   Solving Stokes system... 14+0 iterations.

Number of active cells: 16 (on 3 levels)
Number of degrees of freedom: 268 (162+25+81)


--------------------------------------------------------
An error occurred in line <1633> of file </home/.../dealii/source/distributed/tria.cc> in function
    void dealii::parallel::distributed::Triangulation<dim, spacedim>::DataTransfer::unpack_data(const std::vector<std::tuple<typename dealii::internal::p4est::types<dim>::quadrant*, typename dealii::Triangulation<dim, spacedim>::CellStatus, typename dealii::Triangulation<dim, spacedim>::cell_iterator> >&, unsigned int, const std::function<void(const typename dealii::Triangulation<dim, spacedim>::cell_iterator&, const typename dealii::Triangulation<dim, spacedim>::CellStatus&, const boost::iterator_range<__gnu_cxx::__normal_iterator<const char*, std::vector<char, std::allocator<char> > > >&)>&) const [with int dim = 2; int spacedim = 2; typename dealii::internal::p4est::types<dim>::quadrant = p4est_quadrant; typename dealii::Triangulation<dim, spacedim>::CellStatus = dealii::Triangulation<2, 2>::CellStatus; typename dealii::Triangulation<dim, spacedim>::cell_iterator = dealii::TriaIterator<dealii::CellAccessor<2, 2> >]
The violated condition was: 
    dest_data_variable.size() > 0
Additional information: 
    No data has been received!

Stacktrace:
-----------
#0  /home/.../dealii/build/lib/libdeal_II.g.so.9.2.0-pre: dealii::parallel::distributed::Triangulation<2, 2>::DataTransfer::unpack_data(std::vector<std::tuple<p4est_quadrant*, dealii::Triangulation<2, 2>::CellStatus, dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<std::tuple<p4est_quadrant*, dealii::Triangulation<2, 2>::CellStatus, dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > > const&, unsigned int, std::function<void (dealii::TriaIterator<dealii::CellAccessor<2, 2> > const&, dealii::Triangulation<2, 2>::CellStatus const&, boost::iterator_range<__gnu_cxx::__normal_iterator<char const*, std::vector<char, std::allocator<char> > > > const&)> const&) const
#1  /home/.../dealii/build/lib/libdeal_II.g.so.9.2.0-pre: dealii::parallel::distributed::Triangulation<2, 2>::notify_ready_to_unpack(unsigned int, std::function<void (dealii::TriaIterator<dealii::CellAccessor<2, 2> > const&, dealii::Triangulation<2, 2>::CellStatus, boost::iterator_range<__gnu_cxx::__normal_iterator<char const*, std::vector<char, std::allocator<char> > > > const&)> const&)
#2  /home/.../dealii/build/lib/libdeal_II.g.so.9.2.0-pre: dealii::Particles::ParticleHandler<2, 2>::register_load_callback_function(bool)
#3  ./aspect: aspect::Particle::World<2>::connect_to_signals(aspect::SimulatorSignals<2>&)::{lambda(dealii::parallel::distributed::Triangulation<2, 2>&)#3}::operator()(dealii::parallel::distributed::Triangulation<2, 2>&) const
#4  ./aspect: boost::detail::function::void_function_obj_invoker1<aspect::Particle::World<2>::connect_to_signals(aspect::SimulatorSignals<2>&)::{lambda(dealii::parallel::distributed::Triangulation<2, 2>&)#3}, void, dealii::parallel::distributed::Triangulation<2, 2>&>::invoke(boost::detail::function::function_buffer&, dealii::parallel::distributed::Triangulation<2, 2>&)
#5  ./aspect: boost::function1<void, dealii::parallel::distributed::Triangulation<2, 2>&>::operator()(dealii::parallel::distributed::Triangulation<2, 2>&) const
#6  ./aspect: boost::signals2::detail::void_type boost::signals2::detail::call_with_tuple_args<boost::signals2::detail::void_type>::m_invoke<boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)>, 0u, dealii::parallel::distributed::Triangulation<2, 2>&>(boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)>&, boost::signals2::detail::unsigned_meta_array<0u>, std::tuple<dealii::parallel::distributed::Triangulation<2, 2>&> const&, boost::enable_if<boost::is_void<boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)>::result_type>, void>::type*) const
#7  ./aspect: boost::signals2::detail::void_type boost::signals2::detail::call_with_tuple_args<boost::signals2::detail::void_type>::operator()<boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)>, dealii::parallel::distributed::Triangulation<2, 2>&, 1ul>(boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)>&, std::tuple<dealii::parallel::distributed::Triangulation<2, 2>&> const&, mpl_::size_t<1ul>) const
#8  ./aspect: boost::signals2::detail::void_type boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, dealii::parallel::distributed::Triangulation<2, 2>&>::operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> > >(boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> > const&) const
#9  ./aspect: boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, dealii::parallel::distributed::Triangulation<2, 2>&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> >::dereference() const
#10  ./aspect: boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, dealii::parallel::distributed::Triangulation<2, 2>&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> >::reference boost::iterators::iterator_core_access::dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, dealii::parallel::distributed::Triangulation<2, 2>&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, dealii::parallel::distributed::Triangulation<2, 2>&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> > const&)
#11  ./aspect: boost::iterators::detail::iterator_facade_base<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, dealii::parallel::distributed::Triangulation<2, 2>&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> >, boost::signals2::detail::void_type, boost::iterators::single_pass_traversal_tag, boost::signals2::detail::void_type const&, long, false, false>::operator*() const
#12  ./aspect: void boost::signals2::optional_last_value<void>::operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, dealii::parallel::distributed::Triangulation<2, 2>&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, dealii::parallel::distributed::Triangulation<2, 2>&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, dealii::parallel::distributed::Triangulation<2, 2>&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> >) const
#13  ./aspect: void boost::signals2::detail::combiner_invoker<void>::operator()<boost::signals2::optional_last_value<void>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, dealii::parallel::distributed::Triangulation<2, 2>&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> > >(boost::signals2::optional_last_value<void>&, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, dealii::parallel::distributed::Triangulation<2, 2>&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, dealii::parallel::distributed::Triangulation<2, 2>&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)> >, boost::signals2::mutex> >) const
#14  ./aspect: boost::signals2::detail::signal_impl<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)>, boost::function<void (boost::signals2::connection const&, dealii::parallel::distributed::Triangulation<2, 2>&)>, boost::signals2::mutex>::operator()(dealii::parallel::distributed::Triangulation<2, 2>&)
#15  ./aspect: boost::signals2::signal<void (dealii::parallel::distributed::Triangulation<2, 2>&), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (dealii::parallel::distributed::Triangulation<2, 2>&)>, boost::function<void (boost::signals2::connection const&, dealii::parallel::distributed::Triangulation<2, 2>&)>, boost::signals2::mutex>::operator()(dealii::parallel::distributed::Triangulation<2, 2>&)
#16  ./aspect: aspect::Simulator<2>::refine_mesh(unsigned int)
#17  ./aspect: aspect::Simulator<2>::maybe_do_initial_refinement(unsigned int)
#18  ./aspect: aspect::Simulator<2>::run()
#19  ./aspect: void run_simulator<2>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, bool)
#20  ./aspect: main
--------------------------------------------------------

Calling MPI_Abort now.
To break execution in a GDB session, execute 'break MPI_Abort' before running. You can also put the following into your ~/.gdbinit:
  set breakpoint pending on
  break MPI_Abort
  set breakpoint pending auto
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 255.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

set Dimension                              = 2
set End time                               = 0
set Use years in output instead of seconds = false

subsection Geometry model
  set Model name = box
  subsection Box
    set X extent  = 0.9142
    set Y extent  = 1.0000
  end
end

subsection Boundary velocity model
  set Tangential velocity boundary indicators = left, right
end

subsection Boundary velocity model
  set Zero velocity boundary indicators       = bottom, top
end

subsection Material model
  set Model name = simple
  subsection Simple model
    set Reference density             = 1010
    set Viscosity                     = 1e2
    set Thermal expansion coefficient = 0
  end
end

subsection Gravity model
  set Model name = vertical
  subsection Vertical
    set Magnitude = 10
  end
end

subsection Boundary temperature model
  set List of model names = box
end

subsection Initial temperature model
  set Model name = function
  subsection Function
    set Variable names = x,y,z
    set Function expression = x
  end
end

subsection Material model
  subsection Simple model
    set Density differential for compositional field 1 = -10
  end
end

subsection Mesh refinement
  set Initial adaptive refinement        = 1
  set Initial global refinement          = 2
  set Time steps between mesh refinement = 0
  set Coarsening fraction                = 0
  set Refinement fraction                = 0
end

subsection Postprocess
  set List of postprocessors = velocity statistics, composition statistics, particles

  subsection Particles
    set Number of particles = 1
    set Time between data output = 0
    set Data output format = ascii
    set Particle generator name = random uniform

    subsection Generator
      subsection Probability density function
        set Random cell selection = true
      end
    end
  end
end

[gassmoeller]

Could you check if it only happens on 2 MPI processes when one process has all the particles (no matter how many) and the other has none? This looks like a problem with the particle transfer during mesh refinement, and in particular something about how much data each process expects to get.

[MFraters]

I looked at the 2 particle case, and they both where in a different file/on a different processor. When I made an ascii file with two points on the same location and ran it again, it produced the same error. So it indeed seems to be a problem on a mpi process having no particles.

[bangerth]

The difference, btw, is whether we use functionality in ASPECT (when using deal.II 9.0) or in deal.II where much of this code has moved later and from which ASPECT takes it when available in deal.II (using 9.2.pre).

[marcfehling]

We overhauled the data transfer algorithm with deal.II version 9.1. In its process, we introduced an overstrict assertion while sending variable size data: Every cell was expected to receive variable size data once a corresponding pack/unpack callback has been registered. The logic in the whole send/receive apparatus should be okay though -- it was just this assertion. I guess your problem would vanish if you'll work in release mode.

We removed this assertion in dealii/dealii#10215. Hopefully we can fix this with the 9.2 release with dealii/dealii#10216.

[MFraters]

Thanks for letting us know!

[gassmoeller]

This was fixed in deal.II 9.2.0. deal.II 9.1.0 unfortunately will keep this bug.