Destroying a non-empty channel causes an assertion failure

[JLTastet]

I am not quite sure whether the following code is legit, but since it causes an assertion failure inside HPX, I am opening an issue anyway.

// main.cpp

#include <hpx/hpx.hpp>
#include <hpx/hpx_main.hpp>
#include <hpx/lcos/local/channel.hpp> // OR #include <hpx/lcos/channel.hpp>
#include <iostream>

// HPX_REGISTER_CHANNEL(int); // For hpx::lcos::channel<int>

int main()
{
    {
        hpx::lcos::local::channel<int> ch; // OR hpx::lcos::channel<int> ch(hpx::find_here());
        ch.set(0);
        // ch.get(); // Uncomment to prevent assertion failure
    } // <-- ch is destroyed here
    std::cerr << "This message is not printed" << std::endl;
    return 0;
}

The above snippet, compiled with:

g++ main.cpp -o main `pkg-config --cflags --libs hpx_application` -lhpx_iostreams -D HPX_APPLICATION

results on my system in:

$ ./main 

{stack-trace}: 4 frames:
0x7f0f674006dc  : hpx::detail::backtrace[abi:cxx11](unsigned long) + 0x9c in /home/jtastet/.local/lib/libhpx.so.1
0x7f0f6740c677  : std::__exception_ptr::exception_ptr hpx::detail::get_exception<hpx::exception>(hpx::exception const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x97 in /home/jtastet/.local/lib/libhpx.so.1
0x7f0f6740cb95  : void hpx::detail::throw_exception<hpx::exception>(hpx::exception const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) + 0x55 in /home/jtastet/.local/lib/libhpx.so.1
0x7f0f6740299e  : hpx::detail::assertion_failed_msg(char const*, char const*, char const*, char const*, long) + 0x21e in /home/jtastet/.local/lib/libhpx.so.1
{env}: 52 entries:
  CASTOR_HOME=/castor/cern.ch/user/j/jtastet
  CC=/afs/cern.ch/sw/lcg/external/gcc/6.3/x86_64-slc6/bin/gcc
  CMS_PATH=/cvmfs/cms.cern.ch
  COMPILER_PATH=/afs/cern.ch/sw/lcg/external/gcc/6.3/x86_64-slc6/lib/gcc/x86_64-unknown-linux-gnu/6.3.0
  CPATH=/home/jtastet/.local/include::/usr/include/python2.7:/usr/local/cuda-9.0/include
  CVSROOT=:gserver:cmssw.cvs.cern.ch:/local/reps/CMSSW
  CVS_RSH=ssh
  CXX=/afs/cern.ch/sw/lcg/external/gcc/6.3/x86_64-slc6/bin/g++
  DISPLAY=localhost:10.0
  EDITOR=/bin/nano -w
  FC=/afs/cern.ch/sw/lcg/external/gcc/6.3/x86_64-slc6/bin/gfortran
  GROUP_DIR=/afs/cern.ch/group/zh
  G_BROKEN_FILENAMES=1
  HISTCONTROL=ignoredups
  HISTSIZE=1000
  HOME=/afs/cern.ch/user/j/jtastet
  HOSTNAME=felk40.cern.ch
  KRB5CCNAME=FILE:/tmp/krb5cc_104489_Jm0zh7
  LANG=en_US.UTF-8
  LC_ALL=C
  LD_LIBRARY_PATH=/home/jtastet/.local/lib:/afs/cern.ch/sw/lcg/external/gcc/6.3/x86_64-slc6/lib64:/usr/local/cuda-9.0/lib64:.:/tmp/jtastet/delta-reduction/build
  LESSOPEN=||/usr/bin/lesspipe.sh %s
  LOGNAME=jtastet
  LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
  MAIL=/var/spool/mail/jtastet
  MANPATH=/cvmfs/cms.cern.ch/share/man:
  OLDPWD=/tmp/jtastet
  PATH=/home/jtastet/.local/bin:/afs/cern.ch/sw/lcg/external/CMake/3.5.2/Linux-x86_64/bin:/afs/cern.ch/sw/lcg/external/gcc/6.3/x86_64-slc6/bin:/afs/cern.ch/group/zh/bin:/afs/cern.ch/user/j/jtastet/scripts:/afs/cern.ch/cms/caf/scripts:/cvmfs/cms.cern.ch/common:/cvmfs/cms.cern.ch/bin:/usr/sue/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/cuda-9.0/bin:/afs/cern.ch/user/j/jtastet/perl5/bin:/afs/cern.ch/user/j/jtastet/bin
  PKG_CONFIG_PATH=/home/jtastet/.local/lib/pkgconfig:
  PWD=/tmp/jtastet/delta-reduction
  QTDIR=/usr/lib64/qt-3.3
  QTINC=/usr/lib64/qt-3.3/include
  QTLIB=/usr/lib64/qt-3.3/lib
  RFIO_USE_CASTOR_V2=YES
  SCRAM_ARCH=slc6_amd64_gcc472
  SCRATCH=/tmp/jtastet
  SELINUX_LEVEL_REQUESTED=
  SELINUX_ROLE_REQUESTED=
  SELINUX_USE_CURRENT_RANGE=
  SHELL=/bin/bash
  SHLVL=1
  SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
  SSH_CLIENT=128.141.174.25 37206 22
  SSH_CONNECTION=128.141.174.25 37206 128.141.174.239 22
  SSH_TTY=/dev/pts/1
  STAGE_HOST=castorcms
  TERM=xterm
  TEXINPUTS=:/afs/cern.ch/group/zh/latex/styles
  TMPDIR=/tmp/jtastet
  USER=jtastet
  WWW_HOME=http://cmsdoc.cern.ch/cms.html
  _=./main
{locality-id}: 0
{hostname}: [ (tcp:127.0.0.1:7910) ]
{process-id}: 2794
{function}: hpx::lcos::local::receive_buffer<T, Mutex>::~receive_buffer() [with T = int; Mutex = hpx::lcos::local::no_mutex]
{file}: /home/jtastet/.local/stow/hpx_gcc-6.3/include/hpx/lcos/local/receive_buffer.hpp
{line}: 101
{os-thread}: 1, worker-thread#1
{thread-id}: 00007f0f5adaa760
{thread-description}: <unknown>
{state}: state_running
{auxinfo}: 
{config}:
  HPX_WITH_AGAS_DUMP_REFCNT_ENTRIES=OFF
  HPX_WITH_APEX=OFF
  HPX_WITH_ATTACH_DEBUGGER_ON_TEST_FAILURE=OFF
  HPX_WITH_AUTOMATIC_SERIALIZATION_REGISTRATION=ON
  HPX_WITH_CXX14_RETURN_TYPE_DEDUCTION=TRUE
  HPX_WITH_DEPRECATION_WARNINGS=ON
  HPX_WITH_EXECUTION_POLICY_COMPATIBILITY=ON
  HPX_WITH_EXECUTOR_COMPATIBILITY=ON
  HPX_WITH_GOOGLE_PERFTOOLS=OFF
  HPX_WITH_HWLOC=ON
  HPX_WITH_INCLUSIVE_SCAN_COMPATIBILITY=ON
  HPX_WITH_IO_COUNTERS=ON
  HPX_WITH_IO_POOL=ON
  HPX_WITH_ITTNOTIFY=OFF
  HPX_WITH_LOCAL_DATAFLOW_COMPATIBILITY=ON
  HPX_WITH_LOGGING=ON
  HPX_WITH_MORE_THAN_64_THREADS=(
  HPX_WITH_NATIVE_TLS=ON
  HPX_WITH_NETWORKING=ON
  HPX_WITH_PAPI=OFF
  HPX_WITH_PARCELPORT_ACTION_COUNTERS=OFF
  HPX_WITH_PARCELPORT_LIBFABRIC=OFF
  HPX_WITH_PARCELPORT_MPI=ON
  HPX_WITH_PARCELPORT_MPI_MULTITHREADED=ON
  HPX_WITH_PARCELPORT_TCP=ON
  HPX_WITH_PARCELPORT_VERBS=OFF
  HPX_WITH_PARCEL_COALESCING=ON
  HPX_WITH_PARCEL_PROFILING=OFF
  HPX_WITH_SCHEDULER_LOCAL_STORAGE=OFF
  HPX_WITH_SPINLOCK_DEADLOCK_DETECTION=OFF
  HPX_WITH_STACKTRACES=ON
  HPX_WITH_SWAP_CONTEXT_EMULATION=OFF
  HPX_WITH_THREAD_BACKTRACE_ON_SUSPENSION=OFF
  HPX_WITH_THREAD_CREATION_AND_CLEANUP_RATES=OFF
  HPX_WITH_THREAD_CUMULATIVE_COUNTS=ON
  HPX_WITH_THREAD_DEBUG_INFO=OFF
  HPX_WITH_THREAD_DESCRIPTION_FULL=OFF
  HPX_WITH_THREAD_GUARD_PAGE=ON
  HPX_WITH_THREAD_IDLE_RATES=OFF
  HPX_WITH_THREAD_LOCAL_STORAGE=OFF
  HPX_WITH_THREAD_MANAGER_IDLE_BACKOFF=ON
  HPX_WITH_THREAD_QUEUE_WAITTIME=OFF
  HPX_WITH_THREAD_STACK_MMAP=ON
  HPX_WITH_THREAD_STEALING_COUNTS=ON
  HPX_WITH_THREAD_TARGET_ADDRESS=OFF
  HPX_WITH_TIMER_POOL=ON
  HPX_WITH_TRANSFORM_REDUCE_COMPATIBILITY=ON
  HPX_WITH_TUPLE_RVALUE_SWAP=ON
  HPX_WITH_UNWRAPPED_COMPATIBILITY=ON
  HPX_WITH_VALGRIND=OFF
  HPX_WITH_VERIFY_LOCKS=OFF
  HPX_WITH_VERIFY_LOCKS_BACKTRACE=OFF
  HPX_WITH_VERIFY_LOCKS_GLOBALLY=OFF

  HPX_PARCEL_MAX_CONNECTIONS=512
  HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
  HPX_AGAS_LOCAL_CACHE_SIZE=4096
  HPX_HAVE_MALLOC=jemalloc
  HPX_PREFIX (configured)=/home/jtastet/software/hpx/build
  HPX_PREFIX=/home/jtastet/.local
{version}: V1.1.0 (AGAS: V3.0), Git: 519b074e22
{boost}: V1.65.0
{build-type}: release
{date}: Sep  6 2017 14:33:36
{platform}: linux
{compiler}: GNU C++ version 6.3.0
{stdlib}: GNU libstdc++ version 20161221
{what}: assertion 'buffer_map_.empty()' failed: HPX(assertion_failure)

Aborted (core dumped)

I have looked at the channel documentation, and yet I have not figured out whether each set() should be accompanied by a corresponding get().

If that is the case, then we could consider adding it to the documentation. Otherwise, it looks like a bug ;-)

[JLTastet]

Note that calling ch.close() explicitly does not prevent the assertion failure, although it does call receive_buffer<>::cancel_waiting(), which should result in an empty buffer.

[sithhell]

Am 08.09.2017 11:19 vorm. schrieb "Element-126" <notifications@github.com>: Note that calling ch.close() explicitly does not prevent the assertion failure, although it does call receive_buffer<>::cancel_waiting(), which should result in an empty buffer. I think ultimately, the assertion is correct and should also fire on closing the channel. It points to a potential programming error for not consuming all received values. `cancel_waiting` is the way out to explicitly express that you want to discard the non received values. With that being said, the documentation needs indeed some love and a consensus on the semantics needs to be formed.

[JLTastet]

I almost certainly abused the semantics of channel<> by trying to use it to schedule resources (which actually works quite well until the channel is destructed). Indeed, in this case, closing a channel with zero elements would mean that some resources have not been returned, and would point to an error.

For my current use case, I guess the right thing to do is to implement a new class dedicated to resource management and inspired from channel. I was just being lazy :-)

Otherwise, do you know if it is possible to directly call cancel_waiting() on a channel<>. What surprises me is that although it seems to be called from channel<>::close(), the assertion failure still fires.

Once you are settled on the correct semantics of channel<>, it might make sense to instead throw an exception with a more descriptive error message.

[hkaiser]

@Element-126 The intended mechanism for this is to call close() on the channel. If this does not work it needs fixing...

[hkaiser]

@Element-126 I have added a Boolean option to channel::close() which force-deletes remaining items in the channel. This was necessary as for other use cases the close operation has to leave still pending entries in place, just signifying that no more entries will be available. May I ask you to try #2907 whether it fixes your issue?

[JLTastet]

@hkaiser Thanks a lot ! I will pull your branch and try your fix today.

[JLTastet]

@hkaiser The fix seems to be working. Thanks again !