Bees dump core on "--help"

[kakra]

Related by query: #81

I found the following crash when running bees --help:

           PID: 3017 (bees)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 6 (ABRT)
     Timestamp: Mon 2019-10-14 19:24:01 CEST (3min 1s ago)
  Command Line: /usr/libexec/bees --help
    Executable: /usr/libexec/bees
 Control Group: /user.slice/user-1000.slice/session-c253.scope
          Unit: session-c253.scope
         Slice: user-1000.slice
       Session: c253
     Owner UID: 1000 (kakra)
       Boot ID: d99daeb7f2824f27a245babc61d66fde
    Machine ID: f5de59774fd5867f1cfa08e45cb894ad
      Hostname: vch01
       Storage: /var/lib/systemd/coredump/core.bees.0.d99daeb7f2824f27a245babc61d66fde.3017.1571073841000000.lz4
       Message: Process 3017 (bees) of user 0 dumped core.

                Stack trace of thread 3223:
                #0  0x00007f37f4c4be3b __GI_raise (libc.so.6)
                #1  0x00007f37f4c35535 __GI_abort (libc.so.6)
                #2  0x00007f37f4fd89d7 _ZN9__gnu_cxx27__verbose_terminate_handlerEv (libstdc++.so.6)
                #3  0x00007f37f4ffc496 _ZN10__cxxabiv111__terminateEPFvvE (libstdc++.so.6)
                #4  0x00007f37f4ffc4d1 _ZSt9terminatev (libstdc++.so.6)
                #5  0x00007f37f4ffbeca __gxx_personality_v0 (libstdc++.so.6)
                #6  0x00007f37f4df2de8 _Unwind_ForcedUnwind_Phase2 (libgcc_s.so.1)
                #7  0x00007f37f4df35d5 _Unwind_Resume (libgcc_s.so.1)
                #8  0x00007f37f508467c _ZSt16__ostream_insertIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_PKS3_l (libstdc++.so.6)
                #9  0x000055b93aeca5e9 _ZStlsIcSt11char_traitsIcESaIcEERSt13basic_ostreamIT_T0_ES7_RKNSt7__cxx1112basic_stringIS4_S5_T1_EE (bees)
                #10 0x000055b93aebcd8f _ZZN10BeesThread4execESt8functionIFvvEEENKUlvE_clEv (bees)
                #11 0x00007f37f502761e execute_native_thread_routine (libstdc++.so.6)
                #12 0x00007f37f5154408 start_thread (libpthread.so.0)
                #13 0x00007f37f4d18f4f __clone (libc.so.6)

                Stack trace of thread 3017:
                #0  0x00007f37f5155977 __GI___pthread_timedjoin_ex (libpthread.so.0)
                #1  0x00007f37f5027893 __gthread_join (libstdc++.so.6)
                #2  0x000055b93aebd41a _ZN10BeesThreadD2Ev (bees)
                #3  0x000055b93ae7a5b0 _ZN11BeesContextD4Ev (bees)
                #4  0x000055b93ae75f37 _ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv (bees)
                #5  0x000055b93ae7110b _ZNSt14__shared_countILN9__gnu_cxx12_Lock_policyE2EED4Ev (bees)
                #6  0x000055b93ae71d79 operator() (bees)
                #7  0x000055b93aecbd73 _ZNKSt8functionIFvvEEclEv (bees)
                #8  0x000055b93ae6b83b main (bees)
                #9  0x00007f37f4c36e5b __libc_start_main (libc.so.6)
                #10 0x000055b93ae6c51a _start (bees)

                Stack trace of thread 3222:
                #0  0x00007f37f4d281ab __lll_lock_wait_private (libc.so.6)
                #1  0x00007f37f4c870a3 __GI__IO_fwrite (libc.so.6)
                #2  0x00007f37f5084384 _ZNSt15basic_streambufIcSt11char_traitsIcEE5sputnEPKcl (libstdc++.so.6)
                #3  0x000055b93aeca5e9 _ZStlsIcSt11char_traitsIcESaIcEERSt13basic_ostreamIT_T0_ES7_RKNSt7__cxx1112basic_stringIS4_S5_T1_EE (bees)
                #4  0x000055b93aebcd8f _ZZN10BeesThread4execESt8functionIFvvEEENKUlvE_clEv (bees)
                #5  0x00007f37f502761e execute_native_thread_routine (libstdc++.so.6)
                #6  0x00007f37f5154408 start_thread (libpthread.so.0)
                #7  0x00007f37f4d18f4f __clone (libc.so.6)

Bees version is v0.6-75-g7117cb4

2019-10-14 19:28:32 4944.4944<7> bees: BeesThread destructor status_report
2019-10-14 19:28:32 4944.4944<7> bees: Cancelling thread status_report
2019-10-14 19:28:32 4944.4944<7> bees: Waiting for thread status_report
terminate called without an active exception
Abgebrochen (Speicherabzug geschrieben)

I'm not sure why the message is in German even with LANG=C, but it says "Aborted (core dumped)".

[ghost]

Actually also having problems with the latest git versions:
This is running beesd --help

 # beesd --help
Usage: beesd [options] <btrfs_uuid>
- - -
bees version v0.6-76-g4363463-dirty
2019-11-27 10:22:10 12393.12393<7> bees: Masking signals
2019-11-27 10:22:10 12393.12393<7> bees: BeesThread exec progress_report
2019-11-27 10:22:10 12393.12393<7> bees: BeesThread exec status_report
2019-11-27 10:22:10 12393.12400<7> progress_report: Starting thread progress_report
Usage: /usr/lib/bees/bees [options] fs-root-path [fs-root-path-2...]
Performs best-effort extent-same deduplication on btrfs.

fs-root-path MUST be the root of a btrfs filesystem tree (id 5).
Other directories will be rejected.

Options:
    -h, --help            Show this help

Load management options:
    -c, --thread-count    Worker thread count (default CPU count * factor)
    -C, --thread-factor   Worker thread factor (default 1)
    -G, --thread-min      Minimum worker thread count (default 0)
    -g, --loadavg-target  Target load average for worker threads (default none)

Filesystem tree traversal options:
    -m, --scan-mode       Scanning mode (0..2, default 0)

Workarounds:
    -a, --workaround-btrfs-send    Workaround for btrfs send

Logging options:
    -t, --timestamps      Show timestamps in log output (default)
    -T, --no-timestamps   Omit timestamps in log output
    -p, --absolute-paths  Show absolute paths (default)
    -P, --strip-paths     Strip $CWD from beginning of all paths in the log
    -v, --verbose         Set maximum log level (0..8, default 8)

Optional environment variables:
    BEESHOME    Path to hash table and configuration files
                (default is .beeshome/ in the root of each filesystem).

    BEESSTATUS  File to write status to (tmpfs recommended, e.g. /run).
                No status is written if this variable is unset.


2019-11-27 10:22:10 12393.12393<7> bees: BeesThread destructor status_report
2019-11-27 10:22:10 12393.12393<7> bees: Cancelling thread status_report
2019-11-27 10:22:10 12393.12393<7> bees: Waiting for thread status_report
terminate called without an active exception
Aborted

/etc/bees/6tb.conf:

UUID="fe0a1142-51ab-4181-b635-adbf9f4ea6e6"

## System Vars
# Change carefully
WORK_DIR="/run/bees"
MNT_DIR="$WORK_DIR/mnt/$UUID"
BEESHOME="$MNT_DIR/.beeshome"
BEESSTATUS="$WORK_DIR/$UUID.status"
DB_SIZE=$((256*1024*1024)) # 256MiB in bytes

And this is trying to run bees manually

# BEESHOME="/mnt/6TB/.beeshome/" /usr/lib/bees/bees -v /mnt/6TB/
bees version v0.6-76-g4363463-dirty
2019-11-27 10:27:34 12709.12709<7> bees: Masking signals
2019-11-27 10:27:34 12709.12709<7> bees: BeesThread exec progress_report
2019-11-27 10:27:34 12709.12709<7> bees: BeesThread exec status_report
2019-11-27 10:27:34 12709.12710<7> progress_report: Starting thread progress_report
2019-11-27 10:27:34 12709.12709<7> bees: BeesThread destructor status_report
2019-11-27 10:27:34 12709.12709<7> bees: Cancelling thread status_report
2019-11-27 10:27:34 12709.12709<7> bees: Waiting for thread status_report
terminate called without an active exception
Aborted

I'm using gcc-9.2.0 on Gentoo with kernel 5.4.0.

[kakra]

Is this a coredump? It says "Aborted" without "core dumped".

I've posted a few PR that may have an impact on this. You may want to try merging those locally and rebuild. Also, after changing the GCC version you should clean the build directory. You should be fine, tho, when using the 9999 ebuild.

If you want to use the 9999 ebuild with my PRs applied, just checkout the branches locally and run git format-patch origin/master per each branch. It will add patch files to the working directory. So, please run it in the same directory for each branch. Then symlink /etc/portage/patches/sys-fs/bees-9999 to that git working directory. Rebuilding bees using the 9999 ebuild will now apply the exported patches. To get rid, just delete the patch files and rebuild. That way you can try out and add branch patches any time you want and revert easily.

#136

Maybe also try this and see if it relaxes resource usage and report back there:
#135

[ghost]

I tried two ways. First I did a normal got clone to a local did and did make/make install. Secondly I tried the ebuild 9999 version. I removed all old files first. Same issue both times. I haven't tried those patches yet.

[kakra]

Could you rebuild glibc, then rebuild bees?

[ghost]

Could you rebuild glibc, then rebuild bees?

Did that. Seems to work now. Why is that?

[kakra]

I don't know, a similar (or identical) crash crept in before:
#124

Seems to be Gentoo specific. Rebuilding glibc under new gcc seems to fix it, as you tested. I saw a few similar problems also in wine. Also, to get some programs running which have been compiled with older gcc (i.e., most Steam games), I needed to add -mstackrealign to a few selected packages.

# /etc/portage/package.env
# Steam compat (old pre-gcc-8 compiles)
media-libs/openal stackrealign
media-sound/pulseaudio stackrealign
sys-libs/glibc stackrealign
x11-libs/libxcb stackrealign

Maybe it's something similar in this case. Some other packages may also be affected in your system. You may want to run emerge -DNua world --changed-deps --with-bdeps=y --usepkg=n to recompile everything depending on packages changed lately. Should bring everything back into shape without recompiling the whole world.

[Zygo]

Seems to be Gentoo specific.

FWIW I've never seen this behavior on Debian.

[ghost]

I had a look and glibc was updated not too long ago, so it looks like that might be it. Perhaps a recompile emerge -e worldmight be in order. Thanks for pointing this out.

[kakra]

Seems to be Gentoo specific.

FWIW I've never seen this behavior on Debian.

@Zygo Yes, because in Debian, glibc, gcc and all other packages are from the same building session. In Gentoo, this is different. Packages sometimes need to be built in specific order, and the emerge tool does not always enforce this (it has build time source dependencies but lacks ABI runtime deps for a lot of packages). So, sometimes you have to rebuilt glibc with the updated gcc, and then maybe a few other packages.

@Gatak This is usually what the --changed-deps switch is for, -e world is often not needed (or: almost never). This was only needed back in the days when switching gcc-3 to gcc-4, or after some major glibc ABI changes.