Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: repeat_unit_test.sh don't core dump to stdout #7001

Open
jaki opened this issue Jan 27, 2021 · 0 comments
Open

test: repeat_unit_test.sh don't core dump to stdout #7001

jaki opened this issue Jan 27, 2021 · 0 comments
Assignees
Labels
kind/failing-test Tests and testing infra

Comments

@jaki
Copy link
Contributor

jaki commented Jan 27, 2021

When using --num_repetitions of yb_build.sh for running tests multiple times, it produces a nice, compact output of fail/pass to stdout:

PASSED: iteration 18, 125 sec                                                                                                                                                                                                                                
PASSED: iteration 19, 125 sec                                                                                                                                                                                                                                
[2021-01-26T14:41:54 common-test-env.sh:1150 did_test_succeed] Test failure reason: exit code: 1                                                                                                                                                             
FAILED: iteration 20, 135 sec; test log path: /nfusr/dev-server/jason/logs/repeat_unit_test/pg_index_backfill-test/PgIndexBackfillTest.CreateUniqueIndexWriteAfterSafeTime/2021-01-26T13_57_54/20.log.gz                                                     
PASSED: iteration 21, 133 sec                                                                                                                                                                                                                                
[2021-01-26T14:46:24 common-test-env.sh:1150 did_test_succeed] Test failure reason: exit code: 1                                                                                                                                                             
FAILED: iteration 22, 135 sec; test log path: /nfusr/dev-server/jason/logs/repeat_unit_test/pg_index_backfill-test/PgIndexBackfillTest.CreateUniqueIndexWriteAfterSafeTime/2021-01-26T13_57_54/22.log.gz                                                     
PASSED: iteration 23, 122 sec                                                                                                                                                                                                                                
PASSED: iteration 24, 136 sec

Recently, I saw a core dump that ended up getting spit out:

PASSED: iteration 5, 228 sec                                                                                                                                                                                                                                 
PASSED: iteration 6, 221 sec                                                                                                                                                                                                                                 
[2021-01-26T16:45:44 common-test-env.sh:1150 did_test_succeed] Test failure reason: exit code: 1                                                                                                                                                             
Found a core file at '/tmp/yb_tests__2021-01-26T16_35_41__11657.32444.28901/core.23188', backtrace:                                                                                                                                                          
+ echo ''                                                                                                                                                                                                                                                    
+ gdb -q -n -ex bt -ex 'thread apply all bt' -batch /nfusr/dev-server/jason/code/yugabyte-db/2/build/asan-clang-dynamic-ninja/tests-pgwrapper/pg_index_backfill-test /tmp/yb_tests__2021-01-26T16_35_41__11657.32444.28901/core.23188                                                     
+ grep -Ev '^\[New LWP [0-9]+\]$'                                                                                             
+ /nfusr/dev-server/jason/code/yugabyte-db/2/build-support/dedup_thread_stacks.py                                             
+ tee -a /nfusr/dev-server/jason/logs/repeat_unit_test/pg_index_backfill-test/PgIndexBackfillTest.MasterLeaderStepdown/2021-01-26T16_18_18/7.log                                                                                                                                         
                                                                                                                                                              
warning: core file may not match specified executable file.                                                                   
                                                                                                                                            
warning: File "/opt/yb-build/brew/linuxbrew-20181203T161736v9-3ba4c2ed9b0587040949a4a9a95b576f520bae/Cellar/glibc/2.23/lib/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".                  
To enable execution of this file add                                                                                                                          
        add-auto-load-safe-path /opt/yb-build/brew/linuxbrew-20181203T161736v9-3ba4c2ed9b0587040949a4a9a95b576f520bae/Cellar/glibc/2.23/lib/libthread_db-1.0.so                                                                                                                           
line to your configuration file "/nfusr/dev-server/jason/.gdbinit".                                                                                                                                                                                          
To completely disable this security protection add                                                                            
        set auto-load safe-path /                                                                                                                             
line to your configuration file "/nfusr/dev-server/jason/.gdbinit".                                                           
For more information about this security protection see the                                                                   
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:                                                                                
        info "(gdb)Auto-loading safe path"                                                                                                  
                                                                                                                              
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.                            
                                                                                                                              
warning: File "/opt/yb-build/brew/linuxbrew-20181203T161736v9-3ba4c2ed9b0587040949a4a9a95b576f520bae/Cellar/glibc/2.23/lib/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".                  
                                                                                                                                                                                                                                                             
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.              
Core was generated by `/nfusr/dev-server/jason/code/yugabyte-db/2/build/asan-clang-dynamic-ninja/tests'.                                    
Program terminated with signal 3, Quit.                                                                                       
#0  0x00007ff51e1c146d in nanosleep () at ../sysdeps/unix/syscall-template.S:84                                 
84      ../sysdeps/unix/syscall-template.S: No such file or directory.                                                                      
#0  0x00007ff51e1c146d in nanosleep () at ../sysdeps/unix/syscall-template.S:84                                                                               
#1  0x00007ff51ebe904b in std::__1::this_thread::sleep_for(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > const&) () from /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201207195011-bbf28cf4f1-centos7-linuxbrew/installed/asan/libcxx/lib/libc++.so.1    
#2  0x00007ff545baf88f in std::__1::this_thread::sleep_for<long long, std::__1::ratio<1l, 1000l> > (__d=...) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201207195011-bbf28cf4f1-centos7-linuxbrew/installed/asan/libcxx/include/c++/v1/thread:448

This core dump ends up in the log file, so there's no benefit of printing to stdout. It ruins the nice compact summary of pass/fail.

@jaki jaki added the kind/failing-test Tests and testing infra label Jan 27, 2021
@jaki jaki self-assigned this Jan 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/failing-test Tests and testing infra
Projects
None yet
Development

No branches or pull requests

1 participant