Buildbot stacktraces

Buildbot saves the server's log for each test suite, accessible at a URL of the form http://localhost:8010/builders/continuous_test-linux-c-dbg/builds/359/steps/js-auth/logs/server

Consider the following snippet of log file from that build:

Wed Jan 29 04:39:16.158 [conn786] [tokumx] TokuMX: The largest key allowed is 32768 bytes
Wed Jan 29 04:39:16.158 [conn786] Assertion: 16770:Error 22 (Invalid argument) from the ydb layer. You may have hit a bug. Check the error log for more details.
0x135fecf 0x132f98c 0xc1f5d0 0xc1f789 0xc1c1fc 0xd95d13 0xd959ce 0xd3585a 0xd350b3 0xccdbe5 0xcd6eef 0xcd9aa0 0xcb53c4 0xd5ca1e 0xd071a3 0xd07a64 0xd03df6 0xaf7042 0xc3366d 0x7f442f7d7e9a 
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo15printStackTraceERSo+0x27) [0x135fecf]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo10logContextEPKc+0x5e) [0x132f98c]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo7storage21MsgAssertionExceptionC1EiRKSs+0x10a) [0xc1f5d0]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo7storage15SystemExceptionC2EiiRKSs+0x41) [0xc1f789]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo7storage16handle_ydb_errorEi+0x1e3) [0xc1c1fc]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo7storage10Dictionary4openERKNS_10DescriptorEbb+0x1a1) [0xd95d13]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo7storage10DictionaryC2ERKSsRKNS_7BSONObjERKNS_10DescriptorEbb+0xd6) [0xd959ce]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo16IndexDetailsBase4openEb+0x15c) [0xd3585a]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo16IndexDetailsBase4makeERKNS_7BSONObjEb+0x1b5) [0xd350b3]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo14CollectionBase11IndexerBase7prepareEv+0x70d) [0xccdbe5]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo14CollectionBase11createIndexERKNS_7BSONObjE+0x103) [0xcd6eef]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo14CollectionBase11ensureIndexERKNS_7BSONObjE+0xb6) [0xcd9aa0]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo10Collection11ensureIndexERKNS_7BSONObjE+0x38) [0xcb53c4]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo13insertObjectsEPKcRKSt6vectorINS_7BSONObjESaIS3_EEbmb+0x40d) [0xd5ca1e]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod() [0xd071a3]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x7e7) [0xd07a64]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x6f8) [0xd03df6]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xdc) [0xaf7042]
 /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x4de) [0xc3366d]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f442f7d7e9a]

To get a full stacktrace with line numbers out of this, use the following steps:

1. Go to the logfile's corresponding build page (in this example it would be http://lex1:8010/builders/continuous_test-linux-c-dbg/builds/359/) and take note of the first 7 digits of the mongo and ft-index commit hashes (68d4f7a and e97ae3b respectively, for this example) and whether the enterprise and debugbuild flags are True.

2. ssh to lex1 and look for the tarball in /home.tokubuild/buildmaster/builds. A community tarball is of the form tokumx-<mongo commit hash>-ft-<ft-index commit hash>-linux-x86_64-main.tar.gz, and an enterprise tarball is tokumx--ft--ba--bp--linux-x86_64-main.tar.gz`.

3. Note the binary name mentioned in the stack trace. It is usually mongod but can be mongo or mongos. It's the first part of each line in the stack trace:

    /data/buildslave/continuous_test-linux-c-dbg/build/mongod(_ZN5mongo16IndexDetailsBase4openEb+0x15c) [0xd3585a]
   

4. You will need the binary and the debuginfo (in this case mongod and mongod.dbg) from the tarball and its debuginfo tarball (if the normal tarball is tokumx-blahblah-main.tar.gz, the debuginfo tarball is tokumx-blahblah-debuginfo.tar.gz). Extract both files with commands like these:

   $ tar xzf /home.tokubuild/buildmaster/builds/tokumx-68d4f7a-ft-e97ae3b-linux-x86_64-main.tar.gz tokumx-68d4f7a-ft-e97ae3b-linux-x86_64/bin/mongod
   $ tar xzf /home.tokubuild/buildmaster/builds/tokumx-68d4f7a-ft-e97ae3b-linux-x86_64-debuginfo.tar.gz tokumx-68d4f7a-ft-e97ae3b-linux-x86_64/bin/mongod.dbg

5. Use addr2line on the binary you just extracted with the hex numbers from the first line of the stacktrace. I like the options -Cfpi for better readability.

   $ addr2line -Cfpi -e tokumx-68d4f7a-ft-e97ae3b-linux-x86_64/bin/mongod 0x135fecf 0x132f98c 0xc1f5d0 0xc1f789 0xc1c1fc 0xd95d13 0xd959ce 0xd3585a 0xd350b3 0xccdbe5 0xcd6eef 0xcd9aa0 0xcb53c4 0xd5ca1e 0xd071a3 0xd07a64 0xd03df6 0xaf7042 0xc3366d 0x7f442f7d7e9a 
   ??
   ??:0
   ??
   ??:0
   v8::internal::LCodeGen::DoPower(v8::internal::LPower*) at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/v8/src/x64/lithium-codegen-x64.cc:3218
   FieldOperand at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/v8/src/x64/macro-assembler-x64.h:1433
    (inlined by) v8::internal::LCodeGen::DoRandom(v8::internal::LRandom*) at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/v8/src/x64/lithium-codegen-x64.cc:3272
   v8::internal::List<v8::internal::LDeferredCode*, v8::internal::ZoneAllocationPolicy>::ResizeAddInternal(v8::internal::LDeferredCode* const&, v8::internal::ZoneAllocationPolicy) at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/v8/src/list-inl.h:82
    (inlined by) v8::internal::List<v8::internal::LDeferredCode*, v8::internal::ZoneAllocationPolicy>::ResizeAdd(v8::internal::LDeferredCode* const&, v8::internal::ZoneAllocationPolicy) at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/v8/src/list-inl.h:68
    (inlined by) v8::internal::List<v8::internal::LDeferredCode*, v8::internal::ZoneAllocationPolicy>::Add(v8::internal::LDeferredCode* const&, v8::internal::ZoneAllocationPolicy) at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/v8/src/list-inl.h:42
    (inlined by) v8::internal::ZoneList<v8::internal::LDeferredCode*>::Add(v8::internal::LDeferredCode* const&, v8::internal::Zone*) at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/v8/src/zone.h:202
    (inlined by) v8::internal::LCodeGen::AddDeferredCode(v8::internal::LDeferredCode*) at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/v8/src/x64/lithium-codegen-x64.h:162
    (inlined by) LDeferredCode at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/v8/src/x64/lithium-codegen-x64.h:387
    (inlined by) DeferredInstanceOfKnownGlobal at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/v8/src/x64/lithium-codegen-x64.cc:2020
    (inlined by) v8::internal::LCodeGen::DoInstanceOfKnownGlobal(v8::internal::LInstanceOfKnownGlobal*) at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/v8/src/x64/lithium-codegen-x64.cc:2033
   mongo::ChunkManager::loadExistingRanges(std::string const&) at /home/buildslave/buildslave-3/tarball-linux/build/src/mongo/s/chunk.cpp:652
   mongo::ChunkManager::_load(std::string const&, std::map<mongo::BSONObj, boost::shared_ptr<mongo::Chunk const>, mongo::BSONObjCmp, std::allocator<std::pair<mongo::BSONObj const, boost::shared_ptr<mongo::Chunk const> > > >&, std::set<mongo::Shard, std::less<mongo::Shard>, std::allocator<mongo::Shard> >&, std::map<mongo::Shard, mongo::ChunkVersion, std::less<mongo::Shard>, std::allocator<std::pair<mongo::Shard const, mongo::ChunkVersion> > >&, boost::shared_ptr<mongo::ChunkManager const>) at /home/buildslave/buildslave-3/tarball-linux/build/src/mongo/s/chunk.cpp:777
   boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::program_options::multiple_values> >::~clone_impl() at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/boost/boost/exception/exception.hpp:426
   boost::any::holder<std::string>::~holder() at ??:0
   pcrecpp::RE::GlobalReplace(pcrecpp::StringPiece const&, std::string*) const at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/pcre-8.30/pcrecpp.cc:402
   compile_branch at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/pcre-8.30/pcre_compile.c:4389
    (inlined by) compile_regex at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/pcre-8.30/pcre_compile.c:7068
   pcre_compile2 at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/pcre-8.30/pcre_compile.c:7709
   std::string::_Rep::_M_dispose(std::allocator<char> const&) at /usr/local/gcc-4.7/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/../../../../include/c++/4.7.1/bits/basic_string.h:242
    (inlined by) ~basic_string at /usr/local/gcc-4.7/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/../../../../include/c++/4.7.1/bits/basic_string.h:536
    (inlined by) std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::str() const at /usr/local/gcc-4.7/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/../../../../include/c++/4.7.1/sstream:131
    (inlined by) std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::str() const at /usr/local/gcc-4.7/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/../../../../include/c++/4.7.1/sstream:565
    (inlined by) mongo::hexdump(char const*, unsigned int) at /home/buildslave/buildslave-3/tarball-linux/build/src/mongo/util/util.cpp:41
   mongo::BSONField<mongo::OID>::operator()() const at /home/buildslave/buildslave-3/tarball-linux/build/src/mongo/../mongo/bson/bson_field.h:104
    (inlined by) mongo::DistributedLock::lock_try(std::string const&, bool, mongo::BSONObj*, double) at /home/buildslave/buildslave-3/tarball-linux/build/src/mongo/client/distlock.cpp:846
   mongo::LoggingManager::rotate() at /home/buildslave/buildslave-3/tarball-linux/build/src/mongo/util/log.cpp:153
   std::string::_Rep::_M_dispose(std::allocator<char> const&) at /usr/local/gcc-4.7/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/../../../../include/c++/4.7.1/bits/basic_string.h:242
    (inlined by) ~basic_string at /usr/local/gcc-4.7/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/../../../../include/c++/4.7.1/bits/basic_string.h:536
    (inlined by) mongo::LoggingManager::rotate() at /home/buildslave/buildslave-3/tarball-linux/build/src/mongo/util/log.cpp:163
   mongo::StringBuilderImpl<mongo::TrivialAllocator>::operator<<(mongo::StringData const&) at /home/buildslave/buildslave-3/tarball-linux/build/src/mongo/../mongo/bson/util/builder.h:318
    (inlined by) operator<< <char [8]> at /home/buildslave/buildslave-3/tarball-linux/build/src/mongo/../mongo/util/mongoutils/str.h:53
    (inlined by) mongo::File::read(unsigned long, char*, unsigned int) at /home/buildslave/buildslave-3/tarball-linux/build/src/mongo/util/file.cpp:251
   v8::internal::Map::NumberOfProtoTransitions() at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/v8/src/objects.h:4898
    (inlined by) v8::internal::MarkCompactCollector::ClearNonLivePrototypeTransitions(v8::internal::Map*) at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/v8/src/mark-compact.cc:2631
   GetHoleyElementsKind at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/v8/src/elements-kind.h:167
    (inlined by) IsSimpleMapChangeTransition at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/v8/src/elements-kind.h:190
    (inlined by) v8::internal::LChunkBuilder::DoTransitionElementsKind(v8::internal::HTransitionElementsKind*) at /home/buildslave/buildslave-3/tarball-linux/build/src/third_party/v8/src/x64/lithium-x64.cc:1971
   ??
   ??:0
   

If your stacktrace contains frames from one of the toku libs (e.g. libtokufractaltree.so), then instead, just extract the lib64 directory from the non-debuginfo tarball, and run addr2line on the hex offsets from each line of the stacktrace just after the library name in parentheses. For example, with the stacktrace

Tue Jan 28 09:36:57.516 Backtrace:
0x1373f83 0xafe47c 0x7f991f87e4a0 0x7f991f87e425 0x7f991f881b8b 0x13403de 0xc28b45 0x7f99208b9de2 0x7f99208ba691 0x7f99208bc001 0x7f99208bbe46 0x7f99208bc51d 0x7f99208bd18b 0x7f99208bee37 0x7f99208bfc1e 0x7f99208bfe61 0x7f99208e65ce 0x7f99208e7dba 0x7f99208e8566 0x7f9920845694 
 /data/buildslave/ctest-linux-c-dbg/build/dbg/src/mongo/mongod(_ZN5mongo15printStackTraceERSo+0x27) [0x1373f83]
 /data/buildslave/ctest-linux-c-dbg/build/dbg/src/mongo/mongod(_ZN5mongo10abruptQuitEi+0x119) [0xafe47c]
 /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f991f87e4a0]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f991f87e425]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f991f881b8b]
 /data/buildslave/ctest-linux-c-dbg/build/dbg/src/mongo/mongod(_ZN5mongo9uassertedEiRKSs+0) [0x13403de]
 /data/buildslave/ctest-linux-c-dbg/build/dbg/src/mongo/mongod() [0xc28b45]
 /data/buildslave/ctest-linux-c-dbg/build/dbg/src/third_party/ft-index/src/libtokufractaltree.so(+0x7bde2) [0x7f99208b9de2]
 /data/buildslave/ctest-linux-c-dbg/build/dbg/src/third_party/ft-index/src/libtokufractaltree.so(+0x7c691) [0x7f99208ba691]
 /data/buildslave/ctest-linux-c-dbg/build/dbg/src/third_party/ft-index/src/libtokufractaltree.so(+0x7e001) [0x7f99208bc001]
 /data/buildslave/ctest-linux-c-dbg/build/dbg/src/third_party/ft-index/src/libtokufractaltree.so(+0x7de46) [0x7f99208bbe46]
 /data/buildslave/ctest-linux-c-dbg/build/dbg/src/third_party/ft-index/src/libtokufractaltree.so(+0x7e51d) [0x7f99208bc51d]
 /data/buildslave/ctest-linux-c-dbg/build/dbg/src/third_party/ft-index/src/libtokufractaltree.so(+0x7f18b) [0x7f99208bd18b]
 /data/buildslave/ctest-linux-c-dbg/build/dbg/src/third_party/ft-index/src/libtokufractaltree.so(+0x80e37) [0x7f99208bee37]
 /data/buildslave/ctest-linux-c-dbg/build/dbg/src/third_party/ft-index/src/libtokufractaltree.so(+0x81c1e) [0x7f99208bfc1e]
 /data/buildslave/ctest-linux-c-dbg/build/dbg/src/third_party/ft-index/src/libtokufractaltree.so(+0x81e61) [0x7f99208bfe61]
 /data/buildslave/ctest-linux-c-dbg/build/dbg/src/third_party/ft-index/src/libtokufractaltree.so(+0xa85ce) [0x7f99208e65ce]
 /data/buildslave/ctest-linux-c-dbg/build/dbg/src/third_party/ft-index/src/libtokufractaltree.so(+0xa9dba) [0x7f99208e7dba]
 /data/buildslave/ctest-linux-c-dbg/build/dbg/src/third_party/ft-index/src/libtokufractaltree.so(+0xaa566) [0x7f99208e8566]
 /data/buildslave/ctest-linux-c-dbg/build/dbg/src/third_party/ft-index/src/libtokufractaltree.so(+0x7694) [0x7f9920845694]

you would extract lib64/libtokufractaltree.so from the "main" tarball, and run

$ addr2line -Cfpi -e lib64/libtokufractaltree.so 0x7bde2 0x7c691 0x7e001 0x7de46 0x7e51d 0x7f18b 0x80e37 0x81c1e 0x81e61 0xa85ce 0xa9dba 0xaa566 0x7694