Archive test fails on musl-libc. #671

Animeshz · 2022-02-17T14:39:22Z

Test run output:

2022-02-16T19:10:57.7728864Z  8/27 archive             FAIL            20.87s   killed by signal 6 SIGABRT
2022-02-16T19:10:57.7739690Z >>> ZIM_TEST_DATA_DIR=/builddir/libzim-7.2.0/build/test/data MALLOC_PERTURB_=97 /builddir/libzim-7.2.0/build/test/archive
2022-02-16T19:10:57.7743760Z ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
2022-02-16T19:10:57.7744064Z stdout:
2022-02-16T19:10:57.7744314Z Running main() from ../googletest/src/gtest_main.cc
2022-02-16T19:10:57.7744582Z [==========] Running 16 tests from 1 test suite.
2022-02-16T19:10:57.7745037Z [----------] Global test environment set-up.
2022-02-16T19:10:57.7745369Z [----------] 16 tests from ZimArchive
2022-02-16T19:10:57.7745700Z [ RUN      ] ZimArchive.openingAnInvalidZimArchiveFails
2022-02-16T19:10:57.7746088Z [       OK ] ZimArchive.openingAnInvalidZimArchiveFails (12 ms)
2022-02-16T19:10:57.7746462Z [ RUN      ] ZimArchive.openingAnEmptyZimArchiveSucceeds
2022-02-16T19:10:57.7746850Z [       OK ] ZimArchive.openingAnEmptyZimArchiveSucceeds (0 ms)
2022-02-16T19:10:57.7747450Z [ RUN      ] ZimArchive.nastyEmptyZimArchive
2022-02-16T19:10:57.7747789Z [       OK ] ZimArchive.nastyEmptyZimArchive (3 ms)
2022-02-16T19:10:57.7748135Z [ RUN      ] ZimArchive.wrongChecksumInEmptyZimArchive
2022-02-16T19:10:57.7748492Z [       OK ] ZimArchive.wrongChecksumInEmptyZimArchive (0 ms)
2022-02-16T19:10:57.7748855Z [ RUN      ] ZimArchive.openCreatedArchive
2022-02-16T19:10:57.7749083Z Resolve redirect
2022-02-16T19:10:57.7749361Z Invalid redirection C/foo4 redirecting to (missing) C/NoExistant
2022-02-16T19:10:57.7749621Z set index
2022-02-16T19:10:57.7749860Z [       OK ] ZimArchive.openCreatedArchive (16461 ms)
2022-02-16T19:10:57.7750164Z [ RUN      ] ZimArchive.openRealZimArchive
2022-02-16T19:10:57.7750451Z [       OK ] ZimArchive.openRealZimArchive (3913 ms)
2022-02-16T19:10:57.7750729Z [ RUN      ] ZimArchive.randomEntry
2022-02-16T19:10:57.7750982Z [       OK ] ZimArchive.randomEntry (11 ms)
2022-02-16T19:10:57.7751241Z [ RUN      ] ZimArchive.illustration
2022-02-16T19:10:57.7751508Z [       OK ] ZimArchive.illustration (0 ms)
2022-02-16T19:10:57.7751761Z [ RUN      ] ZimArchive.articleNumber
2022-02-16T19:10:57.7752030Z [       OK ] ZimArchive.articleNumber (11 ms)
2022-02-16T19:10:57.7752273Z [ RUN      ] ZimArchive.validate
2022-02-16T19:10:57.7752481Z stderr:
2022-02-16T19:10:57.7752996Z terminate called after throwing an instance of 'zim::ZimFileFormatError'
2022-02-16T19:10:57.7753305Z   what():  Invalid dirent pointer
2022-02-16T19:10:57.7753670Z ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
2022-02-16T19:10:57.7753831Z 
# ....
2022-02-16T19:10:59.4633936Z 
2022-02-16T19:10:59.4634239Z Summary of Failures:
2022-02-16T19:10:59.4634381Z 
2022-02-16T19:10:59.4634512Z  8/27 archive             FAIL            20.87s   killed by signal 6 SIGABRT
2022-02-16T19:10:59.4634670Z 
2022-02-16T19:10:59.4634674Z 
2022-02-16T19:10:59.4634747Z Ok:                 26  
2022-02-16T19:10:59.4634931Z Expected Fail:      0   
2022-02-16T19:10:59.4635121Z Fail:               1   
2022-02-16T19:10:59.4635302Z Unexpected Pass:    0   
2022-02-16T19:10:59.4635496Z Skipped:            0   
2022-02-16T19:10:59.4635662Z Timeout:            0   
2022-02-16T19:10:59.4635774Z 
2022-02-16T19:10:59.4636232Z Full log written to /builddir/libzim-7.2.0/build/meson-logs/testlog.txt
2022-02-16T19:10:59.5186384Z FAILED: meson-test 
2022-02-16T19:10:59.5186810Z /usr/bin/meson test --no-rebuild --print-errorlogs
2022-02-16T19:10:59.5187109Z ninja: build stopped: subcommand failed.
2022-02-16T19:10:59.5196359Z => ERROR: libzim-7.2.0_1: do_check: '${make_cmd} -C ${meson_builddir} ${makejobs} ${make_check_args} ${make_check_target}' exited with 1

Affected file: test/archive.cpp

Animeshz · 2022-02-18T06:57:09Z

More info from gdb

[animesh@/home/animesh/Projects/void-packages/masterdir-x86_64-musl /]$ ZIM_TEST_DATA_DIR=/builddir/libzim-7.2.0/build/test/data gdb -ex run -args /builddir/libzim-7.2.0/build/test/archive
# ... Some OK lines ...
[ RUN      ] ZimArchive.openCreatedArchive
[New LWP 23105]
[New LWP 23106]
[New LWP 23107]
[New LWP 23108]
[New LWP 23109]
Resolve redirect
Invalid redirection C/foo4 redirecting to (missing) C/NoExistant
set index
[LWP 23107 exited]
[LWP 23105 exited]
[LWP 23106 exited]
[LWP 23108 exited]
[LWP 23109 exited]
# ... Some OK lines ...
[ RUN      ] ZimArchive.validate
terminate called after throwing an instance of 'zim::ZimFileFormatError'
  what():  Invalid dirent pointer

Thread 1 "archive" received signal SIGABRT, Aborted.
__restore_sigs (set=set@entry=0x7fffffffde20) at ./arch/x86_64/syscall_arch.h:40
40		return ret;

(gdb) bt
#0  __restore_sigs (set=set@entry=0x7fffffffde20) at ./arch/x86_64/syscall_arch.h:40
#1  0x00007ffff7fa5620 in raise (sig=sig@entry=6) at src/signal/raise.c:11
#2  0x00007ffff7f6de20 in abort () at src/exit/abort.c:13
#3  0x00007ffff7c655eb in ?? () from /usr/lib64/libstdc++.so.6
#4  0x00007ffff7c929a8 in __cxxabiv1::__terminate(void (*)()) () from /usr/lib64/libstdc++.so.6
#5  0x00007ffff7c92a11 in std::terminate() () from /usr/lib64/libstdc++.so.6
#6  0x00007ffff7c92ca4 in __cxa_throw () from /usr/lib64/libstdc++.so.6
#7  0x00007ffff7eca565 in zim::DirentReader::readDirent(zim::offset_t) [clone .cold] () from /builddir/libzim-7.2.0/build/test/../src/libzim.so.7
#8  0x00007ffff7ed8211 in zim::DirectDirentAccessor::readDirent(zim::offset_t) const () from /builddir/libzim-7.2.0/build/test/../src/libzim.so.7
#9  0x00007ffff7ed82d1 in zim::DirectDirentAccessor::getDirent(zim::entry_index_t) const () from /builddir/libzim-7.2.0/build/test/../src/libzim.so.7
#10 0x00007ffff7ee01b5 in zim::DirentLookup<zim::DirectDirentAccessor>::getDirentKey[abi:cxx11](unsigned int) const ()
   from /builddir/libzim-7.2.0/build/test/../src/libzim.so.7
#11 0x00007ffff7ee1f4a in std::call_once<zim::FileImpl::direntLookup() const::{lambda()#1}>(std::once_flag&, zim::FileImpl::direntLookup() const::{lambda()#1}&&)::{lambda()#2}::_FUN() [clone .lto_priv.0] () from /builddir/libzim-7.2.0/build/test/../src/libzim.so.7
#12 0x00007ffff7fb7f28 in __pthread_once_full (init=0x7ffff7cbc580 <__once_proxy>, control=0x5555555b5810) at src/thread/pthread_once.c:22
#13 __pthread_once_full (control=0x5555555b5810, init=0x7ffff7cbc580 <__once_proxy>) at src/thread/pthread_once.c:11
#14 0x00007ffff7ed558c in zim::FileImpl::direntLookup() const () from /builddir/libzim-7.2.0/build/test/../src/libzim.so.7
#15 0x00007ffff7ed8abf in zim::FileImpl::getTitleAccessor(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
   from /builddir/libzim-7.2.0/build/test/../src/libzim.so.7
#16 0x00007ffff7eda888 in zim::FileImpl::FileImpl(std::shared_ptr<zim::FileCompound>, zim::offset_t, zim::zsize_t) ()
   from /builddir/libzim-7.2.0/build/test/../src/libzim.so.7
#17 0x00007ffff7edad6b in zim::FileImpl::FileImpl(std::shared_ptr<zim::FileCompound>) () from /builddir/libzim-7.2.0/build/test/../src/libzim.so.7
#18 0x00007ffff7edae41 in zim::FileImpl::FileImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
   from /builddir/libzim-7.2.0/build/test/../src/libzim.so.7
#19 0x00007ffff7edaed2 in zim::Archive::Archive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
   from /builddir/libzim-7.2.0/build/test/../src/libzim.so.7
#20 0x00007ffff7edaf4d in zim::validate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::bitset<6ul>) ()
   from /builddir/libzim-7.2.0/build/test/../src/libzim.so.7
#21 0x0000555555566890 in (anonymous namespace)::ZimArchive_validate_Test::TestBody() [clone .lto_priv.0] ()
#22 0x00007ffff7e8bba7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) () from /usr/lib64/libgtest.so.1.11.0
#23 0x00007ffff7e804fe in testing::Test::Run() () from /usr/lib64/libgtest.so.1.11.0
#24 0x00007ffff7e80675 in testing::TestInfo::Run() () from /usr/lib64/libgtest.so.1.11.0
#25 0x00007ffff7e80b19 in testing::TestSuite::Run() () from /usr/lib64/libgtest.so.1.11.0
#26 0x00007ffff7e8124a in testing::internal::UnitTestImpl::RunAllTests() () from /usr/lib64/libgtest.so.1.11.0
#27 0x00007ffff7e8c117 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) () from /usr/lib64/libgtest.so.1.11.0
#28 0x00007ffff7e80738 in testing::UnitTest::Run() () from /usr/lib64/libgtest.so.1.11.0
#29 0x00007ffff7ea60db in main () from /usr/lib64/libgtest_main.so.1.11.0
#30 0x00007ffff7f6d70a in libc_start_main_stage2 (main=0x7ffff7ea60a0 <main>, argc=1, argv=0x7fffffffec48) at src/env/__libc_start_main.c:94
#31 0x0000555555560d19 in _start ()

Referencing musl/arch/x86_64/syscall_arch.h#L40 from src/dirent.cpp#L121 (at #7) originating from test/archive.cpp#L376 (at #20).

Digging in, offset.v is found to be 41165 and totalSize.v is 41155 in dirent.cpp#L121

And failure starts by bug possibly present here src/writer/creator.cpp.

Animeshz · 2022-02-18T11:41:26Z

May I know more about the NoExistant redirection, so that I can help further?

libzim/test/archive.cpp

Line 168 in 67eaa57

    
           creator.addRedirection("foo4", "FooRedirection", "NoExistant"); // Invalid redirection, must be removed by creator

kelson42 · 2022-02-18T11:42:07Z

@Animeshz Like you see, we don't have tested against musl... but we better should. Would be great to have a CI entry for Github actions able to do so.

kelson42 · 2022-02-19T10:41:02Z

@Animeshz This is a question better @veloman-yunkan or @mgautierfr should answer.

veloman-yunkan · 2022-02-19T16:00:32Z

May I know more about the NoExistant redirection, so that I can help further?

libzim/test/archive.cpp

Line 168 in 67eaa57

creator.addRedirection("foo4", "FooRedirection", "NoExistant"); // Invalid redirection, must be removed by creator

The unit-test intentionally adds a redirection to a non-existent entry. During ZIM creation such invalid redirections must be dropped. The unit-test checks that it works as intended. It turns out that with a musl build of libzim it doesn't (provided that your investigation so far has gone in the correct direction)

veloman-yunkan · 2022-06-30T09:23:40Z

The crash is caused by a bug in musl which doesn't correctly handle exceptional executions of std::call_once, as demonstrated by the following test program:

#include <thread>
#include <mutex>
#include <iostream>
#include <stdexcept>

void foo(int i) {
    std::cout << "foo(" << i << "): prethrow" << std::endl;
    throw std::runtime_error("Oops!");
    std::cout << "foo(" << i << "): postthrow" << std::endl;
}

int main() {
    std::once_flag once;
    for ( int i = 1; i < 5; ++i ) {
        std::cout << "loop: " << i << std::endl;
        try {
            std::call_once(once, [=]() { foo(i); });
        } catch(const std::runtime_error& ex) {
            std::cout << "catch: "  << ex.what() << std::endl;
        }
    }
    return 0;
}

Under alpine:

# g++ -std=c++11 -pthread call_once_test.cpp -o call_once_test && ./call_once_test
loop: 1
foo(1): prethrow
terminate called after throwing an instance of 'std::runtime_error'
  what():  Oops!
Aborted (core dumped)

Under Ubuntu the same test program uncovers another bug with the handling of exceptional executions of std::call_once:

$ g++ -std=c++11 -pthread call_once_test.cpp -o call_once_test && ./call_once_test 
loop: 1
foo(1): prethrow
catch: Oops!
loop: 2
^C       # the program hangs here (most likely because the mutex protecting the `std::call_once` call
         # wasn't unlocked after the first exceptional execution)

kelson42 added bug question labels Feb 17, 2022

kelson42 added this to the 7.3.0 milestone Feb 17, 2022

mgautierfr self-assigned this Feb 23, 2022

Animeshz mentioned this issue Mar 5, 2022

Update kiwix-lib-10.0.1 kiwix-tools-3.2.0 libzim-7.2.0 zim-tools-3.1.0 void-linux/void-packages#35607

Closed

kelson42 added the compilation label May 27, 2022

kelson42 mentioned this issue Jun 28, 2022

Tests are failing on AlpineLinux kiwix/libkiwix#792

Closed

veloman-yunkan assigned veloman-yunkan and unassigned mgautierfr Jun 29, 2022

This was referenced Jun 30, 2022

Got rid of std::call_once usage #708

Merged

CI for Alpine #710

Merged

mgautierfr closed this as completed in #708 Jul 19, 2022

kelson42 modified the milestones: 7.4.0, 7.3.0 Jul 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Archive test fails on musl-libc. #671

Archive test fails on musl-libc. #671

Animeshz commented Feb 17, 2022

Animeshz commented Feb 18, 2022 •

edited

Animeshz commented Feb 18, 2022

kelson42 commented Feb 18, 2022 •

edited

kelson42 commented Feb 19, 2022

veloman-yunkan commented Feb 19, 2022

veloman-yunkan commented Jun 30, 2022

Archive test fails on musl-libc. #671

Archive test fails on musl-libc. #671

Comments

Animeshz commented Feb 17, 2022

Animeshz commented Feb 18, 2022 • edited

Animeshz commented Feb 18, 2022

kelson42 commented Feb 18, 2022 • edited

kelson42 commented Feb 19, 2022

veloman-yunkan commented Feb 19, 2022

veloman-yunkan commented Jun 30, 2022

Animeshz commented Feb 18, 2022 •

edited

kelson42 commented Feb 18, 2022 •

edited