Fixes #175 Do not allow to overwrite existing .zim file

[asashnov]

Fixes #175 Do not allow to overwrite existing .zim file

[codecov]

Codecov Report

Merging #193 (6268217) into master (b1b6828) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #193   +/-   ##
=======================================
  Coverage   33.83%   33.83%           
=======================================
  Files          20       20           
  Lines        2007     2007           
  Branches     1166     1166           
=======================================
  Hits          679      679           
  Misses       1326     1326           
  Partials        2        2           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b1b6828...6268217. Read the comment docs.

[kelson42]

@veloman-yunkan @mgautierfr The CI fails here, but no clue why... I kind of suspect a problem with the latest libzim

[veloman-yunkan]

@kelson42 I tried to reproduce (and then debug) the failure locally with the latest libzim without success. On the CI side the crash appears to be reproducible since re-running the CI jobs doesn't help.

[kelson42]

It seems to be pretty much related to the compilation options. I have tested as well, but probably not with the exact same constellation of options.

[veloman-yunkan]

@kelson42 The crash in CI/Linux (native_static) may be explained by https://stackoverflow.com/questions/35116327/when-g-static-link-pthread-cause-segmentation-fault-why

[veloman-yunkan]

I downloaded the deps archive from http://tmp.kiwix.org/ci/deps2_xenial_native_static_zim-tools.tar.xz, used it to build zim-tools and reproduced the crash. Going to debug it now.

[veloman-yunkan]

Looks like http://tmp.kiwix.org/ci/deps2_xenial_native_static_zim-tools.tar.xz (at the time of download) was built on Ubuntu 16.04 xenial (as suggested by the version of gcc/c++ in the stack trace): UPDATE: this is explicitly encoded in the file name too.

Thread 1 "zimwriterfs-zim" received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x00000000004d782f in __gthread_equal (__t1=0, __t2=0) at /usr/include/x86_64-linux-gnu/c++/5/bits/gthr-default.h:680
#2 0x00000000004dd105 in std::operator== (__x=..., __y=...) at /usr/include/c++/5/thread:84
#3 0x00000000004dd200 in std::thread::joinable (this=0x7fffffffd740) at /usr/include/c++/5/thread:170
#4 0x00000000004dd156 in std::thread::~thread (this=0x7fffffffd740, __in_chrg=<optimized out>)
at /usr/include/c++/5/thread:150
#5 0x00000000004d7be6 in zim::writer::Creator::startZimCreation (this=0x7fffffffd920, fname=...)
at ../../SOURCE/libzim/src/writer/creator.cpp:91

zim-tools CI/Linux (native_static) runs on ubuntu-latest which is Ubuntu 18.04 bionic and uses gcc version 7. Thus we arrive at an executable compiled by using headers from gcc v7 and linked with libraries compiled by gcc v5. The fact that it crashes is not surprising. in a xenial container (thus no discrepancy).

[veloman-yunkan]

@kelson42 @mgautierfr BTW, ubuntu-latest in GitHub Actions workflows will gradually change from bionic to focal starting on November 30: https://github.blog/changelog/2020-10-29-github-actions-ubuntu-latest-workflows-will-use-ubuntu-20-04/

[kelson42]

@veloman-yunkan Thank you very much for the investigation
@mgautierfr This looks like a bug in the CI, would you be able to fix it please? We should not use lastest but always a specific version.

[veloman-yunkan]

@kelson42 Sorry, I missed that CI runs in a docker container. Though the build machine was a bionic one, the build actually was performed in a xenial container, so my conclusion is bogus.

[veloman-yunkan]

One mismatch between the kiwix-build CI which produces the deps archives and the zim-tools CI which consumes those archives is that the former uses container image kiwix/kiwix-build_ci:xenial-30 while the latter uses container image kiwix/kiwix-build_ci:xenial-26.

[kelson42]

FYI https://github.blog/changelog/2020-10-29-github-actions-ubuntu-latest-workflows-will-use-ubuntu-20-04/

[veloman-yunkan]

Debugging the crash in CI resulted in the same stack trace that I observed locally after building zim-tools with gcc 9.3 on my Ubuntu 20.04 focal using the deps downloaded from http://tmp.kiwix.org/ci/deps2_xenial_native_static_zim-tools.tar.xz (that were compiled with gcc 5.4 under Ubuntu 16.04 xenial):

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ../build/test/zimwriterfs-zimcreatorfs...done.
warning: Error disabling address space randomization: Operation not permitted
Starting program: /home/runner/zim-tools/build/test/zimwriterfs-zimcreatorfs 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Running main() from gtest_main.cc
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 3 tests from ZimCreatorFSTest
[ RUN      ] ZimCreatorFSTest.MinimalZim
[New Thread 0x7fd00888b700 (LWP 1169)]

Thread 1 "zimwriterfs-zim" received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
#0  0x0000000000000000 in ?? ()
#1  0x000000000052cb0f in __gthread_equal (__t1=0, __t2=0)
    at /usr/include/x86_64-linux-gnu/c++/5/bits/gthr-default.h:680
#2  0x00000000005323e5 in std::operator== (__x=..., __y=...)
    at /usr/include/c++/5/thread:84
#3  0x00000000005324e0 in std::thread::joinable (this=0x7ffcc58622f0)
    at /usr/include/c++/5/thread:170
#4  0x0000000000532436 in std::thread::~thread (this=0x7ffcc58622f0, 
    __in_chrg=<optimized out>) at /usr/include/c++/5/thread:150
#5  0x000000000052cec6 in zim::writer::Creator::startZimCreation (
    this=0x7ffcc58625f0, fname=...)
    at ../../SOURCE/libzim/src/writer/creator.cpp:91
#6  0x0000000000409e59 in ZimCreatorFSTest_MinimalZim_Test::TestBody (
    this=0x36f60a0) at ../test/zimwriterfs-zimcreatorfs.cpp:75
#7  0x00000000004dcd86 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x36f60a0, 
    method=&virtual testing::Test::TestBody(), 
    location=0xb35a9b "the test body") at /usr/src/gtest/src/gtest.cc:2078
#8  0x00000000004d35a9 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x36f60a0, 
    method=&virtual testing::Test::TestBody(), 
    location=0xb35a9b "the test body") at /usr/src/gtest/src/gtest.cc:2114
#9  0x000000000049abb2 in testing::Test::Run (this=0x36f60a0)
    at /usr/src/gtest/src/gtest.cc:2151
#10 0x000000000049be50 in testing::TestInfo::Run (this=0x36b7990)
    at /usr/src/gtest/src/gtest.cc:2326
#11 0x000000000049cb1e in testing::TestCase::Run (this=0x36b7e50)
    at /usr/src/gtest/src/gtest.cc:2444
#12 0x00000000004abfd4 in testing::internal::UnitTestImpl::RunAllTests (
    this=0x36b7aa0) at /usr/src/gtest/src/gtest.cc:4315
#13 0x00000000004dea60 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x36b7aa0, 
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4aba6a <testing::internal::UnitTestImpl::RunAllTests()>, 
    location=0xb36330 "auxiliary test code (environments or event listeners)")
    at /usr/src/gtest/src/gtest.cc:2078
No function contains program counter for selected frame.
#14 0x00000000004d521f in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x36b7aa0, 
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4aba6a <testing::internal::UnitTestImpl::RunAllTests()>, 
    location=0xb36330 "auxiliary test code (environments or event listeners)")
    at /usr/src/gtest/src/gtest.cc:2114
#15 0x00000000004a963d in testing::UnitTest::Run (
    this=0x1924ea0 <testing::UnitTest::GetInstance()::instance>)
    at /usr/src/gtest/src/gtest.cc:3926
#16 0x00000000004f1774 in RUN_ALL_TESTS () at /usr/include/gtest/gtest.h:2288
#17 0x00000000004f1643 in main (argc=1, argv=0x7ffcc5862be8)
    at /usr/src/gtest/src/gtest_main.cc:37
#1  0x000000000052cb0f in __gthread_equal (__t1=0, __t2=0)
    at /usr/include/x86_64-linux-gnu/c++/5/bits/gthr-default.h:680
680	  return __gthrw_(pthread_equal) (__t1, __t2);
Dump of assembler code for function __gthread_equal(__gthread_t, __gthread_t):
   0x000000000052caec <+0>:	push   %rbp
   0x000000000052caed <+1>:	mov    %rsp,%rbp
   0x000000000052caf0 <+4>:	sub    $0x10,%rsp
   0x000000000052caf4 <+8>:	mov    %rdi,-0x8(%rbp)
   0x000000000052caf8 <+12>:	mov    %rsi,-0x10(%rbp)
   0x000000000052cafc <+16>:	mov    -0x10(%rbp),%rdx
   0x000000000052cb00 <+20>:	mov    -0x8(%rbp),%rax
   0x000000000052cb04 <+24>:	mov    %rdx,%rsi
   0x000000000052cb07 <+27>:	mov    %rax,%rdi
   0x000000000052cb0a <+30>:	callq  0x0                                     !!!!!!!!!!!!!!!!!!
=> 0x000000000052cb0f <+35>:	leaveq 
   0x000000000052cb10 <+36>:	retq   
End of assembler dump.

Therefore #193 (comment) most likely was in the right direction.
Also see https://stackoverflow.com/questions/23499656/segmentation-fault-in-stdthreadids-stdoperator

[veloman-yunkan]

I think that mesonbuild/meson#3879 has some relation to the current problem

[veloman-yunkan]

openzim/libzim#443 (specifically the change in meson.build) seems to be the most likely (though probably indirect) culprit

[veloman-yunkan]

The two most recent Release&Nightly runs of kiwix-build (#219 & #220) fail in the same way. #218 was the last passing Release&Nightly build. What's interesting, is that the CI runs of kiwix-build don't fail.