Something's awry with downloading xapian-core in kiwix-build

[julianharty]

This may just be me / my environment however I've had problems caused by what seem to be issues either downloading or extracting the xapian-core dependency as part of kiwix-build.py

Here's the error trace when trying to build the arm branch. It also occurs on the same device (a Pi 3) when trying to build from the master branch

I'll try to run the build in another linux environment to triangulate the scope of the problem.

(kiwix)pi@rachel ~/kiwix-dev/kiwix-build $ pyenv exec python kiwix-build.py --target-platform raspberrypi_dyn
[INSTALL PACKAGES]
 - cmake : SKIP
 - automake : SKIP
 - ccache : SKIP
SKIP, No package to install.
[PREPARE]
prepare sources for toolchain armhf :
  gitclone armhf : OK
  gitupdate armhf : OK
prepare sources pugixml :
  download pugixml : SKIP
  extract pugixml : OK
  patch pugixml : OK
prepare sources zlib :
  download zlib : OK
  extract zlib : OK
  patch zlib : OK
prepare sources lzma :
  download lzma : SKIP
  extract lzma : OK
prepare sources uuid :
  download uuid : OK
  extract uuid : OK
prepare sources xapian-core :
  download xapian-core : SKIP
  extract xapian-core : ERROR
Traceback (most recent call last):
  File "/home/pi/kiwix-dev/kiwix-build/utils.py", line 75, in extract_archive
    archive = tarfile.open(archive_path)
  File "/home/pi/.pyenv/versions/3.4.3/lib/python3.4/tarfile.py", line 1553, in open
    raise ReadError("file could not be opened successfully")
tarfile.ReadError: file could not be opened successfully

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "kiwix-build.py", line 862, in <module>
    builder.run()
  File "kiwix-build.py", line 831, in run
    self.prepare_sources()
  File "kiwix-build.py", line 812, in prepare_sources
    source.prepare()
  File "/home/pi/kiwix-dev/kiwix-build/dependency_utils.py", line 117, in prepare
    self.command('extract', self._extract)
  File "/home/pi/kiwix-dev/kiwix-build/dependency_utils.py", line 86, in command
    return self.target.command(*args, **kwargs)
  File "/home/pi/kiwix-dev/kiwix-build/dependency_utils.py", line 50, in command
    ret = function(*args, context=context)
  File "/home/pi/kiwix-dev/kiwix-build/dependency_utils.py", line 106, in _extract
    name=self.source_dir)
  File "/home/pi/kiwix-dev/kiwix-build/utils.py", line 115, in extract_archive
    if archive is not None:
UnboundLocalError: local variable 'archive' referenced before assignment

[julianharty]

x-ref #32 which has an additional trace of what appears to be the same problem.

[mgautierfr]

I don't know what happen here.
The download is skipped because the archive exists and has the same sha256sum than expected.
So the "extract step" should find the archive_path correctly.

What is the content of the ARCHIVE directory ?

[julianharty]

(kiwix)pi@rachel ~/kiwix-dev/kiwix-build $ ls -lart ARCHIVE/
total 5272
-rw-r--r--  1 pi pi  363189 Apr 16 13:28 pugixml-1.2.tar.gz
-rw-r--r--  1 pi pi 1048328 Apr 16 13:28 xz-5.0.4.tar.bz2
-rw-r--r--  1 pi pi 2799492 Apr 16 13:28 xapian-core-1.4.2.tar.xz
-rw-r--r--  1 pi pi  571091 Apr 16 16:03 zlib-1.2.8.tar.gz
drwxr-xr-x  2 pi pi    4096 Apr 16 16:03 .
-rw-r--r--  1 pi pi  601699 Apr 16 16:03 e2fsprogs-libs-1.43.4.tar.gz
drwxr-xr-x 13 pi pi    4096 Apr 19 08:24 ..

I then ran kiwix-build.py as follows:
python -v kiwix-build.py which provided some clues. It seems utils.py wants to use gzip

prepare sources xapian-core :
  download xapian-core : SKIP
  extract xapian-core : # /home/pi/.pyenv/versions/3.4.3/lib/python3.4/__pycache__/lzma.cpython-34.pyc matches /home/pi/.pyenv/versions/3.4.3/lib/python3.4/lzma.py
# code object from '/home/pi/.pyenv/versions/3.4.3/lib/python3.4/__pycache__/lzma.cpython-34.pyc'
# destroy lzma
# /home/pi/.pyenv/versions/3.4.3/lib/python3.4/__pycache__/gzip.cpython-34.pyc matches /home/pi/.pyenv/versions/3.4.3/lib/python3.4/gzip.py
# code object from '/home/pi/.pyenv/versions/3.4.3/lib/python3.4/__pycache__/gzip.cpython-34.pyc'
import 'gzip' # <_frozen_importlib.SourceFileLoader object at 0x76507f90>
ERROR
Traceback (most recent call last):
  File "/home/pi/kiwix-dev/kiwix-build/utils.py", line 75, in extract_archive
    archive = tarfile.open(archive_path)
  File "/home/pi/.pyenv/versions/3.4.3/lib/python3.4/tarfile.py", line 1553, in open
    raise ReadError("file could not be opened successfully")
tarfile.ReadError: file could not be opened successfully

BTW https://docs.python.org/3/library/tarfile.html does not mention the ability to directly decompress .tar.xz format although it covers various similar compression formats.

I managed to decompress the file on the command line

history 
-- cut --
 1291  cp ARCHIVE/xapian-core-1.4.2.tar.xz ~
 1292  cd ~
 1293  ls -l
 1294  tar xf xapian-core-1.4.2.tar.xz
 1295  ls -l
 1296  ls -la xapian-core-1.4.2/

I handcoded python interactively to see if I could isolate the problem

(kiwix)pi@rachel ~/kiwix-dev/kiwix-build $ python
Python 3.4.3 (default, Apr 16 2017, 09:52:16) 
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tarfile
>>> tarfile.open('ARCHIVE/xapian-core-1.4.2.tar.xz')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pi/.pyenv/versions/3.4.3/lib/python3.4/tarfile.py", line 1553, in open
    raise ReadError("file could not be opened successfully")
tarfile.ReadError: file could not be opened successfully
>>> quit()

This fails for the particular file, but not for a small test.tar.xz file I created by hand (with a single small file).

(kiwix)pi@rachel ~ $ tar -cJf test.tar.xz minicom.log 
(kiwix)pi@rachel ~ $ ls -lart
-- cut --
-rw-r--r--  1 pi   pi       204 Apr 19 09:07 test.tar.xz
(kiwix)pi@rachel ~ $ python
Python 3.4.2 (default, Oct 19 2014, 13:31:11) 
[GCC 4.9.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tarfile
>>> tarfile.open('test.tar.xz')
<tarfile.TarFile object at 0x76a4bf50>
>>> quit()

I tried seeing what Linux thinks both files contain

(kiwix)pi@rachel ~ $ file test.tar.xz 
test.tar.xz: XZ compressed data
(kiwix)pi@rachel ~ $ file xapian-core-1.4.2.tar.xz 
xapian-core-1.4.2.tar.xz: XZ compressed data
(kiwix)pi@rachel ~ $ tar -xJf test.tar.xz 

Both files have the same permissions and owner

ls -la *.xz
-rw-r--r-- 1 pi pi     204 Apr 19 09:07 test.tar.xz
-rw-r--r-- 1 pi pi 2799492 Apr 19 08:45 xapian-core-1.4.2.tar.xz

Sooooo, it seems python 3 (v3.4.3) isn't able to decompress this specific file for some reason. Any more ideas? I'll also keep looking / exploring e.g. what about providing the file in another format? and are there ways we can convince the python tarfile.open to reveal why it doesn't like this file?
PS: I've tried opening the file in the local folder in case the problem was related to the path.

[kelson42]

We now use xapian 1.4.5, @julianharty Do you still have problems?

[kelson42]

I close that ticket because of lack of feedback.