patool dislikes tricky filenames

[yarikoptic]

edit: bug in cPython: https://bugs.python.org/issue38449

While test building 0.11.8 for debian sid, build failed with a number of tests failing with the same error from patool (patoolib=1.12):

======================================================================
ERROR: datalad.tests.test_archives.test_ExtractedArchive
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/build/datalad-0.11.8/.pybuild/cpython3_3.7_datalad/build/datalad/tests/utils.py", line 440, in newfunc
    create_tree(d, tree, archives_leading_dir=archives_leading_dir)
  File "/build/datalad-0.11.8/.pybuild/cpython3_3.7_datalad/build/datalad/utils.py", line 2248, in create_tree
    archives_leading_dir=archives_leading_dir)
  File "/build/datalad-0.11.8/.pybuild/cpython3_3.7_datalad/build/datalad/utils.py", line 2211, in create_tree_archive
    compress_files([dirname], name, path=path, overwrite=overwrite)
  File "/build/datalad-0.11.8/.pybuild/cpython3_3.7_datalad/build/datalad/support/archives.py", line 251, in compress_files
    verbosity=100)
  File "/usr/lib/python3/dist-packages/patoolib/__init__.py", line 505, in _create_archive
    format, compression = get_archive_format(archive)
  File "/usr/lib/python3/dist-packages/patoolib/__init__.py", line 293, in get_archive_format
    raise util.PatoolError("unknown archive format for file `%s'" % filename)
patoolib.util.PatoolError: unknown archive format for file ` "';b&b&c `| .tar.gz'

I do not remember us changing any of those tests since last release but previous release of patool used for building 0.11.7 where no failures happened was also 0.12 (checked -- debian revision was 0.12-3 in both cases).

neurodebian@smaug ~/deb/builds/datalad/0.11.7-1 % grep test_ExtractedArchive datalad_0.11.7-1_amd64.build
datalad.tests.test_archives.test_ExtractedArchive ... ok

So I have no immediate idea and it would need some digging.

edit: also doesn't reproduce for me locally

[yarikoptic]

boiling down to indigestion of ; in a filename (and I think it goes deeper into python stdlib, upgrading my laptop atm):

root@smaug:/tmp/buildd/datalad-0.11.8# python3 -c 'import patoolib.util as ut; print(ut.guess_mime(r";b.tar.gz"))'
(None, None)
root@smaug:/tmp/buildd/datalad-0.11.8# python3 -c 'import patoolib.util as ut; print(ut.guess_mime(r"b.tar.gz"))'
('application/x-tar', 'gzip')

[yarikoptic]

yeap, old

$> python3 -c 'import mimetypes; mimedb = mimetypes.MimeTypes(strict=False); print(mimedb.guess_type(";1.tar.gz"))'
('application/x-tar', 'gzip')

$> python3 --version
Python 3.7.3rc1

new (debian sid):

root@smaug:/tmp/buildd/datalad-0.11.8# python3 -c 'import mimetypes; mimedb = mimetypes.MimeTypes(strict=False); print(mimedb.guess_type(";1.tar.gz"))'
(None, None)
root@smaug:/tmp/buildd/datalad-0.11.8# python3 --version
Python 3.7.5rc1

[yarikoptic]

since it is unlikely to be hit in real life, for now I will

• patch our code to avoid ; in the filename if it confuses mimetypes
• report issue to Debian
• verify status with current cPython git (v3.8.0b1-1174-g2b7dc40b2af) if I manage to build it, and proceed from there

[yarikoptic]

ok -- verified with upstream cpython...

$> ./python3.9 -c 'import mimetypes; mimedb = mimetypes.MimeTypes(strict=False); print(mimedb.guess_type(";1.tar.gz"))'                           
(None, None)

$> ./python3.9 -c 'import mimetypes; mimedb = mimetypes.MimeTypes(strict=False); print(mimedb.guess_type("1.tar.gz"))' 
('application/x-tar', 'gzip')

$> git describe
v3.8.0b1-1174-g2b7dc40b2af

[yarikoptic]

our report was taken very seriously, offending code was reverted, 3.8.0 was released with a "fix". not sure if any reason to keep it open