Pip compile breaks with scandir==1.4

[milin]

pip-compiles fails for scandir==1.4 dependency

scandir==1.4 not in cache, need to check index
Traceback (most recent call last):
  File "/www/env/bin/pip-compile", line 11, in <module>
    sys.exit(cli())
  File "/www/env/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/www/env/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/www/env/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/www/env/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/www/env/lib/python2.7/site-packages/piptools/scripts/compile.py", line 168, in cli
    results = resolver.resolve()
  File "/www/env/lib/python2.7/site-packages/piptools/resolver.py", line 105, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/www/env/lib/python2.7/site-packages/piptools/resolver.py", line 187, in _resolve_one_round
    for best_match in best_matches
  File "/www/env/lib/python2.7/site-packages/piptools/resolver.py", line 188, in <genexpr>
    for dep in self._iter_dependencies(best_match))
  File "/www/env/lib/python2.7/site-packages/piptools/resolver.py", line 261, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/www/env/lib/python2.7/site-packages/piptools/repositories/pypi.py", line 161, in get_dependencies
    dependencies = reqset._prepare_file(self.finder, ireq)
  File "/www/env/lib/python2.7/site-packages/pip/req/req_set.py", line 587, in _prepare_file
    session=self.session, hashes=hashes)
  File "/www/env/lib/python2.7/site-packages/pip/download.py", line 810, in unpack_url
    hashes=hashes
  File "/www/env/lib/python2.7/site-packages/pip/download.py", line 653, in unpack_http_url
    unpack_file(from_path, location, content_type, link)
  File "/www/env/lib/python2.7/site-packages/pip/utils/__init__.py", line 605, in unpack_file
    untar_file(filename, location)
  File "/www/env/lib/python2.7/site-packages/pip/utils/__init__.py", line 551, in untar_file
    path = os.path.join(location, fn)
  File "/www/env/lib/python2.7/posixpath.py", line 80, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc6 in position 27: ordinal not in range(128)
make: *** [install] Error 1

##### Steps to replicate

1. ...have scandir==1.4 in requirements file
2. ...
3. ...

##### Expected result
Should work
...

##### Actual result
Errors out
...

[jochen-ott-by]

I think the problem is the from __future__ import unicode_literals in piptools/_compat/tempfile.py, so this succeeds:

tmpdir = TemporaryDirectory()
assert isinstance(tmpdir.name, unicode)

This unicode directory name causes problems if trying to append a binary str via os.path.join which is not ASCII-encoded.

I propose to either entirely remove the unicode_literals import from the above file or to explicitly use binary strings for the TemporaryDirectory.__init__ default values.

[lefterisnik]

Hi @byjott, I tried both with no success. I get the same error

[jochen-ott-by]

Yes, indeed, that's not enough, as apparently TemporaryDirectory is constructed with non-default arguments in a file also using unicode_literals (specifically from repositories/pypi.py:67).

What finally did work for me was to edit piptools/_compat/tempfile.py and insert the following lines at the top of TemporaryDirectory.__init__:

suffix = suffix.encode()
prefix = prefix.encode()
if dir is not None:
    dir = dir.encode()

This ensures the temporary directory name (self.name in this context) is a binary str, not unicode.

[lefterisnik]

Also, pip-compile works fine when you build pip-tools under python3.

@byjott I will try your solution and I will write back for feedback

[lefterisnik]

@byjott, yes your solution works fine. thanks.

[federicobond]

I can reproduce the problem and confirm that the solution works fine.

[Groxx]

I managed to catch the error:

[localhost] local: pip-compile --no-index --upgrade requirements.in
> /.../env/local/lib/python2.7/site-packages/pip/utils/__init__.py(496)unzip_file()
-> print "caught"
(Pdb) l
491                     fn = split_leading_dir(name)[1]
492                 try:
493                     fn = os.path.join(location, fn)
494                 except:
495                     import pdb; pdb.set_trace()
496  ->                 print "caught"
497                     raise
498                 dir = os.path.dirname(fn)
499                 if fn.endswith('/') or fn.endswith('\\'):
500                     # A directory
501                     ensure_dir(fn)
(Pdb) location
u'/tmp/tmpDj6_ltbuild/scandir'
(Pdb) fn
'test/testdir/subdir/unicod\xc6\x8f.txt'
(Pdb)

Unfortunately, after fixing it once with @byjott's approach, I was unable to reproduce - if you're debugging, watch out! I can reproduce at-will now though, if anyone would like more info.