Allow user to define temp directory location

[kracekumar]

• Pip version: 9.0.1
• Python version: 3.5
• Operating system: Trisquel

Description:

I have two partitions in the Linux box. One with 20 GB (partition 1) where OS runs and data are stored in other partition with 250+ GB space (partition 2).

The partition 1 always has less than 1 GB of space after prolonged use of the machine and so far no other program has complained about the lack of space.

When I run pip I specify an option for cache-dir in the partition 2. Unfortunately, pip creates temp directory in /tmp directory. This creates problem while installing large set of packages like tensorflow. Specifying target option isn't helpful. Here is the code which creates temp directory.

    if options.target_dir:
            options.ignore_installed = True
            temp_target_dir = tempfile.mkdtemp()
            options.target_dir = os.path.abspath(options.target_dir)
            if (os.path.exists(options.target_dir) and not
                    os.path.isdir(options.target_dir)):
                raise CommandError(
                    "Target path exists but is not a directory, will not "
                    "continue."
                )
            install_options.append('--home=' + temp_target_dir)

One option is to use dir option while creating the directory.

• Does it make sense to call tempfile.mkdtemp( dir=options.target_dir)?
• Does it make sense to get the tmp directory from user?

What I've run:

krace@hotbox /m/u/c/tf> pwd
/media/user/code/tf
pip install tensorflow protobuf==3.0.0 numpy==1.11.1 pandas==0.18.1 bokeh==0.12.3 matplotlib pillow==3.4.2 --cache-dir .
Traceback (most recent call last):
  File "/media/user/code/tf/venv/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/media/user/code/tf/venv/lib/python3.5/site-packages/pip/commands/install.py", line 336, in run
    wb.build(autobuilding=True)
  File "/media/user/code/tf/venv/lib/python3.5/site-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/media/user/code/tf/venv/lib/python3.5/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/media/user/code/tf/venv/lib/python3.5/site-packages/pip/req/req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "/media/user/code/tf/venv/lib/python3.5/site-packages/pip/download.py", line 821, in unpack_url
    hashes=hashes
  File "/media/user/code/tf/venv/lib/python3.5/site-packages/pip/download.py", line 663, in unpack_http_url
    unpack_file(from_path, location, content_type, link)
  File "/media/user/code/tf/venv/lib/python3.5/site-packages/pip/utils/__init__.py", line 600, in unpack_file
    flatten=not filename.endswith('.whl')
  File "/media/user/code/tfvenv/lib/python3.5/site-packages/pip/utils/__init__.py", line 502, in unzip_file
    fp.write(data)
OSError: [Errno 28] No space left on device

[pfmoore]

AIUI pip should respect the standard TMPDIR environment variable. Does that work for you?

[kracekumar]

AIUI pip should respect the standard TMPDIR environment variable. Does that work for you?

Yes! I didn't know the existence of TMPDIR.

[pfmoore]

Excellent. There's nothing specific in the pip docs about this, as it's a standard OS facility, but that doesn't help much if you don't know about it :-)

Glad you got it working.

[pembo13]

For anyone who Googled there way here, @kracekumar is right. In my case, I was using some Rackspace provided VMs, and /tmp had "noexec" and pip uses /tmp for anything. Luckily, using TMPDIR to a different path worked.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.