Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Copy manifests and DLLs on Windows. #295

Open
wants to merge 3 commits into from

6 participants

@qris

Hopefully fixes issue #294
#294,
at least it seems to work for me.

Generalises the code for copying multiple files from the source to the
destination, so it should be easier to read, and easier to add more files
to the list in future.

Chris Wilson added some commits
Chris Wilson Copy manifests and DLLs on Windows.
Hopefully fixes issue #294
<pypa#294>,
at least it seems to work for me.

Generalises the code for copying multiple files from the source to the
destination, so it should be easier to read, and easier to add more files
to the list in future.
61161ab
Chris Wilson Show the command that failed. 91b4c98
@carljm
Owner

This looks good to me on code review. I'd like two things in order to merge:

  1. Can you add yourself to AUTHORS (if not already there) and make an entry in docs/news.txt?

  2. Can you find one other Windows user to try this out and verify that it works for them (i.e. doesn't break virtualenv)?

@carljm
Owner

Note to whoever merges this pull request: it fixes both #293 and #294

@thedrow

any progress on this? I can check it out and see if it works later on.

@qris

@thedrow, please do give it a try. @carljm wants someone else to test it before committing it, that could be you.

@thedrow

It does not break anything but it does not seem to change anything either.
Note that I am on Windows 7 and not XP.

@qris

@thedrow, are you actually having the problem mentioned in issue #294? If not, I wouldn't expect it to change anything, or at least I'd hope that it doesn't break previously working setups :)

If you are, could you paste the output that you get when running virtualenv, and compare the contents of c:\python*\bin and your-virtualenv\scripts to see if all the .dll and .manifest files have been copied across?

@thedrow

@qris No I don't have the problem mentioned in #294 but it doesn't break anything. The dlls are not copied as far as I can see. The manifest files are.

@qris

@thedrow, please could you copy and paste the output that you get when running virtualenv, and a list of the contents of c:\python*\bin and your-virtualenv\scripts, so I can see if I can figure out why it's not copying the dlls?

@thedrow

@qris sure I'll do so when I'll get home.

@qris

@thedrow any news?

@pnasrat
Owner

@qris can you update so this is mergeable then I'll try take a look although I don't have much in the way of windows access.

@qris

@pnasrat updated but not tested.

@pnasrat
Owner

@piotr-dobrogost I know you're a windows user - can you test this PR and report back.

@piotr-dobrogost
c:\python\2.7>cmd
Microsoft Windows [Version 6.0.6002]
Copyright (c) 2006 Microsoft Corporation.  All rights reserved.

c:\python\2.7>where python
c:\python\2.7\python.exe
c:\Program Files\Python\2.7\python.exe

c:\python\2.7>python --version
Python 2.7.3

c:\python\2.7>dir /s /b *.dll
c:\python\2.7\DLLs\sqlite3.dll
c:\python\2.7\DLLs\tcl85.dll
c:\python\2.7\DLLs\tclpip85.dll
c:\python\2.7\DLLs\tk85.dll
c:\python\2.7\tcl\dde1.3\tcldde13.dll
c:\python\2.7\tcl\reg1.2\tclreg12.dll
c:\python\2.7\tcl\tix8.4.3\tix84.dll

c:\python\2.7>dir /s /b *.manifest
File Not Found

c:\python\2.7>virtualenv z:\2\issue-295
New python executable in z:\2\issue-295\Scripts\python.exe
Installing distribute.............................................................................................................................................................................................................................done.
Installing pip..................done.

c:\python\2.7>cd z:\2\issue-295

c:\python\2.7>z:

z:\2\issue-295>dir /s /b *.dll
File Not Found

z:\2\issue-295>dir /s /b *.manifest
File Not Found

z:\2\issue-295>
@pnasrat
Owner

The virtualenv you ran had this PR in ?

@qris

It seems that stock Python 2.7 installs no .dll or .manifest files in C:\Python27. Assuming that the created virtualenv works (which seems likely) then it's working as designed.

The interesting case is with Bitnami Djangostack, which DOES include .dll and .manifest files in the same directory as Python.exe, and needs them. Virtualenv also seems designed to support cygwin? That should be tested too.

I retested using bitnami-djangostack-1.4.2-1-windows-installer.exe, and all appears to work:

Setting environment for using Microsoft Visual Studio 2010 x86 tools.

C:\Windows\system32>cd "\Program Files\BitNami DjangoStack\python"

C:\Program Files\BitNami DjangoStack\python>scripts\easy_install.exe pip
Searching for pip
Reading http://pypi.python.org/simple/pip/
Reading http://www.pip-installer.org
Reading http://pip.openplans.org
Best match: pip 1.2.1
Downloading http://pypi.python.org/packages/source/p/pip/pip-1.2.1.tar.gz#md5=db8a6d8a4564d3dc7f337ebed67b1a85
Processing pip-1.2.1.tar.gz
Running pip-1.2.1\setup.py -q bdist_egg --dist-dir c:\users\chris\appdata\local\temp\easy_install-t4hfql\pip-1.2.1\egg-dist-tmp-wp0iyv
warning: no files found matching '*.html' under directory 'docs'
warning: no previously-included files matching '*.txt' found under directory 'docs\_build'
no previously-included directories found matching 'docs\_build\_sources'
Adding pip 1.2.1 to easy-install.pth file
Installing pip-script.py script to C:\PROGRA~1\BITNAM~1\python\Scripts
Installing pip.exe script to C:\PROGRA~1\BITNAM~1\python\Scripts
Installing pip.exe.manifest script to C:\PROGRA~1\BITNAM~1\python\Scripts
Installing pip-2.7-script.py script to C:\PROGRA~1\BITNAM~1\python\Scripts
Installing pip-2.7.exe script to C:\PROGRA~1\BITNAM~1\python\Scripts
Installing pip-2.7.exe.manifest script to C:\PROGRA~1\BITNAM~1\python\Scripts

Installed c:\progra~1\bitnam~1\python\lib\site-packages\pip-1.2.1-py2.7.egg
Processing dependencies for pip
Finished processing dependencies for pip

C:\Program Files\BitNami DjangoStack\python>scripts\pip.exe install https://github.com/aptivate/virtualenv/archive/develop.zip
Downloading/unpacking https://github.com/aptivate/virtualenv/archive/develop.zip

  Downloading develop.zip (2.2MB): 2.2MB downloaded
  Running setup.py egg_info for package from https://github.com/aptivate/virtualenv/archive/develop.zip

    warning: no previously-included files matching '*' found under directory 'docs\_build'
Installing collected packages: virtualenv
  Running setup.py install for virtualenv

    warning: no previously-included files matching '*' found under directory 'docs\_build'
    Installing virtualenv-script.py script to C:\PROGRA~1\BITNAM~1\python\Scripts
    Installing virtualenv.exe script to C:\PROGRA~1\BITNAM~1\python\Scripts
    Installing virtualenv.exe.manifest script to C:\PROGRA~1\BITNAM~1\python\Scripts
    Installing virtualenv-2.7-script.py script to C:\PROGRA~1\BITNAM~1\python\Scripts
    Installing virtualenv-2.7.exe script to C:\PROGRA~1\BITNAM~1\python\Scripts
    Installing virtualenv-2.7.exe.manifest script to C:\PROGRA~1\BITNAM~1\python\Scripts
Successfully installed virtualenv
Cleaning up...

C:\Program Files\BitNami DjangoStack\python>scripts\virtualenv.exe c:\test
New python executable in c:\test\Scripts\python.exe
Installing setuptools................done.
Installing pip...................done.

C:\Program Files\BitNami DjangoStack\python>cd \test

C:\test>dir
 Volume in drive C has no label.
 Volume Serial Number is 0C88-BCD5

 Directory of C:\test

09/12/2012  14:16    <DIR>          .
09/12/2012  14:16    <DIR>          ..
09/12/2012  14:07    <DIR>          Include
09/12/2012  14:16    <DIR>          Lib
09/12/2012  14:16    <DIR>          Scripts
               0 File(s)              0 bytes
               5 Dir(s)   1,951,072,256 bytes free

C:\test>dir Scripts
 Volume in drive C has no label.
 Volume Serial Number is 0C88-BCD5

 Directory of C:\test\Scripts

09/12/2012  14:16    <DIR>          .
09/12/2012  14:16    <DIR>          ..
09/12/2012  14:16             2,271 activate
09/12/2012  14:16               585 activate.bat
09/12/2012  14:16             8,252 activate.ps1
09/12/2012  14:16             1,129 activate_this.py
09/12/2012  14:16           161,784 atl90.dll
09/12/2012  14:16               317 deactivate.bat
09/12/2012  14:16               311 easy_install-2.7-script.py
09/12/2012  14:16             7,168 easy_install-2.7.exe
09/12/2012  14:16               531 easy_install-2.7.exe.manifest
09/12/2012  14:16               303 easy_install-script.py
09/12/2012  14:16             7,168 easy_install.exe
09/12/2012  14:16               527 easy_install.exe.manifest
09/12/2012  14:16         1,156,600 mfc90.dll
09/12/2012  14:16               853 Microsoft.VC90.ATL.manifest
09/12/2012  14:16             1,857 Microsoft.VC90.CRT.manifest
09/12/2012  14:16             2,354 Microsoft.VC90.MFC.manifest
09/12/2012  14:16               858 Microsoft.VC90.OpenMP.manifest
09/12/2012  14:16           568,832 msvcp90.dll
09/12/2012  14:16           655,872 msvcr90.dll
09/12/2012  14:16               269 pip-2.7-script.py
09/12/2012  14:16             7,168 pip-2.7.exe
09/12/2012  14:16               522 pip-2.7.exe.manifest
09/12/2012  14:16               261 pip-script.py
09/12/2012  14:16             7,168 pip.exe
09/12/2012  14:16               518 pip.exe.manifest
09/12/2012  14:16            26,624 python.exe
09/12/2012  14:16         2,303,488 python27.dll
09/12/2012  14:16            27,136 pythonw.exe
09/12/2012  14:16            54,272 vcomp90.dll
              29 File(s)      5,004,998 bytes
               2 Dir(s)   1,951,072,256 bytes free

C:\test>scripts\pip.exe
Usage: pip COMMAND [OPTIONS]
You must give a command (use "pip help" to see a list of commands)

C:\test>scripts\python.exe
Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> quit
Use quit() or Ctrl-Z plus Return to exit
>>> quit()

C:\test>scripts\pythonw.exe

C:\test>
@pfmoore
Owner

The standard Python installer puts the Python DLLs into the Windows system directory (assuming you're installing for all users). This is necessary if you're creating Python-based services or OLE servers, AIUI.

There doesn't seem like there's anything wrong with the change, although the title seems wrong (the DLLs were copied in the old code as well as the new one, all I can see that's different is that manifests are now copied). That may well be needed but I've never encountered a case myself where I've needed the manifest file specifically.

It looks like the original code may have changed to copy the DLLs since the original issue #294 was filed. Can the OP retest the issue to confirm that (a) the DLLs are now copied with the mainline branch and (b) the issue still exists, indicating that the manifests are also needed.

BTW, I've just checked my PC, which has all-users installs of Python 2.6-3.3. The DLLs are in C:\Windows\System32, and there are no manifests anywhere that I can see. Could the original issue be related to the non-standard Python installer being used requiring manifests for some reason which is not the case for standard Python? I thought the norm was to embed the manifest files in the DLLs/EXEs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jul 4, 2012
  1. Copy manifests and DLLs on Windows.

    Chris Wilson authored
    Hopefully fixes issue #294
    <pypa#294>,
    at least it seems to work for me.
    
    Generalises the code for copying multiple files from the source to the
    destination, so it should be easier to read, and easier to add more files
    to the list in future.
  2. Show the command that failed.

    Chris Wilson authored
Commits on Oct 24, 2012
  1. @qris
This page is out of date. Refresh to see the latest.
Showing with 32 additions and 32 deletions.
  1. +32 −32 virtualenv.py
View
64 virtualenv.py
@@ -43,7 +43,7 @@
except ImportError:
import configparser as ConfigParser
-join = os.path.join
+from os.path import join
py_version = 'python%s.%s' % (sys.version_info[0], sys.version_info[1])
is_jython = sys.platform.startswith('java')
@@ -1336,6 +1336,9 @@ def install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages, clear):
os.unlink(pyd_pth)
if sys.executable != py_executable:
+ source_dir = os.path.dirname(sys.executable)
+ dest_dir = os.path.dirname(py_executable)
+
## FIXME: could I just hard link?
executable = sys.executable
if is_cygwin and os.path.exists(executable + '.exe'):
@@ -1345,37 +1348,33 @@ def install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages, clear):
logger.info('Executable actually exists in %s' % executable)
shutil.copyfile(executable, py_executable)
make_exe(py_executable)
+
if is_win or is_cygwin:
- pythonw = os.path.join(os.path.dirname(sys.executable), 'pythonw.exe')
- if os.path.exists(pythonw):
- logger.info('Also created pythonw.exe')
- shutil.copyfile(pythonw, os.path.join(os.path.dirname(py_executable), 'pythonw.exe'))
- python_d = os.path.join(os.path.dirname(sys.executable), 'python_d.exe')
- python_d_dest = os.path.join(os.path.dirname(py_executable), 'python_d.exe')
- if os.path.exists(python_d):
- logger.info('Also created python_d.exe')
- shutil.copyfile(python_d, python_d_dest)
- elif os.path.exists(python_d_dest):
- logger.info('Removed python_d.exe as it is no longer at the source')
- os.unlink(python_d_dest)
- # we need to copy the DLL to enforce that windows will load the correct one.
- # may not exist if we are cygwin.
- py_executable_dll = 'python%s%s.dll' % (
- sys.version_info[0], sys.version_info[1])
- py_executable_dll_d = 'python%s%s_d.dll' % (
- sys.version_info[0], sys.version_info[1])
- pythondll = os.path.join(os.path.dirname(sys.executable), py_executable_dll)
- pythondll_d = os.path.join(os.path.dirname(sys.executable), py_executable_dll_d)
- pythondll_d_dest = os.path.join(os.path.dirname(py_executable), py_executable_dll_d)
- if os.path.exists(pythondll):
- logger.info('Also created %s' % py_executable_dll)
- shutil.copyfile(pythondll, os.path.join(os.path.dirname(py_executable), py_executable_dll))
- if os.path.exists(pythondll_d):
- logger.info('Also created %s' % py_executable_dll_d)
- shutil.copyfile(pythondll_d, pythondll_d_dest)
- elif os.path.exists(pythondll_d_dest):
- logger.info('Removed %s as the source does not exist' % pythondll_d_dest)
- os.unlink(pythondll_d_dest)
+ files_to_sync = set(('pythonw.exe', 'python_d.exe',
+ 'python%s%s.dll' % (sys.version_info[0], sys.version_info[1]),
+ 'python%s%s_d.dll' % (sys.version_info[0], sys.version_info[1]),
+ ))
+
+ # Copy any manifests and DLLs we can find
+ import glob
+
+ for dir in (source_dir, dest_dir):
+ for pattern in ('*.manifest', '*.dll'):
+ for found_file in glob.iglob(join(dir, pattern)):
+ files_to_sync.add(os.path.basename(found_file))
+
+ for filename in files_to_sync:
+ source_file = join(source_dir, filename)
+ dest_file = join(dest_dir, filename)
+
+ if os.path.exists(source_file):
+ logger.info('Also created %s' % filename)
+ shutil.copyfile(source_file, dest_file)
+ elif os.path.exists(dest_file):
+ logger.info('Removed %s as it is no longer at the source' %
+ filename)
+ os.unlink(dest_file)
+
if is_pypy:
# make a symlink python --> pypy-c
python_executable = os.path.join(os.path.dirname(py_executable), 'python')
@@ -1486,7 +1485,8 @@ def install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages, clear):
logger.fatal('ERROR: The executable %s could not be run: %s' % (py_executable, e))
sys.exit(100)
else:
- raise e
+ raise Exception("Failed to run Python for testing: %s: %s" %
+ (cmd, e))
proc_stdout = proc_stdout.strip().decode("utf-8")
proc_stdout = os.path.normcase(os.path.abspath(proc_stdout))
Something went wrong with that request. Please try again.