Skip to content

PYTHONHOME is being set in batch files needlessly #22

Closed
piotr-dobrogost opened this Issue Nov 3, 2012 · 15 comments

3 participants

@piotr-dobrogost
Collaborator

What's the reason for python.bat and other batch files to set PYTHONHOME environment variable? Python executable does not set this variable and I think setting it by python.bat and other scripts is wrong.

@davidmarble
Owner

Hi Piotr,

Please don't use github issues to ask questions about how something works. Just look me up on my website and send me an email.

These scripts need to know which python executable to call.

  1. First, they look to see if PYTHONHOME is already set.

    For example, let's say you have a virutalenv in c:\Users\piotr\Envs\testenv. You activate it with workon testenv. Now when you call any virtualenvwrapper script, PYTHONHOME will be set to c:\Users\piotr\Envs\testenv so all commands within the script will use that specific python with its site-packages, additional executables installed, etc.

  2. Second, if PYTHONHOME it not set, for any reason, the script searches your PATH to find the first python.exe.

Now, let's say you do not have an active virtualenv, but you do have PYTHONHOME set. All these scripts check for this environment variable first, because you may have multiple python versions installed. This allows you to switch between those python versions by setting PYTHONHOME. Otherwise a call to python (which will call python.bat) will always use the first python.exe it finds in the PATH. It can be much more painful to re-arrange your PATH when you want to switch python versions -- instead, just change PYTHONHOME.

@davidmarble davidmarble closed this Nov 3, 2012
@piotr-dobrogost
Collaborator

Please don't use github issues to ask questions about how something works.

I think setting PYTHONHOME is wrong and that's the issue I raised. Very often raising some issue leads to discussion and involves explaining how something is supposed to work. I see nothing wrong with it. I very often prefer public discussion because it might be of interest to other people and save their time.

Nowhere in virtualenvwrapper.sh PYTHONHOME is being set. What's so specific about Windows that you use this variable in your port so extensively?

@davidmarble
Owner
@piotr-dobrogost
Collaborator

I'd like to bring your attention to the fact that PYTHONHOME is not being set in activate.bat. On the contrary, when virtualenv is activated it gets cleared.

The example of virtualenvwrapper-win using PYTHONHOME whereas virtualenvwrapper does not is cdvirtualenv function. Virtualenvwrapper uses VIRTUAL_ENV whereas virtualenvwrapper-win uses PYTHONHOME

@piotr-dobrogost
Collaborator

@davidmarble
Could you please take a look at my last comment and write what you think? I'd like to establish what should be fixed before proceeding with the changes.

@davidmarble
Owner

I don't know how much you already understand about Python at a deep level and Windows, since we've only corresponded on a few issues here on github.

Please first confirm that you've read through and understand the following python source file (at minimum the instructive comments at the top), and if you still feel strongly that davidmarble/virtualenvwrapper-win is doing things the wrong way or could be simpler, please explain further what approach you would like to see and how it would fit within the constraints of getpathp.c below.

http://svn.python.org/projects/python/trunk/PC/getpathp.c

I'll paste the most relevant comments from the top of the file:

This describes how sys.path is formed on Windows.  It describes the
functionality, not the implementation (ie, the order in which these
are actually fetched is different)

* Python always adds an empty entry at the start, which corresponds
  to the current directory.

* If the PYTHONPATH env. var. exists, its entries are added next.

* We look in the registry for "application paths" - that is, sub-keys
  under the main PythonPath registry key.  These are added next (the
  order of sub-key processing is undefined).
  HKEY_CURRENT_USER is searched and added first.
  HKEY_LOCAL_MACHINE is searched and added next.
  (Note that all known installers only use HKLM, so HKCU is typically
  empty)

* We attempt to locate the "Python Home" - if the PYTHONHOME env var
  is set, we believe it.  Otherwise, we use the path of our host .EXE's
  to try and locate our "landmark" (lib\\os.py) and deduce our home.
  - If we DO have a Python Home: The relevant sub-directories (Lib,
     plat-win, lib-tk, etc) are based on the Python Home
  - If we DO NOT have a Python Home, the core Python Path is
     loaded from the registry.  This is the main PythonPath key,
     and both HKLM and HKCU are combined to form the path)

* Iff - we can not locate the Python Home, have not had a PYTHONPATH
  specified, and can't locate any Registry entries (ie, we have _nothing_
  we can assume is a good path), a default path with relative entries is
  used (eg. .\Lib;.\plat-win, etc)
@davidmarble
Owner

@piotr-dobrogost Perhaps it is best to change my project to be consistent with the other two (virtualenv, virtualenvwrapper). These were a few of my assumptions when I began virtualenvwrapper-win that may be wrong now that I've had more experience:

  1. I wanted to be certain the active virtualenv paths are placed at the front of sys.path. I use PYTHONPATH in virtualenvwrapper-win scripts because it's the only way to be certain that active virtualenv paths will always be first.

    • For example, let's say I set a custom PYTHONPATH for the current user or machine to always include a specific directory, such as c:\pyutils, which contains a module utilX (let's say it's utilX==1.2 using pip notation). But inside a virtualenv of mine, I want to use utilX==1.1. Under virtualenv currently, I would have to manually add the path to utilX==1.1 do this. With virtualenvwrapper-win, I simply add utilX==1.1 to my virtualenv (via pip or easy_install), and when I activate that virtualenv version 1.1 will always be in sys.path before utilX==1.2.
    • Some counter arguments:
      • If the user calls /path/to/virtualenv/bin/python.exe directly, the extra work I've put into activate.bat will be useless -- i.e. PYTHONPATH will not be modified. This is the primary reason I'm considering my assumptions from a year ago were wrong.
      • If the user sets a custom PYTHONPATH, they should always expect it to override anything in their virtualenv.
  2. I set PYTHONHOME to the proper value when a virtualenv is activated for a couple reasons.
    a. If the user sets the PYTHONHOME env variable by default for their user/machine, for example because they depend on it for system scripting, they will expect their virtualenv tools to keep the variable pointing to the current python installation.
    b. getpathp.c already tries to programmatically figure out PYTHONHOME if it's not set. Setting it explicitly merely simplifies this step, and makes the variable available for other uses (e.g. system scripting).

    • But once again:
      • If the user calls /path/to/virtualenv/bin/python.exe directly, the extra work I've put into activate.bat to set PYTHONHOME will be useless.
@piotr-dobrogost
Collaborator

Thanks for taking time to describe your reasoning behind using PYTHONHOME and PYTHONPATH in this project. Also, thanks for sharing a link to getpathp.c.

I believe both PYTHONHOME and PYTHONPATH are means for a user to directly influence the Python's environment and as such should not be manipulated by any scripts other than those authored by user. The whole process of finding out paths to be used by Python interpreter normally is based on the path of the interpreter itself. That's why it's enough to make given virtualenv's interpreter the first interpreter on the PATH as a mean to activate this virtualenv. This, as you noticed yourself, makes manipulation of other environment variables unnecessary. Also it tells a lot that virtualenvwrapper does not use environment variables other than PATH in spite of the hope articulated in this statement from mkvirtualenv.bat:

In activate.bat, add VIRTUAL_ENV directories to PYTHONPATH and set PYTHONHOME to the VIRTUAL_ENV.
This should be a change adopted by virtualenv.

Also, I think the matter of whether to modify PYTHONPATH/PYTHONHOME came out during development of the "original" virtualenvwrapper and we could try to find out such a discussion and see what arguments were brought up there.
EDIT
As I did not find the discussion on this topic in existing issues I created this one - Should virtualenvwrapper modify PYTHONHOME and/or PYTHONPATH?

@vernondcole

I am glad I found this (unfortunately closed) issue. I just spent an entire day tearing my hair out trying to figure out why all of my various python versions suddenly quit working. It was actually a relief to discover that virtualenvwrapper stepping on PYTHONHOME was the only cause of my problems.
I have been trying to set up virtualenv to switch between versions of Python to work on a cross-version open source project. (http://sf.net/projects/adodbapi). Until today, I used an array of .bat files on the search path to switch versions. i.e. py24.bat started Python 2.4, etc. This does not work so well with version 3.2, so I was hoping for a better way. Virtualenv works so well on Linux that I figured that it would be great when I dragged my old Windows box back out. David, you have done a marvelous job of getting Windows to do things I never knew were possible. Nevertheless, I have yet to figure out a way of calling "mkvirtualenv --python={whatever}" that works. So I come here looking for a cookbook and discovered that this question is still a work in progress.
I will watch this thread and if I can do anything to help will stick my paddle in. Whatever gets worked out, the documentation needs to be expanded. Starting the installation instructions with "pip" assumes a lot -- such as distribute being installed, and the path already being messed with to call it. Some instructions on how the search path is expected to be set up are in order. For example, pythonxx/scripts must be searched before pythonxx or the python.bat file will never be called. Also the Python 3.3 launcher in c:\windows\py.exe needs to be considered.

@davidmarble davidmarble reopened this Mar 25, 2013
@vernondcole

I have just been trying out Python 3.3. The launcher is truly wonderful. I can throw away all of my old pynn.bat files and run any version of python using it. Both "py -2.3 test.py" and "py -3.2-32 test.py" work on my machine. There is also a built-in venv module which will do great things in the future, but I do not see a way to get it to support older versions of Python. Clearly any new virtual environment work will need to be compatible with it. http://docs.python.org/3/library/venv.html

The official line on PYTHONHOME seems to be written up in the Python 3.4 documentation at:
http://docs.python.org/dev/using/windows.html

@piotr-dobrogost
Collaborator

@davidmarble

Do we bite the bullet? :)

@davidmarble
Owner

Here's my first contribution after this thread -- a Windows python launcher that works for 2.5-3.3 and supports both the command prompt and MSYS/MINGW32.

https://github.com/davidmarble/pywin

Next I'll turn to simplifying a lot of virtualenvwrapper-win.

@piotr-dobrogost
Collaborator

pywin is an interesting idea. Is pywin going to be a dependency of virtualenvwrapper-win?

@davidmarble
Owner

The idea is that virtualenvwrapper-win will work whether the user has pywin, py.exe, or no launcher at all.

@davidmarble
Owner

@piotr-dobrogost and @vernondcole,

I just finished a bunch of changes and released version 1.1 of virtualenvwrapper-win.

Download or install via pypi and give it a spin. It depends on virtualenv==1.9.1 (should be automatically installed). README has been updated with minor differences (no more pyassoc or python.bat -- use davidmarble/pywin if you want similar updated features).

See the commit history for issues that were fixed, which included removing any modifications to PYTHONHOME and PYTHONPATH. Please open a new issue if you encounter bugs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.