Add/improve support for MSYS and Cygwin shells on Windows #107

Closed
wants to merge 2 commits into
from

Conversation

Projects
None yet
7 participants
Contributor

greghaskins commented Mar 27, 2011

Virtualenv currently copies the ACTIVATE_SH script on Windows when it detects it's being run from cygwin, and this patch adds support for doing the same with MSYS/MinGW as well. The only trick is that os.abspath doesn't produce paths that MSYS likes, so I've added a couple lines to translate c:\path\to\env to /c/path/to/env just for that file.

I try to stay away from cmd.exe as much as possible, so I've been using a patched version of virtualenv on my own installation of MSYS for a few weeks and everything has worked perfectly.

Update: Despite what I said above, @cliffxuan has pointed out that virtualenv actually doesn't treat Cygwin specially. since the OSTYPE trick doesn't work. So, I've expanded the scope of this branch slightly to include some fixes to help on that front as well.

This should do the right thing with network shares (which start with slashes already). That's not something I've tested extensively, though.

Contributor

pnasrat commented Jun 3, 2011

This has been merged manually closing

pnasrat closed this Jun 3, 2011

Which branch was this merged to? I can't find the commit on either the develop or master branches.

pnasrat reopened this Jun 8, 2011

Contributor

pnasrat commented Jun 8, 2011

My bad I'm sure I checked the git history and the code, but you seem to be correct. Will look at this weekend, patch appreciated - sorry for the confusion.

Does this give you any output on Cygwin? I've tried on Cygwin using both Cygwin Python and Windows Python. It does not generate any output.

os.environ.get('OSTYPE')

From shell, this does work though.
$ echo $OSTYPE
cygwin

Contributor

greghaskins commented Aug 27, 2011

Hmm, same here. OSTYPE is defined as cygwin in the shell, but is None from Python's eyes. Just like you, I'm getting the same result in both Cygwin Python and Windows-native Python.

The patch above only really focused on adding MSYS support (I kept the Cygwin detection part the same as-was), but looking at it now, it seems the OSTYPE check actually never occurs, since sys.platform == 'cygwin' when running under Cygwin Python. (sys.platform is how Cygwin is detected elsewhere in the code, incidentally.) This behavior is confirmed by the fact that virtualenv sets up activate.csh, and activate.fish, and not activate.bat when run under Cygwin Python, but creates activate.bat and none of the others when run under Win32 Python (even if done so from a Cygwin bash prompt).

Therefore if you create a Windows Python virtualenv, you will not be able to use it under Cygwin unless you make your own activate script. And conversely, if you set up the environment with Cygwin Python, you won't be able to use it from Cmd or MSYS.

I think I'll update my patch to account for this shortcoming as well, and will post a fresh commit. It'd be nice to have something that "just works" with any of Cmd, Cygwin bash, or MSYS bash.

Correct, your patch does work for MSYS + Window Python. Thanks a lot for looking into this problem for Cygiwn.

I am working on a patch on virtualenvwrapper for Cygwin + Windows Python. It is almost working apart from the fact that I have to manually construct the activate.sh script. Because virtualenvwrapper is based on a shell script, I have the luxury to explicitly export OSTYPE when it is sourced. However, the better way is like what you have described -- to address the issue in virtualenv.

This is the function that I use to determine if the platform is Cygwin. It's may not be the best one, but I didn't find anything better. Hope it's of any use.

def is_cygwin():
try:
uname = subprocess.Popen(['uname'], stdout=subprocess.PIPE).communicate()[0]
return uname.startswith('CYGWIN')
except:
#pure windows
return False

@greghaskins greghaskins Enhanced MSYS/Cygwin support for Windows virtualenvs.
Win32 envs may now be created/activated/used in Cmd, MSYS bash, and/or Cygwin
bash without requiring separate installations or configurations. This excludes
virtualenvs with interpreters compiled against the Cygwin dll, however, which
may only be activated in the Cygwin environment itself.
d681c0f
Contributor

greghaskins commented Aug 27, 2011

Ah, the joys of development on Windows, eh? I've been using this forked version of virtualenv since March---perhaps if we straighten out these kinks the authors will be kind enough to fold this into the next mainline release, and make our lives a little easier the next time we reach for pip install. I think the latest release of virtualenvwrapper pulled in a patch for MSYS, so hopefully we're on our way to feature-parity with our Linux comrades.

After some testing, I made a few small tweaks to my patch so the activate script will work with either MSYS or Cygwin. Instead of attacking this on the Python side (I was considering an approach similar to what you proposed above), I just let the activate script handle the tricky bits with a little runtime checking and cygpath translation:

VIRTUAL_ENV="$(if [ "$OSTYPE" == "cygwin" ]; then cygpath -u 'c:\path\to\env'; else echo '/c/path/to/env'; fi;)"

which works because OSTYPE is present in the shell environment, even if Python can't see it.

With this modification, I can create a virtualenv in any of Cmd, MSYS, or Cygwin (as long as it uses the Win32 Python) and the same environment can be activated/used from any of the other shells. Of course it works fine if you just use one of those at a time.

The only slight drawback with using MSYS/Cygwin at the moment is that a few installed scripts/entry points may need to be run with an explicit python /path/to/script.py, since bash sometimes doesn't like the DOS-style hashbang paths that get put in at the top. This isn't an issue for anything installed via setuptools/pip, since those get nice exe wrappers which handle everything. I suspect that making the environment relocatable might be a solution, since that changes the scripts' first lines to #!/usr/bin/env pythonX.Y and injects activate-this.py, but I haven't tested that idea. This issue has affected me a few times over the last several months, and I usually just end up manually tweaking the hashbangs back to #!/usr/bin/env python, then all is well.

I played around a little with translating a Cygwin setup into something usable under Cmd/MSYS, but that proved a little trickier, since Cygwin Python has that dll dependency. It's certainly feasible to strap that somewhere onto the virtualenv's PATH, though, so that might be the next step. I was using cygpath in a subprocess to translate things like /usr/local/bin/whatever into paths Windows/MSYS can understand. The issue I mentioned above is more apparent, though, since Cygwin Python relies much more heavily on that interpreter line without having exe binaries for entry points like Windows Python does.

Joys of working on Windows platform indeed.

Yes, thanks to the MSYS patch of virtualenvwrapper, my patch is largely based on that.

Nice fix to include the platform check in the activate shell script. My virtualenvwrapper patch will depend on this, otherwise it will get the path in the wrong format.

My fork of virtualenvwrapper is here: https://bitbucket.org/cliffxuan/virtualenvwrapper-for-cygwin-windows-python
Have not done many tests yet. Works on my machine ok. It requires two extra things:

  1. a CYGWIN_HOME environment variable pointing to the location of the Cygwin install path.
  2. your fix on activate script. (let's hope they will take it into the trunk version)

Interesting thought you have there about using Cygiwn python from Cmd/MSYS.

Have you thought about the TERM environment variable? This could make things easier if MSYS supports this, too. Cygwin seems fine:

$ python -c "import os; print os.environ.get('TERM')"
cygwin

(running on win32 in cygwin bash)

Contributor

greghaskins commented Oct 18, 2011

Unfortunately, TERM doesn't seem to be set by MSYS (at least, not on my system). Since I have both shells installed, the above just returns cygwin for me, even when run from the Windows python.exe in MSYS. Putting the OSTYPE check in the activate script works well enough, and has the added bonus of making the same virtualenv activatable from either shell environment.

@cliffxuan has a fork of virtualenvwrapper that takes advantage of the functionality provided by this pull request, as does my own [somewhat stale] fork, each using a slightly different approach. Until this patch (or something similar) makes it into virtualenv proper, any solution in this area will still feel kinda hack-y.

Even if TERM is set on MSYS, it's not reliable. TERM is the name of the terminal that you are using.

On windows command prompt, you get:
$ echo $TERM cygwin

On mintty:
$ echo $TERM xterm

On rxvt-native:
$ echo $TERM rxvt-cygwin-native

From screen:
$ echo $TERM screen

The patch based on OS and OSTYPE is working very well for me.
I use it with MSYS and with virtualenvwrapper on a daily basis.

I just wait for it to be merged to not have to manually patch virtualenv.

Contributor

carljm commented Dec 12, 2011

Merged this as part of #197. Thanks all for the work on this!

carljm closed this Dec 12, 2011

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment