port to win32 #113

Closed
wants to merge 1 commit into
from

Projects

None yet

5 participants

@mindw
mindw commented Jan 26, 2014

Really fast and brutal fix to some win32 issues

@mindw
mindw commented Feb 6, 2014

ping?

@bitzl
bitzl commented Feb 16, 2014

Windows support would be great!

@pfmoore
Contributor
pfmoore commented Feb 26, 2014

I would also like to see Windows support. Is there any particular issue with this patch - it looks OK to me from a quick glance?

I would be happy to contribute time to testing and working on Windows support if the project needs people with Windows expertise.

@Ivoz
Contributor
Ivoz commented Feb 27, 2014

https://github.com/kennethreitz/envoy might be helpful for subprocess running

@pfmoore pfmoore referenced this pull request in pypa/pip Feb 27, 2014
Merged

Consolidate builder into an Invoke Task #1596

@pfmoore
Contributor
pfmoore commented Feb 27, 2014

On Windows, a number of the tests fail. But this is likely to be because the tests are not written to be portable, and this PR does not update the tests. I can look at creating a PR to make the tests portable to Windows, but it's going to be a bit of work. One thing - can someone tell me how to run a subset of the tests (one test or one test file) under spec? The suggestions I've found for nose don't seem to work for me with spec - I am tending to get "0 tests run", for example with spec cli or spec cli.py or spec tests/cli.py or spec tests/cli (you get the idea...)

@bitprophet bitprophet commented on the diff Feb 27, 2014
invoke/runner.py
import select
import sys
-from .vendor import pexpect
-
+try:
+ from .vendor import pexpect
+except:
@bitprophet
bitprophet Feb 27, 2014 Member

What actual exception (or exceptions, if there's >1 likely) is pexpect throwing under Windows? Please try to catch that instead :)

@pfmoore
pfmoore Feb 27, 2014 Contributor

It'll be an import error - from the import of pty within pexpect - because pty doesn't exist on Windows. It might be better just to do a straight platform check round the pty stuff.

@bitprophet
bitprophet Feb 27, 2014 Member

I don't think we're modifying pexpect at all, just vendoring it, so I'd prefer to avoid touching it if possible. If these changes work well enough as-is and we can just change this line to except ImportError:, that's fine too. Just don't want a bare except.

@pfmoore
pfmoore Feb 27, 2014 Contributor

Sorry, what I meant was in runner.run, just do

if pty and sys.platform == 'win32': raise RuntimeError("ptys not supported")

Then there's no need to import pexpect at all, and we can skip it altogether on Windows.

I'm currently putting together an alternative patch that puts the platform-specific stuff in a single file. It might make for a cleaner structure for future work.

@bitprophet
bitprophet Feb 27, 2014 Member

Gotcha, and yea given the constraints of ptys on Windows, that makes sense. And big +1 on corralling platform specific stuff; again my philosophy for Windows support on my projects is basically "POSIX is 1st class citizen; Windows is 2nd class and lives off to the side". I don't like having to do that but it's the nature of the platforms & their distributions amongst the userbase.

@pfmoore
pfmoore Feb 27, 2014 Contributor

Sounds good. The only thought I have is that sometimes the "POSIX first class" philosophy can lead to non-portable code which could actually be trivially made portable but wasn't simply because nobody thought about portability. But it's up to us Windows users to keep you honest on that one :-)

@bitprophet bitprophet commented on the diff Feb 27, 2014
invoke/util.py
@@ -16,14 +14,40 @@
for x in ('debug',):
globals()[x] = getattr(log, x)
+def __pty_size_windows():
+ res=None
@bitprophet
bitprophet Feb 27, 2014 Member

needs spaces

@pfmoore
Contributor
pfmoore commented Feb 27, 2014

By the way, pip has switched to using invoke for some of its release process scripts, and I can confirm that with this patch, invoke works on Windows for that usage.

@bitprophet bitprophet commented on the diff Feb 27, 2014
.gitignore
@@ -2,3 +2,5 @@ docs/_build
.coverage
*.egg-info
.tox
+
@bitprophet
bitprophet Feb 27, 2014 Member

any particular reason for the blank line?

@mindw
mindw Feb 28, 2014

None. It's added automatically by GitExt.

@bitprophet bitprophet commented on the diff Feb 27, 2014
invoke/util.py
@@ -50,3 +74,6 @@ def pty_size():
except AttributeError:
pass
return cols, rows
+
+
+pty_size = __pty_size_windows if platform.system() == 'Windows' else __pty_size_unix
@bitprophet
bitprophet Feb 27, 2014 Member

My other codebases w/ Windows contributions seem to use sys.platform == 'win32' as their "is Windows" test. What's the salient difference between that and using platform.system?

@pfmoore
pfmoore Feb 27, 2014 Contributor

The platform module is newer, I think (but still pretty old - it's in Python 2.6). There may be subtleties around cygwin, but I don't use that so I'm not sure. Otherwise it's probably just what people are used to.

BTW, what versions of Python does invoke support? (Python 3.3 has os.get_terminal_size, not that that's much use for a few years yet :-))

@bitprophet
bitprophet Feb 27, 2014 Member

Says on the top of the doc index page (http://pyinvoke.org) that it supports 2.6+ and 3.2+ ;) so sadly Python 3-only calls are a no go.

@pfmoore
pfmoore Feb 27, 2014 Contributor

Wasn't expecting them to be... (Sorry, I should have checked the docs closer).

I've just uploaded an alternative PR #119 that moves the platform-specific stuff to a separate module. Let me know if you prefer that organisation to this.

@bitprophet
Member

Thanks for the contributions! Left a bunch of line comments, will wait for responses/changes.

@pfmoore Use spec --tests=tests/whatever.py, spec is still technically a nose plugin and that's a nose flag; it's what inv test uses: https://github.com/pyinvoke/invocations/blob/master/invocations/testing.py#L14

@bitprophet
Member

@pfmoore Oh, also, I would support updating the tests to be portable, but hopefully in a way that doesn't make them significantly more verbose/boilerplatey. I want to support Windows but I also don't want to sacrifice significant code clarity for it :(

If it's easier to just have a quick high level integration test file that can be run by Windows-using devs (esp given Travis-CI doesn't do Windows, there's no good way to have strong regression testing for the platform) then that might be a better approach.

@pfmoore
Contributor
pfmoore commented Feb 27, 2014

@bitprophet Thanks for the test invocation hint - not sure how I didn't try that variation!

Test portability would probably need to be a mix of skipping the ones that will never work on Windows (pty stuff), a bit of fiddling round things like line ending assumptions. Probably the biggest one would be the OS commands being run in the tests - they use bash syntax in a few places, and assume a standard Unix command set is available. That can be fixed, but it could be a bit intrusive - I'll see how it goes.

A short term approach might be to just have a high-level integration test for now, with phased changes to the main tests to improve coverage as we go along.

One other thing I spotted in the tests is the ANSI colour support. Windows doesn't use ANSI codes, so that'll show up as garbage. How unhappy would you be with changes to the way that's done? It could be a dependency on colorama (which does colour in a cross-platform way) or vendoring it, or some wrappers around an optional dependency, falling back to ANSI on Unix and black-and-white on Windows?

Maybe the best approach would be a "compatibility" module that encapsulated the stuff that differs between platforms: That would keep the impact on code clarity minimal.

@bitprophet
Member

@pfmoore:

  • Portability: The phased test support sounds reasonable, I'd like to prioritize getting the tool functional for Windows users, and wouldn't want "make the tests perfect on Windows" to hold that up.
  • Colors: I need to port more of these Invoke-labeled Fab tickets over, but see Fabric #101 for my previous thoughts on color - colorama is on that list IIRC so if it does this well that might be a +1 for it. EDIT: Looks like 'blessings' was the top-of-mind (lol business terms) last time I looked at that ticket; we should see whether it handles x-platform. But let's keep that discussion over there.
  • Compatibility module: as per my sub-comment higher up, my default is to treat POSIX as the primary implementation and have code branch off into Windows specific overrides as needed. I am not 100% wedded to that but IMO it makes things easier to reason about for what I see as the primary case, especially in situations where Windows-specific code is only in a few spots.
@bitprophet
Member

Merging this discussion into #119 so it's not in two places. Thanks!

@bitprophet bitprophet closed this Aug 25, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment