Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pr 280 support windows #282

Merged
merged 27 commits into from
Jul 6, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
94e975f
Add Appveyor support
mdboom Jun 25, 2015
d9450cf
Fix up color printing on Windows
pv Jun 27, 2015
624009d
Allow multi-argument range spec for Git
pv Jun 27, 2015
913851b
Fix up util.check_output + util.which on Windows
pv Jun 27, 2015
65ac571
Fix up executable discovery for virtualenv & conda on Windows
pv Jun 27, 2015
efbde6f
Fix up memory benchmark test on Windows
pv Jun 27, 2015
7e5f51e
Fix issues in test_environment on windows
pv Jun 27, 2015
9344e17
Use colorama instead of psutil for tests, because psutil is not pure-…
pv Jun 27, 2015
7fa11ff
Work around the 260 char PATH_MAX on Windows
pv Jun 27, 2015
1b96efc
Fix up test failure in test_workflow on 32-bit Python 2.x
pv Jun 27, 2015
2f85a69
Fix right-aligning of benchmark results; Windows cmd terminal wraps o…
pv Jun 27, 2015
1362881
Fix multiprocessing test failures, when run via python setup.py test …
pv Jun 27, 2015
5fa41ed
Ensure setup.py test passes the given extra arguments to pytest
pv Jun 27, 2015
6d48594
Run pytest in appveyor with --tb=native to avoid pytest bug
pv Jun 27, 2015
1d84c83
Try harder to remove directories on Windows
pv Jun 27, 2015
c74a558
Fix division by zero in asv continuous
pv Jun 27, 2015
413fe9d
Disable colorama colors on Windows -- too many failure modes in practice
pv Jun 27, 2015
b685890
Mark a failing test on Windows/Appveyor as xfail
pv Jun 28, 2015
f353c7b
Skip multiprocessing using tests when run from py.test on Windows
pv Jun 28, 2015
cc6acca
Convert xrange to while loop, because xrange on 32-bit Python 2.x can…
pv Jun 28, 2015
35d8c99
Tell jquery to parse incoming data as json, for json files (asv previ…
pv Jun 28, 2015
9eda5f5
Fix up serving long path names on Windows for asv preview
pv Jun 28, 2015
aae7a03
Refactor Conda/Virtualenv by moving common helper methods to Environm…
pv Jun 28, 2015
0d2a125
Enable running all test_environment tests if conda is available
pv Jun 28, 2015
0f0ef02
Mark hglib requiring test as skipif rather than xfail
pv Jun 28, 2015
53457ec
Add test checking check_presence doesn't return obvious false negatives
pv Jun 28, 2015
d880953
Add comment explaining not using xrange + make Environment.setup priv…
pv Jul 6, 2015
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions .continuous-integration/appveyor/install-miniconda.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Sample script to install anaconda under windows
# Authors: Stuart Mumford
# Borrwed from: Olivier Grisel and Kyle Kastner
# License: BSD 3 clause

$MINICONDA_URL = "http://repo.continuum.io/miniconda/"

function DownloadMiniconda ($version, $platform_suffix) {
$webclient = New-Object System.Net.WebClient
$filename = "Miniconda-" + $version + "-Windows-" + $platform_suffix + ".exe"

$url = $MINICONDA_URL + $filename

$basedir = $pwd.Path + "\"
$filepath = $basedir + $filename
if (Test-Path $filename) {
Write-Host "Reusing" $filepath
return $filepath
}

# Download and retry up to 3 times in case of network transient errors.
Write-Host "Downloading" $filename "from" $url
$retry_attempts = 2
for($i=0; $i -lt $retry_attempts; $i++){
try {
$webclient.DownloadFile($url, $filepath)
break
}
Catch [Exception]{
Start-Sleep 1
}
}
if (Test-Path $filepath) {
Write-Host "File saved at" $filepath
} else {
# Retry once to get the error message if any at the last try
$webclient.DownloadFile($url, $filepath)
}
return $filepath
}

function InstallMiniconda ($python_version, $architecture, $python_home) {
Write-Host "Installing miniconda" $python_version "for" $architecture "bit architecture to" $python_home
if (Test-Path $python_home) {
Write-Host $python_home "already exists, skipping."
return $false
}
if ($architecture -eq "x86") {
$platform_suffix = "x86"
} else {
$platform_suffix = "x86_64"
}
$filepath = DownloadMiniconda $python_version $platform_suffix
Write-Host "Installing" $filepath "to" $python_home
$args = "/InstallationType=AllUsers /S /AddToPath=1 /RegisterPython=1 /D=" + $python_home
Write-Host $filepath $args
Start-Process -FilePath $filepath -ArgumentList $args -Wait -Passthru
#Start-Sleep -s 15
if (Test-Path C:\conda) {
Write-Host "Miniconda $python_version ($architecture) installation complete"
} else {
Write-Host "Failed to install Python in $python_home"
Exit 1
}
}

function main () {
InstallMiniconda $env:MINICONDA_VERSION $env:PLATFORM $env:PYTHON
}

main
47 changes: 47 additions & 0 deletions .continuous-integration/appveyor/windows_sdk.cmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
:: To build extensions for 64 bit Python 3, we need to configure environment
:: variables to use the MSVC 2010 C++ compilers from GRMSDKX_EN_DVD.iso of:
:: MS Windows SDK for Windows 7 and .NET Framework 4 (SDK v7.1)
::
:: To build extensions for 64 bit Python 2, we need to configure environment
:: variables to use the MSVC 2008 C++ compilers from GRMSDKX_EN_DVD.iso of:
:: MS Windows SDK for Windows 7 and .NET Framework 3.5 (SDK v7.0)
::
:: 32 bit builds do not require specific environment configurations.
::
:: Note: this script needs to be run with the /E:ON and /V:ON flags for the
:: cmd interpreter, at least for (SDK v7.0)
::
:: More details at:
:: https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
:: http://stackoverflow.com/a/13751649/163740
::
:: Author: Olivier Grisel
:: License: CC0 1.0 Universal: http://creativecommons.org/publicdomain/zero/1.0/
@ECHO OFF

SET COMMAND_TO_RUN=%*
SET WIN_SDK_ROOT=C:\Program Files\Microsoft SDKs\Windows

SET MAJOR_PYTHON_VERSION="%PYTHON_VERSION:~0,1%"
IF %MAJOR_PYTHON_VERSION% == "2" (
SET WINDOWS_SDK_VERSION="v7.0"
) ELSE IF %MAJOR_PYTHON_VERSION% == "3" (
SET WINDOWS_SDK_VERSION="v7.1"
) ELSE (
ECHO Unsupported Python version: "%MAJOR_PYTHON_VERSION%"
EXIT 1
)

IF "%PYTHON_ARCH%"=="64" (
ECHO Configuring Windows SDK %WINDOWS_SDK_VERSION% for Python %MAJOR_PYTHON_VERSION% on a 64 bit architecture
SET DISTUTILS_USE_SDK=1
SET MSSdk=1
"%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Setup\WindowsSdkVer.exe" -q -version:%WINDOWS_SDK_VERSION%
"%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Bin\SetEnv.cmd" /x64 /release
ECHO Executing: %COMMAND_TO_RUN%
call %COMMAND_TO_RUN% || EXIT 1
) ELSE (
ECHO Using default MSVC build environment for 32 bit architecture
ECHO Executing: %COMMAND_TO_RUN%
call %COMMAND_TO_RUN% || EXIT 1
)
47 changes: 47 additions & 0 deletions appveyor.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# AppVeyor.com is a Continuous Integration service to build and run tests under
# Windows

environment:

global:
PYTHON: "C:\\conda"
MINICONDA_VERSION: "latest"
CMD_IN_ENV: "cmd /E:ON /V:ON /C .\\.continuous-integration\\appveyor\\windows_sdk.cmd"

matrix:

- PYTHON_VERSION: "2.6"
platform: x64
PYTHON_ARCH: "64"
- PYTHON_VERSION: "2.7"
platform: x64
PYTHON_ARCH: "64"
- PYTHON_VERSION: "2.7"
platform: x86
PYTHON_ARCH: "32"
- PYTHON_VERSION: "3.4"
platform: x64
PYTHON_ARCH: "64"

install:
# Install miniconda using a powershell script.
- "powershell .continuous-integration/appveyor/install-miniconda.ps1"
- "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"

# Install the build and runtime dependencies of the project.
- "conda update --yes conda"
# Create a conda environment
- "conda create -q --yes -n test python=%PYTHON_VERSION%"
- "activate test"

# Check that we have the expected version of Python
- "python --version"

# Install specified version of dependencies
- "conda install -q --yes six pytest"

# Not a .NET project
build: false

test_script:
- "%CMD_IN_ENV% python setup.py test -a --tb=native"
8 changes: 7 additions & 1 deletion asv/benchmarks.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@
from . import util


WIN = (os.name == "nt")


# Can't use benchmark.__file__, because that points to the compiled
# file, so it can't be run by another version of Python.
BENCHMARK_RUN_SCRIPT = os.path.join(
Expand Down Expand Up @@ -75,7 +78,10 @@ def run_benchmark(benchmark, root, env, show_stderr=False, quick=False,
log.info(initial_message)

def log_result(msg):
padding = " "*(util.get_terminal_width() - len(initial_message) - 14 - 1 - len(msg))
padding_length = util.get_terminal_width() - len(initial_message) - 14 - 1 - len(msg)
if WIN:
padding_length -= 1
padding = " "*padding_length
log.add(" {0}{1}".format(padding, msg))

with log.indent():
Expand Down
9 changes: 8 additions & 1 deletion asv/commands/continuous.py
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,14 @@ def run(cls, conf, branch=None, base=None, factor=2.0, show_stderr=False, bench=
table = []
slowed_down = False
for name, benchmark in six.iteritems(all_benchmarks):
change = after[name] / before[name]
if before[name] == 0:
if after[name] == 0:
change = 1.0
else:
change = float('inf')
else:
change = after[name] / before[name]

if change > factor or change < 1.0 / factor:
table.append(
(change, before[name], after[name], name, benchmark))
Expand Down
7 changes: 6 additions & 1 deletion asv/commands/preview.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,12 @@ def run_from_conf_args(cls, conf, args):
def run(cls, conf, port=0, browser=False):
os.chdir(conf.html_dir)

Handler = SimpleHTTPServer.SimpleHTTPRequestHandler
class Handler(SimpleHTTPServer.SimpleHTTPRequestHandler):
def translate_path(self, path):
path = SimpleHTTPServer.SimpleHTTPRequestHandler.translate_path(
self, path)
return util.long_path(path)

httpd, base_url = create_httpd(Handler, port=port)

log.info("Serving at {0}".format(base_url))
Expand Down
2 changes: 1 addition & 1 deletion asv/commands/publish.py
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ def run(cls, conf):
log.set_nitems(5)

if os.path.exists(conf.html_dir):
shutil.rmtree(conf.html_dir)
util.long_path_rmtree(conf.html_dir)

benchmarks = Benchmarks.load(conf)

Expand Down
7 changes: 5 additions & 2 deletions asv/console.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@
from __future__ import (absolute_import, division, print_function,
unicode_literals)

import os
import warnings
import codecs
import contextlib
import locale
Expand All @@ -18,6 +20,8 @@
import six
from six.moves import xrange, input

WIN = (os.name == "nt")


def isatty(file):
"""
Expand Down Expand Up @@ -168,11 +172,10 @@ def color_print(*args, **kwargs):
"""

file = kwargs.get('file', sys.stdout)

end = kwargs.get('end', '')

write = file.write
if isatty(file):
if isatty(file) and not WIN:
for i in xrange(0, len(args), 2):
msg = args[i]
if i + 1 == len(args):
Expand Down
44 changes: 42 additions & 2 deletions asv/environment.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
import os
import shutil
import sys
import subprocess

import six

Expand All @@ -22,6 +23,9 @@
from . import wheel_cache


WIN = (os.name == "nt")


def iter_configuration_matrix(matrix):
"""
Iterate through all combinations of the given configuration
Expand Down Expand Up @@ -244,9 +248,20 @@ def check_presence(self):
'python': self._python,
'requirements': self._requirements
}

if info != expected_info:
return False

for executable in ['pip', 'python']:
exe = self.find_executable(executable)
if not os.path.isfile(exe):
return False

try:
self.run_executable('python', ['-c', 'pass'])
except (subprocess.CalledProcessError, OSError):
return False

return True

def create(self):
Expand All @@ -259,7 +274,7 @@ def create(self):

if not self.check_presence():
if os.path.exists(self._path):
shutil.rmtree(self._path)
util.long_path_rmtree(self._path)

if not os.path.exists(self._env_dir):
try:
Expand All @@ -279,7 +294,7 @@ def create(self):
except:
log.error("Failure creating environment for {0}".format(self.name))
if os.path.exists(self._path):
shutil.rmtree(self._path)
util.long_path_rmtree(self._path)
raise

self.save_info_file(self._path)
Expand Down Expand Up @@ -353,6 +368,31 @@ def can_install_project(self):
"""
return True

def find_executable(self, executable):
"""
Find an executable (eg. python, pip) in the environment.
"""

# Assume standard virtualenv/Conda layout
if WIN:
executable += ".exe"

exe = os.path.join(self._path, 'Scripts', executable)
if os.path.isfile(exe):
return exe
exe = os.path.join(self._path, executable)
if os.path.isfile(exe):
return exe

return os.path.join(self._path, 'bin', executable)

def run_executable(self, executable, args, **kwargs):
"""
Run a given executable (eg. python, pip) in the environment.
"""
exe = self.find_executable(executable)
return util.check_output([exe] + args, **kwargs)

def load_info_file(self, path):
path = os.path.join(path, 'asv-env-info.json')
return util.load_json(path)
Expand Down
8 changes: 7 additions & 1 deletion asv/graph.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,15 +116,21 @@ def resample_data(self, val):
max_time = max(x[0] for x in val)
step_size = int((max_time - min_time) / RESAMPLED_POINTS)

# This loop cannot use xrange, because xrange on Python2 on
# 32-bit systems can only deal with 32-bit integers, and
# Javascript timestamps (1000*unix_timestamp) handled here
# overflow this range
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The comment addresses this nicely.

new_val = []
j = 0
for i in xrange(min_time + step_size, max_time + step_size, step_size):
i = min_time + step_size
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this change? Less likely for a change to inadvertently fail the loop end condition when using xrange...

Copy link
Collaborator Author

@pv pv Jul 6, 2015 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

while i < max_time + step_size:
chunk = []
while j < len(val) and val[j][0] < i:
chunk.append(val[j][1])
j += 1
if len(chunk):
new_val.append((i, _mean_with_none(chunk)))
i += step_size
return new_val

def get_data(self):
Expand Down
Loading