Py2 #23098

TheGoddessInari · 2019-03-26T05:48:31Z

da31023: Rework mach.bat to support VS2019 and user-supplied environments.
4551f60: Default mach.bat to using py -2.
03e4708: Don't assume the user's environment in mach_bootstrap.

./mach build -d does not report any errors
./mach test-tidy does not report any errors
These changes fix #23083 (GitHub issue number if applicable)

There are tests for these changes OR
These changes do not require tests because it changes the python bootstrap slightly and the changes are obvious.

The virtualenv changes shouldn't disrupt anything as it just uses the existing python (being used) to call itself -m virtualenv, and only if it exists. imp is still apparently the preferred builtin way to find this in Python 2.x without actually importing it. Importing it would cause an unused import warning in tidy. It still picks up the new things in _virtualenv, just no longer has a special case for Win32/MSYS because it's no longer needed.

The .bat change is the simplest I could think of that allows fallback in both cases and is no worse than before. where /Q is documented by Microsoft to return 0 if a successful match is found, and print nothing in either case.

This change is

highfive · 2019-03-26T05:48:37Z

Thanks for the pull request, and welcome! The Servo team is excited to review your changes, and you should hear from @Manishearth (or someone else) soon.

highfive · 2019-03-26T05:48:38Z

Heads up! This PR modifies the following files:

@wafflespeanut: python/mach_bootstrap.py

TheGoddessInari · 2019-03-26T13:14:05Z

I can't quite figure out what's up with the automated Trusty failures.

I've tried on minimal 16.04 and 18.04 VMs with python-virtualenv (which I believe both buildbots say they have) and it works fine.

I made a revision to match the same style with minimal changes and not require virtualenv 'early'.

TheGoddessInari · 2019-03-26T17:38:04Z

As I'm getting further and further into bootstrapping Servo, it looks like mach.bat needs to be reworked to support Visual Studio 2019, as well as not screw up the environment if one is already provided (whether VS Command Line, or manually provided).

Otherwise, it's simply too aggressive and clobbers the environment which causes even further quiet errors that are confusing to resolve.

TheGoddessInari · 2019-03-26T18:33:29Z

Failing any syntax errors, that should support both VS2019 and user-supplied environments from mach.bat. Bases an existing environment off of %VCINSTALLDIR%, same as cmake-rs.

There is technically a semantics change with using setlocal/endlocal in mach.bat, but it keeps the environment clean so is probably a net win. Would reinitialize the Visual Studio environment (if one isn't supplied) each time mach.bat is called, but also makes abundantly clear which compiler is being used.

jdm · 2019-03-29T16:19:56Z

These changes break using mach on linux:

$ ./mach test-tidy --no-progress --all
Python virtualenv is not installed. Please install it prior to running mach.
The command "./mach test-tidy --no-progress --all" exited with 1.

jdm

I'm fine with the goal of these changes, but I don't know enough about the low-level details of imports and modules and different versions to suggest how not to break our CI, unfortunately.

python/mach_bootstrap.py

jdm · 2019-03-30T02:28:04Z

@bors-servo try

bors-servo · 2019-03-30T02:28:09Z

⌛ Trying commit 86bb79e with merge 95f2587...


        Auto merge of #23098 - TheGoddessInari:py2, r=<try>

Py2  86bb79e: Rework mach.bat to support VS2019 and user-supplied environments. f6bd2d2: Default mach.bat to using py -2. 81d48c9: Don't assume the user's environment in mach_bootstrap. ---  - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix #23083 (GitHub issue number if applicable)  - [ ] There are tests for these changes OR - [X] These changes do not require tests because it changes the python bootstrap slightly and the changes are obvious.   The virtualenv changes shouldn't disrupt anything as it just uses the existing python (being used) to call itself `-m virtualenv`, and only if it exists. imp is still apparently the preferred builtin way to find this in Python 2.x without actually importing it. Importing it would cause an unused import warning in tidy. It still picks up the new things in _virtualenv, just no longer has a special case for Win32/MSYS because it's no longer needed. The .bat change is the simplest I could think of that allows fallback in both cases and is no worse than before. `where /Q` is documented by Microsoft to return 0 if a successful match is found, and print nothing in either case.  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/23098)

jdm · 2019-03-30T02:34:26Z

Presumably we will need:

this to change to follow this instead
this to run a pip install instead

Depending on the results of the try run, we may need to adjust https://github.com/servo/saltfs/ for the rest of our CI as well.

jdm · 2019-03-30T03:15:09Z

Windows taskcluster:

C:\Users\task_1553915082\repo>mach fetch 
Visual Studio 2015, 2017, or 2019 is not installed.

TheGoddessInari · 2019-03-30T03:46:19Z

Cleaned up the mach issue (I forgot about the environment), as well as the requested change on mach_bootstrap.py.


        Don't assume the user's environment in mach_bootstrap.

On Windows with multiple Pythons installed, this was causing python2.7 to bootstrap a 3.7 virtualenv that it couldn't make use of. PIP_NAMES wasn't used at all, and VIRTUALENV_NAMES ends up being unused now.


        Default mach.bat to using py -2.

Servo already assumes the user has Python, this is the primary way to make sure that Python 2 is preferred, and you should get a sensible error message if you have python 3 but not 2. But first, check that py.exe exists because if a system has only Python 2.x only, it won't have it. If it doesn't, then try python.exe.


        Rework mach.bat to support VS2019 and user-supplied environments.

As a bonus, use setlocal to avoid environment pollution.

jdm · 2019-04-01T15:29:29Z

Hmm, even sudo pip install virtualenv on travisci is not making the module import check pass :<

bors-servo · 2019-04-01T19:59:38Z

⌛ Trying commit dee8803 with merge 1760da1...


        Auto merge of #23098 - TheGoddessInari:py2, r=<try>

Py2  da31023: Rework mach.bat to support VS2019 and user-supplied environments. 4551f60: Default mach.bat to using py -2. 03e4708: Don't assume the user's environment in mach_bootstrap. ---  - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix #23083 (GitHub issue number if applicable)  - [ ] There are tests for these changes OR - [X] These changes do not require tests because it changes the python bootstrap slightly and the changes are obvious.   The virtualenv changes shouldn't disrupt anything as it just uses the existing python (being used) to call itself `-m virtualenv`, and only if it exists. imp is still apparently the preferred builtin way to find this in Python 2.x without actually importing it. Importing it would cause an unused import warning in tidy. It still picks up the new things in _virtualenv, just no longer has a special case for Win32/MSYS because it's no longer needed. The .bat change is the simplest I could think of that allows fallback in both cases and is no worse than before. `where /Q` is documented by Microsoft to return 0 if a successful match is found, and print nothing in either case.  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/23098)

bors-servo · 2019-04-01T21:07:08Z

💔 Test failed - mac-rel-wpt2


        Install virtualenv python module.

jdm · 2019-04-01T21:13:55Z

@bors-servo r+

bors-servo · 2019-04-01T21:13:56Z

📌 Commit c0052c0 has been approved by jdm

bors-servo · 2019-04-01T21:14:40Z

⌛ Testing commit c0052c0 with merge 01fc4a0...


        Auto merge of #23098 - TheGoddessInari:py2, r=jdm

Py2  da31023: Rework mach.bat to support VS2019 and user-supplied environments. 4551f60: Default mach.bat to using py -2. 03e4708: Don't assume the user's environment in mach_bootstrap. ---  - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix #23083 (GitHub issue number if applicable)  - [ ] There are tests for these changes OR - [X] These changes do not require tests because it changes the python bootstrap slightly and the changes are obvious.   The virtualenv changes shouldn't disrupt anything as it just uses the existing python (being used) to call itself `-m virtualenv`, and only if it exists. imp is still apparently the preferred builtin way to find this in Python 2.x without actually importing it. Importing it would cause an unused import warning in tidy. It still picks up the new things in _virtualenv, just no longer has a special case for Win32/MSYS because it's no longer needed. The .bat change is the simplest I could think of that allows fallback in both cases and is no worse than before. `where /Q` is documented by Microsoft to return 0 if a successful match is found, and print nothing in either case.  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/23098)

bors-servo · 2019-04-01T21:43:27Z

💔 Test failed - linux-rel-wpt

jdm · 2019-04-01T22:48:06Z

@bors-servo retry

#13480


        Auto merge of #23098 - TheGoddessInari:py2, r=jdm

Py2  da31023: Rework mach.bat to support VS2019 and user-supplied environments. 4551f60: Default mach.bat to using py -2. 03e4708: Don't assume the user's environment in mach_bootstrap. ---  - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix #23083 (GitHub issue number if applicable)  - [ ] There are tests for these changes OR - [X] These changes do not require tests because it changes the python bootstrap slightly and the changes are obvious.   The virtualenv changes shouldn't disrupt anything as it just uses the existing python (being used) to call itself `-m virtualenv`, and only if it exists. imp is still apparently the preferred builtin way to find this in Python 2.x without actually importing it. Importing it would cause an unused import warning in tidy. It still picks up the new things in _virtualenv, just no longer has a special case for Win32/MSYS because it's no longer needed. The .bat change is the simplest I could think of that allows fallback in both cases and is no worse than before. `where /Q` is documented by Microsoft to return 0 if a successful match is found, and print nothing in either case.  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/23098)

bors-servo · 2019-04-01T22:48:11Z

⌛ Testing commit c0052c0 with merge 3340214...

bors-servo · 2019-04-02T00:22:07Z

☀️ Test successful - android-mac, arm32, arm64, linux-rel-css, linux-rel-wpt, mac-rel-css1, mac-rel-css2, mac-rel-wpt1, mac-rel-wpt2, mac-rel-wpt3, mac-rel-wpt4, magicleap, status-taskcluster
Approved by: jdm
Pushing 3340214 to master...


        Auto merge of #24051 - TheGoddessInari:fix_windows, r=<try>

build_commands: Make Visual Studio detection not assume environment  Ever since 3dd1bfc, servo breaks building if you're not running Visual Studio 2015 or 2017 in the default location on C:. I rewrote the PR to not assume environment, similarly to #23098. We need to stop meeting like this. 🦊 ---  - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy` does not report any errors - [ ] These changes fix #___ (GitHub issue number if applicable)  - [ ] There are tests for these changes OR - [ ] These changes do not require tests because 3dd1bfc didn't. Realistically, there should be environment tests for Visual Studio 2015, 2017, and 2019, but I won't have the time for it soon.    --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/24051)


        Auto merge of #24051 - TheGoddessInari:fix_windows, r=<try>

build_commands: Make Visual Studio detection not assume environment  Ever since 3dd1bfc, servo breaks building if you're not running Visual Studio 2015 or 2017 in the default location on C:. I rewrote the PR to not assume environment, similarly to #23098. We need to stop meeting like this. 🦊 ---  - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy` does not report any errors - [ ] These changes fix #___ (GitHub issue number if applicable)  - [ ] There are tests for these changes OR - [ ] These changes do not require tests because 3dd1bfc didn't. Realistically, there should be environment tests for Visual Studio 2015, 2017, and 2019, but I won't have the time for it soon.    --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/24051)


        Auto merge of #24051 - TheGoddessInari:fix_windows, r=<try>

build_commands: Make Visual Studio detection not assume environment  Ever since 681d7b1, servo breaks building if you're not running Visual Studio 2015 or 2017 in the default location on C:. I rewrote the PR to not assume environment, similarly to #23098. We need to stop meeting like this. 🦊 ---  - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy` does not report any errors - [ ] These changes fix #___ (GitHub issue number if applicable)  - [ ] There are tests for these changes OR - [ ] These changes do not require tests because 681d7b1 didn't. Realistically, there should be environment tests for Visual Studio 2015, 2017, and 2019, but I won't have the time for it soon.    --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/24051)


        Auto merge of #24051 - TheGoddessInari:fix_windows, r=jdm

build_commands: Make Visual Studio detection not assume environment  Ever since 681d7b1, servo breaks building if you're not running Visual Studio 2015 or 2017 in the default location on C:. I rewrote the PR to not assume environment, similarly to #23098. We need to stop meeting like this. 🦊 ---  - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy` does not report any errors - [ ] These changes fix #___ (GitHub issue number if applicable)  - [ ] There are tests for these changes OR - [ ] These changes do not require tests because 681d7b1 didn't. Realistically, there should be environment tests for Visual Studio 2015, 2017, and 2019, but I won't have the time for it soon.    --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/24051)

highfive assigned Manishearth Mar 26, 2019

highfive added the S-awaiting-review label Mar 26, 2019

TheGoddessInari force-pushed the TheGoddessInari:py2 branch from a5fe296 to fe2eddb Mar 26, 2019

TheGoddessInari force-pushed the TheGoddessInari:py2 branch 2 times, most recently from 61a6b53 to 86bb79e Mar 26, 2019

jdm requested changes Mar 29, 2019

View changes

python/mach_bootstrap.py Outdated Show resolved Hide resolved

python/mach_bootstrap.py Show resolved Hide resolved

jdm assigned jdm and unassigned Manishearth Mar 29, 2019

jdm added S-needs-code-changes and removed S-awaiting-review labels Mar 29, 2019

TheGoddessInari force-pushed the TheGoddessInari:py2 branch from 86bb79e to da31023 Mar 30, 2019

highfive added S-awaiting-review and removed S-needs-code-changes labels Mar 30, 2019

TheGoddessInari added 3 commits Mar 26, 2019

Don't assume the user's environment in mach_bootstrap.

44162ce

On Windows with multiple Pythons installed, this was causing python2.7 to bootstrap a 3.7 virtualenv that it couldn't make use of. PIP_NAMES wasn't used at all, and VIRTUALENV_NAMES ends up being unused now.

Rework mach.bat to support VS2019 and user-supplied environments.

9037291

As a bonus, use setlocal to avoid environment pollution.

jdm force-pushed the TheGoddessInari:py2 branch from 06a70d3 to 09bc83f Apr 1, 2019

highfive added the S-tests-failed label Apr 1, 2019

Install virtualenv python module.

Loading status checks…

c0052c0

jdm force-pushed the TheGoddessInari:py2 branch from dee8803 to c0052c0 Apr 1, 2019

highfive removed the S-tests-failed label Apr 1, 2019

highfive added S-awaiting-merge and removed S-awaiting-review labels Apr 1, 2019

highfive added S-tests-failed and removed S-awaiting-merge labels Apr 1, 2019

highfive added S-awaiting-merge and removed S-tests-failed labels Apr 1, 2019

highfive removed the S-awaiting-merge label Apr 2, 2019

jdm mentioned this pull request Apr 16, 2019

Add virtualenv installation to build documentation. #23212

Closed

3 of 3 tasks complete

TheGoddessInari mentioned this pull request Aug 26, 2019

build_commands: Make Visual Studio detection not assume environment #24051

Merged

2 of 5 tasks complete

servo / servo

Py2 #23098

Py2 #23098

TheGoddessInari commented Mar 26, 2019 •

edited

highfive commented Mar 26, 2019

highfive commented Mar 26, 2019

TheGoddessInari commented Mar 26, 2019

TheGoddessInari commented Mar 26, 2019

TheGoddessInari commented Mar 26, 2019

jdm commented Mar 29, 2019

jdm left a comment

jdm commented Mar 30, 2019

bors-servo commented Mar 30, 2019

jdm commented Mar 30, 2019

jdm commented Mar 30, 2019

TheGoddessInari commented Mar 30, 2019

jdm commented Apr 1, 2019

bors-servo commented Apr 1, 2019

bors-servo commented Apr 1, 2019

jdm commented Apr 1, 2019

bors-servo commented Apr 1, 2019

bors-servo commented Apr 1, 2019

bors-servo commented Apr 1, 2019

jdm commented Apr 1, 2019

bors-servo commented Apr 1, 2019

bors-servo commented Apr 2, 2019

servo / servo

Join GitHub today

Py2 #23098

Py2 #23098

Conversation

TheGoddessInari commented Mar 26, 2019 • edited

highfive commented Mar 26, 2019

highfive commented Mar 26, 2019

TheGoddessInari commented Mar 26, 2019

TheGoddessInari commented Mar 26, 2019

TheGoddessInari commented Mar 26, 2019

jdm commented Mar 29, 2019

jdm left a comment

jdm commented Mar 30, 2019

bors-servo commented Mar 30, 2019

jdm commented Mar 30, 2019

jdm commented Mar 30, 2019

TheGoddessInari commented Mar 30, 2019

jdm commented Apr 1, 2019

bors-servo commented Apr 1, 2019

bors-servo commented Apr 1, 2019

jdm commented Apr 1, 2019

bors-servo commented Apr 1, 2019

bors-servo commented Apr 1, 2019

bors-servo commented Apr 1, 2019

jdm commented Apr 1, 2019

bors-servo commented Apr 1, 2019

bors-servo commented Apr 2, 2019

TheGoddessInari commented Mar 26, 2019 •

edited