Mach bootstrap: Improve and support more platforms #15230

UK992 · 2017-01-26T00:57:40Z

r? @aneeshusa or @wafflespeanut

./mach build -d does not report any errors
./mach test-tidy does not report any errors
These changes fix #__ (github issue number if applicable).

There are tests for these changes OR
These changes do not require tests because _____

This change is

highfive · 2017-01-26T00:57:43Z

Heads up! This PR modifies the following files:

@wafflespeanut: python/servo/bootstrap.py

aneeshusa

Core concept is good, but many of the additions have dubious usefulness IMO.

aneeshusa · 2017-01-26T01:18:00Z

python/servo/bootstrap.py

@@ -219,7 +247,8 @@ def bootstrap(context, force=False):
        bootstrapper = windows_msvc
    elif "linux-gnu" in host_triple():
        distro, version, _ = platform.linux_distribution()
-        if distro == 'Ubuntu' and version == '14.04':
+        if distro in ('CentOS', 'CentOS Linux', 'Fedora', 'Ubuntu'):


After bootstrapping, is it possible to build Servo even without the MS fonts?

Yes, it's possible, tested on Ubuntu 16.04.

Same comment about using a set or list here instead of a tuple.

aneeshusa · 2017-01-26T01:18:00Z

python/servo/bootstrap.py

-        return 1
-    print("done")
+    virtualenv_script_dir = os.path.join(context.topdir, 'python', '_virtualenv', _get_virtualenv_script_dir())
+    if not os.path.exists(os.path.join(virtualenv_script_dir, 'salt-call')):


This is not quite right, since from time to time we will need to update Salt to stay in locksetep with saltfs. I'd prefer to continue running pip install every time, which is a (fast) no-op if the dependencies are up to date.

I reverted this change.

aneeshusa · 2017-01-26T01:18:00Z

python/servo/bootstrap.py

-        print('Output: {}\nError: {}'.format(out, err))
-        return 1
-    print("done")
+    virtualenv_script_dir = os.path.join(context.topdir, 'python', '_virtualenv', _get_virtualenv_script_dir())


It would be nice to add an absolute=True (optional, kwarg?) arg to _get_virtualenv_script_dir so thatthe path prefixes don't need to be appended here and in mach_bootstrap.py.

aneeshusa · 2017-01-26T01:18:00Z

python/servo/bootstrap.py

+        if process.returncode:
+            out, err = process.communicate()
+            if "pycrypto/setup.py" in err and not retry:
+                return install_salt_dependencies(context, force)


I think it makes more sense to do this before running pip install, to avoid having to call pip install twice.

since install_salt_dependencies only run, when failed to install pycrypto dependency of Salt, probably one time run, i think it's better run pip install twice, then install_salt_dependencies on every run.

aneeshusa · 2017-01-26T01:18:00Z

python/servo/bootstrap.py

+            subprocess.check_call(['sudo', '-v'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+            command.insert(0, 'sudo')
+        except:
+            command = ['su', 'root', '-c', ' '.join(command)]


Where is su needed? I much prefer using sudo in all circumstances, since su 1) uses root's environment instead of the user's environment and 2) requires the root password instead of the user's password.

On Fedora with user account without password, sudoing is not possible.

IMO that's an insecure/broken setup (e.g. another sudo benefit is its auditing support), and I'd rather not support it. Additionally, the su codepath does not handle quoting/escaping of the command that it passes to bash, which is also easy to write incorrectly. Users with such setups can always install dependencies manually.

Also, in case a user uses sudo with timestamp_timeout set to 0, the additional sudo -v call means they will have twice as many sudo prompts.

aneeshusa · 2017-01-26T01:18:00Z

python/servo/bootstrap.py

+    return subprocess.call(command)
+
+
+# This install needed dependencies for Salt


nit: remove, comment is redundant with function name

wafflespeanut · 2017-02-04T03:31:01Z

Review ping @aneeshusa

aneeshusa

Looking better, thanks for this PR! I have some more review comments.

aneeshusa · 2017-02-07T08:13:45Z

python/mach_bootstrap.py

@@ -96,11 +96,12 @@ def _get_exec_path(names, is_valid_path=lambda _path: True):
    return None


-def _get_virtualenv_script_dir():
+def _get_virtualenv_script_dir(absolute=True):


We're no longer using this for the Salt bootstrap, so let's back out the change to this function and its call site in _activate_virtualenv.

aneeshusa · 2017-02-07T08:13:45Z

python/servo/bootstrap.py

+            subprocess.check_call(['sudo', '-v'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+            command.insert(0, 'sudo')
+        except:
+            command = ['su', 'root', '-c', ' '.join(command)]


IMO that's an insecure/broken setup (e.g. another sudo benefit is its auditing support), and I'd rather not support it. Additionally, the su codepath does not handle quoting/escaping of the command that it passes to bash, which is also easy to write incorrectly. Users with such setups can always install dependencies manually.

Also, in case a user uses sudo with timestamp_timeout set to 0, the additional sudo -v call means they will have twice as many sudo prompts.

aneeshusa · 2017-02-07T08:13:45Z

python/servo/bootstrap.py

+    print("Installing missing Salt dependencies...")
+    if context.distro == 'Ubuntu':
+        run_as_root(['apt-get', 'install', '-y', 'build-essential', 'libssl-dev', 'libffi-dev', 'python-dev'])
+    elif context.distro in ('CentOS', 'CentOS Linux', 'Fedora'):


Use a set or list instead of a tuple.

aneeshusa · 2017-02-07T08:13:45Z

python/servo/bootstrap.py

@@ -219,7 +247,8 @@ def bootstrap(context, force=False):
        bootstrapper = windows_msvc
    elif "linux-gnu" in host_triple():
        distro, version, _ = platform.linux_distribution()
-        if distro == 'Ubuntu' and version == '14.04':
+        if distro in ('CentOS', 'CentOS Linux', 'Fedora', 'Ubuntu'):


Same comment about using a set or list here instead of a tuple.

aneeshusa · 2017-02-07T08:13:45Z

python/servo/bootstrap.py

+    return subprocess.call(command)
+
+
+def install_salt_dependencies(context, force):


This method should use the force parameter as it is used elsewhere: if force is False, then it should prompt the user before installing packages.

aneeshusa · 2017-02-07T08:13:45Z

python/servo/bootstrap.py

@@ -29,6 +51,9 @@ def salt(context, force=False):
    process.wait()
    if process.returncode:
        out, err = process.communicate()
+        if "pycrypto/setup.py" in err and not retry:
+            return install_salt_dependencies(context, force)


Hmm, I don't know where my previous comment about this went.

Checking for a substring of the error message in this way seems fragile to me; Salt has changed crypto libraries in the past.

The mutual recursion here is error-prone as well; if in the future there is a new system-level dependency for Salt, then this code won't install it, causing the error on each Salt invocation and endlessly recursing through salt and install_salt_dependencies in a loop, causing the process to hang.

I still think it's better to check for the the Salt dependencies on each run before installing Salt. We can make this faster by using dpkg -s <package> <names> <here> first, and only running apt-get install if it returns nonzero; I'm not sure about the CentOS/Fedora equivalent.


        Mach bootstrap: Improve and support more platforms

UK992 · 2017-02-08T18:57:13Z

All comments addressed!

jdm · 2017-02-27T18:37:34Z

@aneeshusa Review ping!

wafflespeanut · 2017-03-04T12:47:09Z

This has been around for way too long. I'd better land this.

@bors-servo r=aneeshusa

bors-servo · 2017-03-04T12:47:10Z

📌 Commit f21d448 has been approved by aneeshusa

bors-servo · 2017-03-04T12:47:14Z

⌛ Testing commit f21d448 with merge d8a8e3e...


        Auto merge of #15230 - UK992:mach-bootstrap, r=aneeshusa

@aneeshusa

Mach bootstrap: Improve and support more platforms  r? @aneeshusa or @wafflespeanut ---  - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy` does not report any errors - [ ] These changes fix #__ (github issue number if applicable).  - [ ] There are tests for these changes OR - [ ] These changes do not require tests because _____   --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/15230)

bors-servo · 2017-03-04T13:33:36Z

☀️ Test successful - android, arm32, arm64, linux-dev, linux-rel-css, linux-rel-wpt, mac-dev-unit, mac-rel-css, mac-rel-wpt1, mac-rel-wpt2, windows-gnu-dev, windows-msvc-dev
Approved by: aneeshusa
Pushing d8a8e3e to master...

highfive assigned aneeshusa Jan 26, 2017

highfive added the S-awaiting-review label Jan 26, 2017

aneeshusa requested changes Jan 26, 2017

View changes

aneeshusa added S-needs-code-changes and removed S-awaiting-review labels Jan 26, 2017

highfive added S-awaiting-review and removed S-needs-code-changes labels Jan 26, 2017

jdm mentioned this pull request Feb 2, 2017

Build Failing #12356

Closed

aneeshusa requested changes Feb 7, 2017

View changes

aneeshusa added S-needs-code-changes and removed S-awaiting-review labels Feb 7, 2017

UK992 force-pushed the UK992:mach-bootstrap branch from 5c54f6a to ab98e8c Feb 7, 2017

highfive added S-awaiting-review and removed S-needs-code-changes labels Feb 7, 2017

UK992 force-pushed the UK992:mach-bootstrap branch 2 times, most recently from 439f9e1 to 226f16a Feb 7, 2017

Mach bootstrap: Improve and support more platforms

Loading status checks…

f21d448

UK992 force-pushed the UK992:mach-bootstrap branch from 226f16a to f21d448 Feb 8, 2017

highfive added S-awaiting-merge and removed S-awaiting-review labels Mar 4, 2017

bors-servo merged commit f21d448 into servo:master Mar 4, 2017
3 checks passed

3 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details

continuous-integration/travis-ci/pr The Travis CI build passed
Details

homu Test successful
Details

highfive removed the S-awaiting-merge label Mar 4, 2017

servo / servo

Mach bootstrap: Improve and support more platforms #15230

Mach bootstrap: Improve and support more platforms #15230

UK992 commented Jan 26, 2017 •

edited by larsbergstrom

highfive commented Jan 26, 2017

aneeshusa left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

wafflespeanut commented Feb 4, 2017

aneeshusa left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

UK992 commented Feb 8, 2017

jdm commented Feb 27, 2017

wafflespeanut commented Mar 4, 2017

bors-servo commented Mar 4, 2017

bors-servo commented Mar 4, 2017

bors-servo commented Mar 4, 2017

		return subprocess.call(command)


		# This install needed dependencies for Salt

		return subprocess.call(command)


		def install_salt_dependencies(context, force):

servo / servo

Join GitHub today

Mach bootstrap: Improve and support more platforms #15230

Mach bootstrap: Improve and support more platforms #15230

Conversation

UK992 commented Jan 26, 2017 • edited by larsbergstrom

highfive commented Jan 26, 2017

aneeshusa left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

wafflespeanut commented Feb 4, 2017

aneeshusa left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

UK992 commented Feb 8, 2017

jdm commented Feb 27, 2017

wafflespeanut commented Mar 4, 2017

bors-servo commented Mar 4, 2017

bors-servo commented Mar 4, 2017

bors-servo commented Mar 4, 2017

UK992 commented Jan 26, 2017 •

edited by larsbergstrom