New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make bootstrap handle linux setup, include gstreamer, make bootstrap run standalone #21586

Merged
merged 18 commits into from Sep 12, 2018

Conversation

Projects
None yet
4 participants
@Manishearth
Member

Manishearth commented Sep 3, 2018

Previously ./mach bootstrap wasn't very useful on Linux, it would just install salt. This PR moves that to ./mach bootstrap-salt, and has ./mach bootstrap perform the installation steps mentioned in the README.

With these changes ./mach bootstrap is a valid way of setting up the servo build. Additionally, this adds gstreamer setup to this for cases where gstreamer is not up to date.

Edit: And now, mach bootstrap is able to be run on a fresh system with just python installed (no need for virtualenv or libffi).

Fixes #21446 , fixes #21417

r? @jdm


This change is Reviewable

@highfive

This comment has been minimized.

highfive commented Sep 3, 2018

Heads up! This PR modifies the following files:

  • @wafflespeanut: python/servo/bootstrap.py, python/servo/command_base.py, python/servo/bootstrap_commands.py, python/servo/testing_commands.py
@@ -0,0 +1,4 @@
wget https://github.com/ferjm/gstreamer-1.14.1-ubuntu-trusty/raw/master/gstreamer.tar.gz -O gstreamer.tar.gz

This comment has been minimized.

@Manishearth

Manishearth Sep 3, 2018

Member

Should I just make this part of the python file?

Also, should we try to host these binaries on our s3 bucket? (I don't know how, but I can ask)

This comment has been minimized.

@jdm

jdm Sep 5, 2018

Member

Yeah, s3 would be preferable. I've uploaded stuff there before; all you need is the credentials and the appropriate aws s3 command from the terminal.

@Manishearth Manishearth force-pushed the Manishearth:gstreamer-bootstrap branch from 5effef5 to 0a6ef61 Sep 3, 2018

else:
bootstrapper = linux
else:
raise Exception("mach bootstrap does not support %s, please file a bug" % distro)

This comment has been minimized.

@Manishearth

Manishearth Sep 3, 2018

Member

I'll open an issue for filling this in for macs

@Manishearth Manishearth force-pushed the Manishearth:gstreamer-bootstrap branch 4 times, most recently from bc77075 to f72ba55 Sep 3, 2018

@Manishearth

This comment has been minimized.

Member

Manishearth commented Sep 3, 2018

An annoying bit is that mach itself is pretty heavyweight and needs a bunch of deps like libffi to work. I can try and make mach bootstrap work independently of mach, if that is desired, that way folks just have to have rust installed and run ./mach bootstrap.

(I'm currently testing this on a VM)

@Manishearth

This comment has been minimized.

Member

Manishearth commented Sep 3, 2018

I've got the independent bootstrap working in my bootstrap-next branch but I'm still working out some small issues.

I wonder if the gstreamer setup should be in ./mach bootstrap or mach_bootstrap.py. Stuff is kinda scattered between the two, currently. I feel like it belongs in ./mach bootstrap because while the support/linux/gstreamer based setup doesn't require sudo, the apt-get one does, and mach_bootstrap.py is all about silent non-sudo installation.

@Manishearth Manishearth force-pushed the Manishearth:gstreamer-bootstrap branch from f72ba55 to d629f3b Sep 3, 2018

@Manishearth

This comment has been minimized.

Member

Manishearth commented Sep 3, 2018

Merged in the branch. We can now recommend just running ./mach bootstrap for Ubuntu and Fedora.

We don't install rust, though (but mach bootstrap could if we wanted it to).

Also found out a couple trusty-only dependency changes from the spidermonkey upgrade, which are also now installed via bootstrap.

@Manishearth Manishearth changed the title from Make bootstrap handle linux setup, include gstreamer to Make bootstrap handle linux setup, include gstreamer, make bootstrap run standalone Sep 3, 2018

@Manishearth

This comment has been minimized.

Member

Manishearth commented Sep 4, 2018

@bors-servo try=linux

bors-servo added a commit that referenced this pull request Sep 4, 2018

Auto merge of #21586 - Manishearth:gstreamer-bootstrap, r=<try>
Make bootstrap handle linux setup, include gstreamer, make bootstrap run standalone

Previously `./mach bootstrap` wasn't very useful on Linux, it would just install salt. This PR moves that to `./mach bootstrap-salt`, and has `./mach bootstrap` perform the installation steps mentioned in the README.

With these changes `./mach bootstrap` is a valid way of setting up the servo build. Additionally, this adds gstreamer setup to this for cases where gstreamer is not up to date.

Edit: And now, mach bootstrap is able to be run on a fresh system with just python installed (no need for virtualenv or libffi).

Fixes #21446 , fixes #21417

r? @jdm

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/21586)
<!-- Reviewable:end -->
@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Sep 4, 2018

⌛️ Trying commit c91594c with merge 25f9fa8...

@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Sep 4, 2018

☀️ Test successful - linux-dev
State: approved= try=True

@Manishearth

This comment has been minimized.

Member

Manishearth commented Sep 4, 2018

wow, try=linux is fast

@jdm

These changes look like a really nice improvement for common pain points. Thanks!

if specific == "salt":
bootstrapper = salt
elif specific == "gstreamer":
bootstrapper = gstreamer

This comment has been minimized.

@jdm

jdm Sep 5, 2018

Member
boostrappers = {
    "salt": salt,
     "gstreamer": gstreamer,
}
bootstrapper = bootstrappers.get(specific, linux)
gstdir = os.path.join(cur, "support", "linux", "gstreamer")
if not os.path.isdir(os.path.join(gstdir, "gstreamer", "lib")):
os.chdir(gstdir)
subprocess.call(["bash", "gstreamer.sh"])

This comment has been minimized.

@jdm

jdm Sep 5, 2018

Member

Use check_call, and pass a cwd argument instead of calling chdir explicitly.

@@ -59,6 +59,7 @@ files = [
"./resources/hsts_preload.json",
"./tests/wpt/metadata/MANIFEST.json",
"./support/android/openssl.sh",
"./support/linux/gstreamer/gstreamer.sh",

This comment has been minimized.

@jdm

jdm Sep 5, 2018

Member

Why not make this script pass the check instead?

This comment has been minimized.

@Manishearth

Manishearth Sep 5, 2018

Member

It can't, it contains giant URLs that can't really be split up further. Most of its lines are too long anyway, the tidy script doesn't get us much

@@ -0,0 +1,4 @@
wget https://github.com/ferjm/gstreamer-1.14.1-ubuntu-trusty/raw/master/gstreamer.tar.gz -O gstreamer.tar.gz

This comment has been minimized.

@jdm

jdm Sep 5, 2018

Member

Yeah, s3 would be preferable. I've uploaded stuff there before; all you need is the credentials and the appropriate aws s3 command from the terminal.

@@ -477,6 +477,23 @@ def get_nightly_binary_path(self, nightly_date):
bin_folder = path.join(destination_folder, "PFiles", "Mozilla research", "Servo Tech Demo")
return path.join(bin_folder, "servo{}".format(BIN_SUFFIX))
def needs_gstreamer_env(self, target):
gstpath = path.join(self.context.topdir, "support", "linux", "gstreamer", "gstreamer")

This comment has been minimized.

@jdm

jdm Sep 5, 2018

Member

Let's make a separate helper for this path instead of duplicating it in three places.

'/usr/bin/g++-4.9'])
if clang:
run_as_root(["bash", "-c", 'wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -'])
run_as_root(["apt-add-repository" "deb http://apt.llvm.org/trusty/ llvm-toolchain-xenial-4.0 main"], force)

This comment has been minimized.

@jdm

jdm Sep 5, 2018

Member

Missing comma(s?)

@Manishearth

This comment has been minimized.

Member

Manishearth commented Sep 6, 2018

Updated

@jdm

This comment has been minimized.

Member

jdm commented Sep 6, 2018

@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Sep 6, 2018

📌 Commit 19f74d5 has been approved by jdm

@Manishearth

This comment has been minimized.

Member

Manishearth commented Sep 11, 2018

@bors-servo try=android

@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Sep 11, 2018

⌛️ Trying commit 2cc4e1d with merge 8a4eda5...

bors-servo added a commit that referenced this pull request Sep 11, 2018

Auto merge of #21586 - Manishearth:gstreamer-bootstrap, r=<try>
Make bootstrap handle linux setup, include gstreamer, make bootstrap run standalone

Previously `./mach bootstrap` wasn't very useful on Linux, it would just install salt. This PR moves that to `./mach bootstrap-salt`, and has `./mach bootstrap` perform the installation steps mentioned in the README.

With these changes `./mach bootstrap` is a valid way of setting up the servo build. Additionally, this adds gstreamer setup to this for cases where gstreamer is not up to date.

Edit: And now, mach bootstrap is able to be run on a fresh system with just python installed (no need for virtualenv or libffi).

Fixes #21446 , fixes #21417

r? @jdm

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/21586)
<!-- Reviewable:end -->
@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Sep 11, 2018

☀️ Test successful - android, android-x86
State: approved= try=True

@Manishearth

This comment has been minimized.

Member

Manishearth commented Sep 11, 2018

@bors-servo r=jdm

@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Sep 11, 2018

📌 Commit 2cc4e1d has been approved by jdm

@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Sep 11, 2018

⌛️ Testing commit 2cc4e1d with merge 9dd85e3...

bors-servo added a commit that referenced this pull request Sep 11, 2018

Auto merge of #21586 - Manishearth:gstreamer-bootstrap, r=jdm
Make bootstrap handle linux setup, include gstreamer, make bootstrap run standalone

Previously `./mach bootstrap` wasn't very useful on Linux, it would just install salt. This PR moves that to `./mach bootstrap-salt`, and has `./mach bootstrap` perform the installation steps mentioned in the README.

With these changes `./mach bootstrap` is a valid way of setting up the servo build. Additionally, this adds gstreamer setup to this for cases where gstreamer is not up to date.

Edit: And now, mach bootstrap is able to be run on a fresh system with just python installed (no need for virtualenv or libffi).

Fixes #21446 , fixes #21417

r? @jdm

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/21586)
<!-- Reviewable:end -->
@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Sep 11, 2018

💔 Test failed - mac-rel-css2

@jdm

This comment has been minimized.

Member

jdm commented Sep 11, 2018

@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Sep 11, 2018

⌛️ Testing commit 2cc4e1d with merge 02dc71e...

bors-servo added a commit that referenced this pull request Sep 11, 2018

Auto merge of #21586 - Manishearth:gstreamer-bootstrap, r=jdm
Make bootstrap handle linux setup, include gstreamer, make bootstrap run standalone

Previously `./mach bootstrap` wasn't very useful on Linux, it would just install salt. This PR moves that to `./mach bootstrap-salt`, and has `./mach bootstrap` perform the installation steps mentioned in the README.

With these changes `./mach bootstrap` is a valid way of setting up the servo build. Additionally, this adds gstreamer setup to this for cases where gstreamer is not up to date.

Edit: And now, mach bootstrap is able to be run on a fresh system with just python installed (no need for virtualenv or libffi).

Fixes #21446 , fixes #21417

r? @jdm

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/21586)
<!-- Reviewable:end -->
@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Sep 11, 2018

💔 Test failed - mac-rel-wpt4

@jdm

This comment has been minimized.

Member

jdm commented Sep 12, 2018

@bors-servo retry

@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Sep 12, 2018

⌛️ Testing commit 2cc4e1d with merge babed6c...

bors-servo added a commit that referenced this pull request Sep 12, 2018

Auto merge of #21586 - Manishearth:gstreamer-bootstrap, r=jdm
Make bootstrap handle linux setup, include gstreamer, make bootstrap run standalone

Previously `./mach bootstrap` wasn't very useful on Linux, it would just install salt. This PR moves that to `./mach bootstrap-salt`, and has `./mach bootstrap` perform the installation steps mentioned in the README.

With these changes `./mach bootstrap` is a valid way of setting up the servo build. Additionally, this adds gstreamer setup to this for cases where gstreamer is not up to date.

Edit: And now, mach bootstrap is able to be run on a fresh system with just python installed (no need for virtualenv or libffi).

Fixes #21446 , fixes #21417

r? @jdm

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/21586)
<!-- Reviewable:end -->
@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Sep 12, 2018

@bors-servo bors-servo merged commit 2cc4e1d into servo:master Sep 12, 2018

3 of 4 checks passed

Taskcluster (pull_request) TaskGroup: failure
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
homu Test successful
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment