Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Android refactor #263

Merged
merged 6 commits into from Mar 22, 2016
Merged

Android refactor #263

merged 6 commits into from Mar 22, 2016

Conversation

@aneeshusa
Copy link
Member

aneeshusa commented Mar 22, 2016

Main highlights:

  • Update to SHA512 hashes and HTTPS URLs
  • Use multiple directories + symlinks to be more robust during version updates for Android
  • Moves states around to make their purpose more clear

Helps with #209.
Supersedes #259, #260.

cc @larsbergstrom @edunham

I recommend reviewing this commit by commit and reading the commit messages - feel free to ask questions.

Also, we should check that this doesn't break buildbot; I don't know how to do that.


This change is Reviewable

@aneeshusa
Copy link
Member Author

aneeshusa commented Mar 22, 2016

Also, the reason this is WIP is that we should version the URL that we download B2G from so I can give it a similar treatment as the Android states, preferably something like

https://servo-rust.s3.amazonaws.com/B2G/v1/B2G.tgz

or

https://servo-rust.s3.amazonaws.com/B2G/v1/B2G-v1.tgz
@aneeshusa
Copy link
Member Author

aneeshusa commented Mar 22, 2016

(Also, if we're updating the B2G URL: I always find capital letters in URLs to look weird, so something like https://servo-rust.s3.amazonaws.com/b2g/v1/b2g-v1.tgz would be a nice-to-have. Disclaimer: I'm don't know about any branding-related concerns for B2G.)

@larsbergstrom
Copy link
Contributor

larsbergstrom commented Mar 22, 2016

I would not invest too much effort into the B2G rules - I suspect that once we have Aarch64 working (requires a servo fix before we can gate on it) there will be limited need to keep it running. Between Android & Arm32/Aarch64, we'll have most of the coverage we want for those platforms.

I don't know when we would remove it, but I can promise there won't be a v2 of the binaries :-)

@aneeshusa
Copy link
Member Author

aneeshusa commented Mar 22, 2016

Do we need to install the servo-build-dependencies on servo-master? I don't think it's running a buildbot instance, so I don't see why it would need the servo build dependencies.

@larsbergstrom
Copy link
Contributor

larsbergstrom commented Mar 22, 2016

The only thing that I see in that list that we need is git. Good catch!

@aneeshusa
Copy link
Member Author

aneeshusa commented Mar 22, 2016

What would we need git for? I'm assuming for the buildbot master?

@larsbergstrom
Copy link
Contributor

larsbergstrom commented Mar 22, 2016

@aneeshusa Maybe we're doing it wrong, but servo-master is where we have our "main" salt enlistment, and our process for doing the actual updates is after PRs land in this repo, we ssh to the master, git pull origin master in that directory and then do our highstate.

@aneeshusa
Copy link
Member Author

aneeshusa commented Mar 22, 2016

Salt actually has built-in support for using git as a fileserver backend: https://docs.saltstack.com/en/2015.5/topics/tutorials/gitfs.html.

Our current workflow is OK for now, so I'll add a state to install git on the master, and look at gitfs later (edit: see #264).

@aneeshusa
Copy link
Member Author

aneeshusa commented Mar 22, 2016

Hmm, it looks like we're hitting the Travis log length limit due to how many files we extract: each extracted file shows up twice, once in the INFO logging and once at the end. How do you feel about dropping down the log level to warning? We'll miss out on all the lines starting with [INFO ], but we should still be able to debug based on the highstate output at the end or locally with Vagrant.

(We're really close, too! Travis kills us in the middle of outputting the highstate summary at the very end.)

@larsbergstrom
Copy link
Contributor

larsbergstrom commented Mar 22, 2016

I think that dropping it down to warning is fine. I get the most out of seeing which task failed and what the error message was, personally.

@aneeshusa aneeshusa changed the title [WIP] Android refactor Android refactor Mar 22, 2016
@edunham
Copy link
Contributor

edunham commented Mar 22, 2016

Reviewed 9 of 9 files at r1, 5 of 5 files at r2, 3 of 3 files at r3, 4 of 4 files at r4, 2 of 2 files at r5, 1 of 1 files at r6, 2 of 2 files at r7, 1 of 1 files at r8.
Review status: all files reviewed at latest revision, all discussions resolved.


Comments from the review on Reviewable.io

@bors-servo
Copy link
Contributor

bors-servo commented Mar 22, 2016

The latest upstream changes (presumably #250) made this pull request unmergeable. Please resolve the merge conflicts.

aneeshusa added 6 commits Mar 21, 2016
This change groups the various sls files that set up the Servo build
environment into a single folder to:
 - make their purpose more clear in the top.sls file
 - make it easier to add auxiliary files (i.e. map.jinja)

Also, don't install the Servo build dependencies on the Buildbot master,
but make sure to keep git installed to update the Salt file tree.
Use a SHA512 hash instead of a SHA1 hash to verify the B2G download.
Update the style guide.
Use separate directories for separate versions of the
SDK, NDK, and toolchain, and use symlinks to point to the current
versions. This is more robust for a few reasons:
 - Partially-completed upgrades to new versions won't touch
   existing versions on the disk
 - The symlinks aren't updated until the respective new version is
   completely installed, allowing for more transactional updates.
 - The symlinks also allow for constant paths in the bash_profile file
   and the buildbot config, which means less moving parts to break.
   In particular, the relevant buildbot config is on the master, and
   using symlinks makes it unnecessary to use Salt orchestration to
   gate changes to the master buildbot config on changes on the cross
   builders.

Ideally, these states would also use file.directory with clean: True
to clean out old versions of the SDK, NDK, and toolchain, but I wasn't
able to get this working properly yet (the just-downloaded files would
keep getting cleaned away).

This also eliminates our use of cmd.wait, which was recently put on the
deprecation path in Salt, and replaces it with cmd.run + creates: True.
The buildbot configuration already sets the correct environment
variables for Android builds, so it's unnecessary to set them via
.bash_profile.
archive.extracted is fairly loud because each extracted file is listed
both in INFO log level output and the highstate summary.
This gets around exceeding Travis's 4MB log length limit.
@aneeshusa aneeshusa force-pushed the aneeshusa:android-refactor branch from d3114e6 to 1c21ea7 Mar 22, 2016
@aneeshusa
Copy link
Member Author

aneeshusa commented Mar 22, 2016

@edunham I needed to rebase to handle the fixup and squash commits anyways, so I rebased on top of the latest master.

@edunham
Copy link
Contributor

edunham commented Mar 22, 2016

Reviewable did not handle that rebase very gracefully. Still good, though.


Reviewed 3 of 9 files at r1, 11 of 11 files at r9, 5 of 5 files at r10, 3 of 3 files at r11, 4 of 4 files at r12, 2 of 2 files at r13, 1 of 1 files at r14.
Review status: all files reviewed at latest revision, all discussions resolved.


Comments from the review on Reviewable.io

@edunham
Copy link
Contributor

edunham commented Mar 22, 2016

@bors-servo
Copy link
Contributor

bors-servo commented Mar 22, 2016

📌 Commit 1c21ea7 has been approved by edunham

@bors-servo
Copy link
Contributor

bors-servo commented Mar 22, 2016

Testing commit 1c21ea7 with merge b5173aa...

bors-servo added a commit that referenced this pull request Mar 22, 2016
Android refactor

Main highlights:
  - Update to SHA512 hashes and HTTPS URLs
  - Use multiple directories + symlinks to be more robust during version updates for Android
  - Moves states around to make their purpose more clear

Helps with #209.
Supersedes #259, #260.

cc @larsbergstrom @edunham

I recommend reviewing this commit by commit and reading the commit messages - feel free to ask questions.

Also, we should check that this doesn't break buildbot; I don't know how to do that.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/saltfs/263)
<!-- Reviewable:end -->
@bors-servo
Copy link
Contributor

bors-servo commented Mar 22, 2016

☀️ Test successful - travis

@bors-servo bors-servo merged commit 1c21ea7 into servo:master Mar 22, 2016
3 checks passed
3 checks passed
code-review/reviewable Review complete: all files reviewed, all discussions resolved
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
homu Test successful
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.