Nightlies #409

edunham · 2016-06-29T05:54:46Z

2 separate config files because the Android upload to the servo-rust bucket in the Rust account, and the everything-else upload to the servo-developer-preview bucket in the Servo account, require different credentials. @aneeshusa, could you remind me of the right way to salt the near-identical files which end up with different pillar data and different names?

Windows nightly requires:

Appropriate package command
Either refactoring the DynamicServoYAMLFactory to handle how all windows commands need to be run in their own bash shells (correct solution) or writing a new custom Windows factory (fast but bad solution)

This change is


        Add schedulers for Linux, OSX nightlies

Windows should wait on factories.py refactor to fix:: windows = ServoFactory([ # TODO: convert this to use DynamicServoFactory # We need to run each command in a bash login shell, which breaks the # heuristics used by DynamicServoFactory.make_step steps.Compile(command=make_win_command("./mach build -d -v"), env=envs.build_windows), steps.Test(command=make_win_command("./mach test-unit"), env=envs.build_windows), # TODO: run lockfile_changed.sh and manifest_changed.sh scripts ])


        add build steps for nightlies


        Install and configure s3cmd on linux and osx slaves, for nightlies

Already added the credentials to /srv/pillar on the master.

aneeshusa · 2016-06-29T11:44:49Z

Just woke up and haven't looked at this yet, but you must be psychic @edunham - I was about to open an issue/PR for this last night! This should also finish off/fix servo/servo#10339.

aneeshusa · 2016-06-29T11:45:54Z

buildbot/master/files/config/master.cfg

+    builderNames=[
+        "android-nightly",
+        "linux-nightly",
+        "mac-nightly"


nit: leave a comma at the end of this line to reduce future diff noise

aneeshusa · 2016-06-29T11:49:38Z

I would prefer to use environment variables + logEnviron: false to pass the AWS IAM Key Id and Secret Access Key. Using a config file implies it's readable by any process running as servo, and we don't have any user isolation yet (e.g. IIRC Homu also runs as servo). Using environment variables limits the scope to which these credentials are exposed.
In the future I'd even like to see us using https://docs.aws.amazon.com/STS/latest/APIReference/API_GetSessionToken.html to generate temporary session tokens so even if a token gets leaked it's NBD.

This can be done by modifying the make_step functions and adding a new heurisitic for s3cmd. The environment variables are AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY - I was also looking into this last night.

By the way - any reason we are using s3cmd instead of the AWS CLI?

aneeshusa · 2016-06-29T13:14:59Z

buildbot/master/files/config/steps.yml

+mac-nightly:
+  - ./mach build --release
+  - ./mach package --release
+  - s3cmd -c ~/.s3cfg-servo put target/release/*.tar.gz s3://servo-developer-preview/nightly/`date +%Y`/


Two things:

Prefer "$(command)" form to backticks - style guide.

I believe that since make_step splits the command up into a list before passing it to Buildbot, Buildbot will exec the command directly, instead of through a shell, and the command substitution won't work. IMO it would be better to make a script for this in the servo repo in the etc/ci directory and call it from here.

Also the mac package script now generates a .dmg instead of a .tar.gz and it is put in the target directory (not debug or release directories)

aneeshusa · 2016-06-29T13:15:47Z

buildbot/master/files/config/steps.yml

+  - ./mach build --release
+  - ./mach package --release
+  - s3cmd -c ~/.s3cfg-servo put target/release/*.tar.gz s3://servo-developer-preview/nightly/`date +%Y`/
+  - rm -rf target/


Why did you add the rm -rf lines? Buildbot will clean out the working directory at the start of each build IIRC (or I think maybe the git step does this), so it should not be necessary to have as a standalone step.

aneeshusa · 2016-06-29T13:20:20Z

It may also be nice to set up a MailNotifier so Buildbot sends emails when a nightly build fails, but that can be saved for a follow-up.

metajack · 2016-06-29T13:55:11Z

Is there a reason to continue to upload android builds to a Rust controlled bucket? Why don't we just dump everything in the servo bucket?


        Use Servo nightlies bucket for Android


        Improve nightly upload steps:

* Fix directory structure to sort by target and ditch year directory; realistically we probably won't be in "developer preview" in 2-3 years when the # of files starts getting unmanageable * Upload .dmg from correct location on osx * Upload everything to Servo's AWS account

larsbergstrom · 2016-06-29T19:09:37Z

servo-master1 builds get:

[ FAIL ] Buildbot master config check failed:

         Configuration Errors:

           Unknown builder 'linux-nightly' in scheduler 'Nightly'

           Unknown builder 'mac-nightly' in scheduler 'Nightly'

           Unknown builder 'linux-nightly' in scheduler 'force'

           Unknown builder 'mac-nightly' in scheduler 'force'

larsbergstrom · 2016-06-29T19:12:17Z

I think they need to be added here:
https://github.com/edunham/saltfs/blob/281a9e39d65248a0e681b81086aa45330d1892cc/buildbot/master/files/config/master.cfg#L171

aneeshusa · 2016-06-29T19:15:28Z

I would rather not rush this PR too much, since we have some time before release and we can always make a build manually for release. @edunham if you want I can make a PR to your branch to implement passing credentials as environment variables.

larsbergstrom · 2016-06-29T19:33:42Z

I'm not super excited about manual builds from user machines and would rather hold off on releasing binaries until we do it from machines that were at least originally installed from a clean state with only the dependencies we've documented & listed in salt. Maybe manual builds from those machines would make me feel less stressed, but publishing bits to the world from laptops the team has taken through security checkpoints in all sorts of interesting countries seems dubious :-)

larsbergstrom · 2016-06-30T01:41:06Z

nightly/files/s3cfg-rust

+gpg_passphrase =
+guess_mime_type = True
+host_base = s3.amazonaws.com
+host_bucket = %(bucket)s.s3.amazonaws.com


Is the s after %(bucket) supposed to be there?

larsbergstrom · 2016-06-30T01:41:20Z

nightly/files/s3cfg-servo

+gpg_passphrase =
+guess_mime_type = True
+host_base = s3.amazonaws.com
+host_bucket = %(bucket)s.s3.amazonaws.com


Same question about the s here.

larsbergstrom · 2016-06-30T01:41:29Z

nightly/files/s3cfg-servo

+use_https = True
+use_mime_magic = True
+verbosity = WARNING
+website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/


And here :-)

larsbergstrom · 2016-06-30T01:41:47Z

nightly/files/s3cfg-rust

+use_https = True
+use_mime_magic = True
+verbosity = WARNING
+website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/


Same question :-)

aneeshusa · 2016-06-30T01:49:20Z

This will need servo/servo#11943 at the least, and then to use that script.


        Auto merge of #410 - aneeshusa:add-linux-and-osx-nightly-builds, r=la…

@larsbergstrom

…rsbergstrom Add linux and osx nightly builds Alternative to #409 that is more secure IMO. Requires servo/servo#11943. cc @larsbergstrom @edunham  --- This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/saltfs/410)


        Auto merge of #410 - aneeshusa:add-linux-and-osx-nightly-builds, r=la…

@larsbergstrom

…rsbergstrom Add linux and osx nightly builds Alternative to #409 that is more secure IMO. Requires servo/servo#11943. cc @larsbergstrom @edunham  --- This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/saltfs/410)

bors-servo · 2016-06-30T07:28:52Z

☔ The latest upstream changes (presumably #410) made this pull request unmergeable. Please resolve the merge conflicts.

aneeshusa · 2016-06-30T12:31:17Z

Filed #413 and #414 as follow-ups. Closing in favor of #410 (and #411).

edunham added 3 commits Jun 24, 2016

add build steps for nightlies

0516df1

Install and configure s3cmd on linux and osx slaves, for nightlies

Loading status checks…

72f805f

Already added the credentials to /srv/pillar on the master.

aneeshusa reviewed Jun 29, 2016
View changes

buildbot/master/files/config/master.cfg

builderNames=[

"android-nightly",

"linux-nightly",

"mac-nightly"

This comment has been minimized.

Sign in to view

Copy link

aneeshusa Jun 29, 2016
Member

nit: leave a comma at the end of this line to reduce future diff noise

aneeshusa reviewed Jun 29, 2016
View changes

edunham added 2 commits Jun 29, 2016

Use Servo nightlies bucket for Android

8b5c1bf

larsbergstrom reviewed Jun 30, 2016
View changes

nightly/files/s3cfg-servo Outdated

gpg_passphrase =

guess_mime_type = True

host_base = s3.amazonaws.com

host_bucket = %(bucket)s.s3.amazonaws.com

This comment has been minimized.

Sign in to view

Copy link

larsbergstrom Jun 30, 2016
Contributor

Same question about the s here.

larsbergstrom reviewed Jun 30, 2016
View changes

nightly/files/s3cfg-servo Outdated

use_https = True

use_mime_magic = True

verbosity = WARNING

website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/

This comment has been minimized.

Sign in to view

Copy link

larsbergstrom Jun 30, 2016
Contributor

And here :-)

larsbergstrom reviewed Jun 30, 2016
View changes

nightly/files/s3cfg-rust Outdated

use_https = True

use_mime_magic = True

verbosity = WARNING

website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/

This comment has been minimized.

Sign in to view

Copy link

larsbergstrom Jun 30, 2016
Contributor

Same question :-)

aneeshusa mentioned this pull request Jun 30, 2016

Add linux and osx nightly builds #410

Merged

aneeshusa closed this Jun 30, 2016

servo / saltfs

Nightlies #409

Nightlies #409

edunham commented Jun 29, 2016 •

edited by larsbergstrom

aneeshusa commented Jun 29, 2016 •

edited

This comment has been minimized.

aneeshusa commented Jun 29, 2016 •

edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

aneeshusa commented Jun 29, 2016

metajack commented Jun 29, 2016

larsbergstrom commented Jun 29, 2016

larsbergstrom commented Jun 29, 2016

aneeshusa commented Jun 29, 2016

larsbergstrom commented Jun 29, 2016

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

aneeshusa commented Jun 30, 2016 •

edited

bors-servo commented Jun 30, 2016

aneeshusa commented Jun 30, 2016

servo / saltfs

Join GitHub today

Nightlies #409

Nightlies #409

Conversation

edunham commented Jun 29, 2016 • edited by larsbergstrom

aneeshusa commented Jun 29, 2016 • edited

This comment has been minimized.

aneeshusa commented Jun 29, 2016 • edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

aneeshusa commented Jun 29, 2016

metajack commented Jun 29, 2016

larsbergstrom commented Jun 29, 2016

larsbergstrom commented Jun 29, 2016

aneeshusa commented Jun 29, 2016

larsbergstrom commented Jun 29, 2016

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

aneeshusa commented Jun 30, 2016 • edited

bors-servo commented Jun 30, 2016

aneeshusa commented Jun 30, 2016

edunham commented Jun 29, 2016 •

edited by larsbergstrom

aneeshusa commented Jun 29, 2016 •

edited

aneeshusa commented Jun 29, 2016 •

edited

aneeshusa commented Jun 30, 2016 •

edited