Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload build logs to S3 #8846

Merged
merged 4 commits into from Jun 9, 2016
Merged

Upload build logs to S3 #8846

merged 4 commits into from Jun 9, 2016

Conversation

@islemaster
Copy link
Member

islemaster commented Jun 8, 2016

Uploads build logs from any server running the ci_build script to S3. This includes staging, test, levelbuilder, and production-daemon.

These are the same logs that we e-mail out and post to Slack, though in the latter they've been getting truncated due to API limits. Now they get uploaded to the cdo-build-logs bucket as <hostname>/<build-start-time> where the time is a compact ISO 8601 representation (example: test/20160608T161630-0700).

Slack logging now includes a link to the full build log on S3, which will look something like this (note: link intentionally broken):

websites failed to build! 🕐 4:15 minutes ☁ Log on S3

This link will show up for both successful and failed builds. I've also modified the Slack behavior on failed builds, so instead of trying to post the whole build log they will now post only the last 40 lines of the log, since the full log is easily available now.

Build logs are not public. The link we post to Slack is a presigned URL, that gives public access to the private log file, valid for 60 minutes after the build completes.

We attach some build metadata to the uploaded logs:

  • commit: Full commit hash
  • duration: Build time in seconds
  • success: true if exited with code 0, false otherwise

I haven't enabled versioning on the cdo-build-logs bucket since the logs themselves are timestamped and unlikely to overlap. I've set up a lifecycle rule to permanently delete logs older than 30 days.

I reused some of the code for the UI test log uploader in this implementation, which resulted in some refactoring to that process as well.

islemaster added 3 commits Jun 8, 2016
The ci_build script hasn't actually been responsible for sending mail since (at least) September 2014.  There's no reason it should require 'mail'.
Refactors S3 upload from UI test logs to use with CI build logs as well.

Logs are uploaded to cdo-build-logs/<hostname>/<build-start-time> using a compact ISO 8601 time representation.  They are currently private.

We upload logs for both successful and failed builds, with links posted in Slack.  Slack output for failed builds has been adjusted to show only the last 40 lines of the log, since the full log is now available on S3.
In the case of build logs we don't want to make them public, but we do want to provide a link in Slack.  The solution to this is to generate a "presigned" url which is only valid for a limited time.

Also fixes a bug in the truncated log for Slack.
require 'cdo/rake_utils'
require 'cdo/hip_chat'
require 'cdo/only_one'
require 'mail'

This comment has been minimized.

Copy link
@islemaster

islemaster Jun 8, 2016

Author Member

No mailing occurs in this file; this is an old unused dependency.

log_url
else
options = {bucket: @bucket, key: key, expires_in: 60.minutes}
options[:version_id] = result[:version_id] unless result[:version_id].nil?

This comment has been minimized.

Copy link
@islemaster

islemaster Jun 8, 2016

Author Member

Note: I'm not 100% sure that this will work, and have opened a StackOverflow question looking for more information. This code path isn't exercised in our current configuration anyway - our versioned logs (UI test logs) are public, and our private logs are in an unversioned bucket - but I wanted to give it a shot in case we change our minds.

@islemaster

This comment has been minimized.

Copy link
Member Author

islemaster commented Jun 8, 2016

Testing: I'm able to run aws/ci_build locally although I can't get a successful build out of it. The build log uploads to S3 and the presigned URL that spits out to the stub Slack integration works. I also reverified that uploading and public urls for the UI test logs work.

Beyond that, we just need to try this out on staging.

@Bjvanminnen

This comment has been minimized.

Copy link
Contributor

Bjvanminnen commented Jun 8, 2016

very cool. lgtm

@wjordan

This comment has been minimized.

Copy link
Member

wjordan commented Jun 9, 2016

Didn't look it over too carefully, but the overall approach looks good! once you hand-test it on staging that should be good enough for this CI-internal feature. 👍

@islemaster islemaster merged commit a073f2c into staging Jun 9, 2016
1 check passed
1 check passed
hound No violations found. Woof!
@islemaster islemaster deleted the build-logs-to-s3 branch Jun 9, 2016
@islemaster

This comment has been minimized.

Copy link
Member Author

islemaster commented Jun 9, 2016

Post-merge, this didn't work on a successful staging build. I don't entirely understand why, but I suspect it has something to do with this line and I wonder if the way we're capturing stdout isn't working... or if we need to be capturing stderr too. I'm pretty sure we are running aws/ci_build on staging. Thoughts?

I'm not sure a revert is needed... if I'm right, the ci_build script is just exiting early before any of the S3 upload and logging stuff happens. In retrospect I should have realized none of the nice success logging in this file was reaching Slack, at least on success. I'm tempted to intentionally break staging to see that part work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.