Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Upload build logs to S3 #8846
Uploads build logs from any server running the
These are the same logs that we e-mail out and post to Slack, though in the latter they've been getting truncated due to API limits. Now they get uploaded to the
Slack logging now includes a link to the full build log on S3, which will look something like this (note: link intentionally broken):
This link will show up for both successful and failed builds. I've also modified the Slack behavior on failed builds, so instead of trying to post the whole build log they will now post only the last 40 lines of the log, since the full log is easily available now.
Build logs are not public. The link we post to Slack is a presigned URL, that gives public access to the private log file, valid for 60 minutes after the build completes.
We attach some build metadata to the uploaded logs:
I haven't enabled versioning on the
I reused some of the code for the UI test log uploader in this implementation, which resulted in some refactoring to that process as well.
The ci_build script hasn't actually been responsible for sending mail since (at least) September 2014. There's no reason it should require 'mail'.
Refactors S3 upload from UI test logs to use with CI build logs as well. Logs are uploaded to cdo-build-logs/<hostname>/<build-start-time> using a compact ISO 8601 time representation. They are currently private. We upload logs for both successful and failed builds, with links posted in Slack. Slack output for failed builds has been adjusted to show only the last 40 lines of the log, since the full log is now available on S3.
In the case of build logs we don't want to make them public, but we do want to provide a link in Slack. The solution to this is to generate a "presigned" url which is only valid for a limited time. Also fixes a bug in the truncated log for Slack.
Testing: I'm able to run
Beyond that, we just need to try this out on staging.
Post-merge, this didn't work on a successful staging build. I don't entirely understand why, but I suspect it has something to do with this line and I wonder if the way we're capturing stdout isn't working... or if we need to be capturing stderr too. I'm pretty sure we are running
I'm not sure a revert is needed... if I'm right, the ci_build script is just exiting early before any of the S3 upload and logging stuff happens. In retrospect I should have realized none of the nice success logging in this file was reaching Slack, at least on success. I'm tempted to intentionally break staging to see that part work.