Skip to content
This repository has been archived by the owner on Oct 12, 2023. It is now read-only.

Create and test multi-branch pipeline for the https://github.com/mdn/interactive-examples repo on new CI/CD service. #41

Closed
7 tasks done
escattone opened this issue Aug 17, 2018 · 8 comments
Closed
7 tasks done

Create and test multi-branch pipeline for the https://github.com/mdn/interactive-examples repo on new CI/CD service. #41

escattone opened this issue Aug 17, 2018 · 8 comments
Assignees
@escattone
Milestone
Sprint 4 Q3 2018

Comments

@escattone
Copy link
Contributor

escattone commented Aug 17, 2018 •
edited
Loading

This is a follow-on task from #22.

User story

As an MDN developer, I want to ensure that the Jenkinsfile for https://github.com/mdn/interactive-examples runs on MDN's new IT-owned CI/CD service so that we can continue to build the site, send notifications, and sync to its S3 bucket.

Acceptance criteria

  • The mdn/interactive-examples Jenkinsfile runs successfully for master and prod branches
  • Notifications successfully sent to IRC #mdndev
  • build stage runs successfully for the prod branch
  • s3 sync stage runs successfully for the prod branch
  • Follow-on tasks are documented

Tasks

  • Configure Jenkins for multi-branch pipeline for mdn/interactive-examples
  • Install awscli on Jenkins service
  • Create AWS IAM role for syncing from Jenkins to the S3 bucket
  • Install credentials for IAM role above on the Jenkins service
  • Create new S3 bucket for interactive examples (add to Terraform?)
  • Modify Jenkinsfile to handle both the IT-owned and the MozMEAO-owned Jenkins services
  • Test as detailed in the acceptance criteria
@escattone escattone mentioned this issue Aug 17, 2018
Closed
@jwhitlock jwhitlock added this to the Sprint 4 Q3 2018 milestone Aug 29, 2018
@escattone escattone self-assigned this Sep 6, 2018
@escattone
Copy link
Contributor Author

Yesterday (Sept. 6, 2018):

  • Configured Jenkins for multi-branch pipeline for mdn/interactive-examples
  • The awscli was already installed on the Jenkins service
  • Created AWS IAM policy, group, and programmatic user for syncing from Jenkins to the S3 bucket
  • Installed (with the help of @limed) the credentials for the new IAM user above on the Jenkins service
  • The S3 bucket for interactive examples (mdninteractive-b77d14bceaaa9ea4) was already created

Today (Sept. 7, 2018):

@escattone
Copy link
Contributor Author

escattone commented Sep 7, 2018 •
edited
Loading

All branches of https://github.com/mdn/interactive-examples other than the prod branch don't do anything other than checkout the repo, so there's not much to test. Still, here's the latest console log from the master branch on the new IT-owned Jenkins service:

Branch indexing
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to https://github.com/mdn/interactive-examples.git
 > git config remote.origin.url https://github.com/mdn/interactive-examples.git # timeout=10
Fetching origin...
Fetching upstream changes from origin
 > git --version # timeout=10
 > git config --get remote.origin.url # timeout=10
 > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/*
Seen branch in repository origin/add-background-size-css-examples
Seen branch in repository origin/add-color-css-exmaples
Seen branch in repository origin/add-docs-to-gitignore
Seen branch in repository origin/add-position-css-examples
Seen branch in repository origin/adds-animation-css-example
Seen branch in repository origin/adds-background-image-css-example
Seen branch in repository origin/adds-box-shadow-css-example
Seen branch in repository origin/adds-js-example-array-from
Seen branch in repository origin/adds-text-align-css-example
Seen branch in repository origin/dp_jenkins_deploy_to_s3
Seen branch in repository origin/dp_test_deploy
Seen branch in repository origin/enable-deploy-on-build-complete
Seen branch in repository origin/fix-css-examples-regression
Seen branch in repository origin/gh-pages
Seen branch in repository origin/js-function-expression
Seen branch in repository origin/master
Seen branch in repository origin/prod
Seen branch in repository origin/remove-docs-from-git
Seen branch in repository origin/remove-google-prettify
Seen branch in repository origin/revert-576-next-iteration-of-the-html-editor
Seen branch in repository origin/ui-tweaks
Seen branch in repository origin/update-docs
Seen branch in repository origin/update-docs-add-editorconfig
Seen branch in repository origin/update-license
Seen branch in repository origin/updates-based-on-review-feedback
Seen 25 remote branches
Obtained Jenkinsfile from bc0fa77f412dc73f066d34fd55317578f4f363c9
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/interactive-examples_master-H47OSS7QUQKIYL3PM5BRXGEQAHLYXSUQZ4QIIJLRJU6HMIG2PLUA
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Prepare)
[Pipeline] checkout
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/mdn/interactive-examples.git # timeout=10
Fetching without tags
Fetching upstream changes from https://github.com/mdn/interactive-examples.git
 > git --version # timeout=10
 > git fetch --no-tags --progress https://github.com/mdn/interactive-examples.git +refs/heads/*:refs/remotes/origin/*
Checking out Revision bc0fa77f412dc73f066d34fd55317578f4f363c9 (master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bc0fa77f412dc73f066d34fd55317578f4f363c9
Commit message: "Merge pull request #1136 from escattone/fix-irc-notify-permissions-1483072"
 > git rev-list --no-walk 65b090c70e1c64702879f6e839f21bd6238c7d9f # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

All good on the master branch! ✅

@escattone
Copy link
Contributor Author

escattone commented Sep 7, 2018 •
edited
Loading

The interesting case is the prod branch, where it builds the site and then syncs the results of the build to an S3 bucket. Here's the latest console log from that branch on the new IT-owned Jenkins service (with a large chunk of the output from theaws s3 sync command removed to avoid scrolling through an annoyingly large log):

Branch indexing
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to https://github.com/mdn/interactive-examples.git
 > git config remote.origin.url https://github.com/mdn/interactive-examples.git # timeout=10
Fetching origin...
Fetching upstream changes from origin
 > git --version # timeout=10
 > git config --get remote.origin.url # timeout=10
 > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/*
Seen branch in repository origin/add-background-size-css-examples
Seen branch in repository origin/add-color-css-exmaples
Seen branch in repository origin/add-docs-to-gitignore
Seen branch in repository origin/add-position-css-examples
Seen branch in repository origin/adds-animation-css-example
Seen branch in repository origin/adds-background-image-css-example
Seen branch in repository origin/adds-box-shadow-css-example
Seen branch in repository origin/adds-js-example-array-from
Seen branch in repository origin/adds-text-align-css-example
Seen branch in repository origin/dp_jenkins_deploy_to_s3
Seen branch in repository origin/dp_test_deploy
Seen branch in repository origin/enable-deploy-on-build-complete
Seen branch in repository origin/fix-css-examples-regression
Seen branch in repository origin/gh-pages
Seen branch in repository origin/js-function-expression
Seen branch in repository origin/master
Seen branch in repository origin/prod
Seen branch in repository origin/remove-docs-from-git
Seen branch in repository origin/remove-google-prettify
Seen branch in repository origin/revert-576-next-iteration-of-the-html-editor
Seen branch in repository origin/ui-tweaks
Seen branch in repository origin/update-docs
Seen branch in repository origin/update-docs-add-editorconfig
Seen branch in repository origin/update-license
Seen branch in repository origin/updates-based-on-review-feedback
Seen 25 remote branches
Obtained Jenkinsfile from 37766896580f2e87ef2c2881c984ebd5950746b6
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/interactive-examples_prod-Q2S2J2VGDYYJQNLYXIDOJCRBI45FMWCOU2FTV6FR5GPGJMXKZGBQ
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Prepare)
[Pipeline] checkout
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/mdn/interactive-examples.git # timeout=10
Fetching without tags
Fetching upstream changes from https://github.com/mdn/interactive-examples.git
 > git --version # timeout=10
 > git fetch --no-tags --progress https://github.com/mdn/interactive-examples.git +refs/heads/*:refs/remotes/origin/*
Checking out Revision 37766896580f2e87ef2c2881c984ebd5950746b6 (prod)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 37766896580f2e87ef2c2881c984ebd5950746b6
Commit message: "Deploying latest generated pages"
 > git rev-list --no-walk fb1c34ac62e5cbd1dde3ce8d3a423d979f8ef846 # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (build)
[Pipeline] sh
[interactive-examples_prod-Q2S2J2VGDYYJQNLYXIDOJCRBI45FMWCOU2FTV6FR5GPGJMXKZGBQ] Running shell script
+ bin/build.sh
++ dirname bin/build.sh
+ cd bin/..
+ echo 'Starting build'
Starting build
+ docker run -v /var/lib/jenkins/workspace/interactive-examples_prod-Q2S2J2VGDYYJQNLYXIDOJCRBI45FMWCOU2FTV6FR5GPGJMXKZGBQ:/mdn -w /mdn node:latest bash -c '/usr/local/bin/npm install && /usr/local/bin/npm run build'
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.4 (node_modules/sane/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.4: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.4 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.4: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

audited 28927 packages in 8.561s
found 145 vulnerabilities (62 low, 83 moderate)
  run `npm audit fix` to fix them, or `npm audit` for details

> interactive-examples@1.0.0 build /mdn
> npm-run-all build-css-bundle build-js-bundle build-tabbed-js-bundle build-pages


> interactive-examples@1.0.0 build-css-bundle /mdn
> browserify js/editable-css.js -o js/editor-css-bundle.js


> interactive-examples@1.0.0 build-js-bundle /mdn
> browserify js/editable-js.js -o js/editor-js-bundle.js


> interactive-examples@1.0.0 build-tabbed-js-bundle /mdn
> browserify js/editor.js -o js/editor-bundle.js


> interactive-examples@1.0.0 build-pages /mdn
> node index.js

Pages built successfully
+ echo 'Build finished'
Build finished
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (s3 sync)
[Pipeline] sh
[interactive-examples_prod-Q2S2J2VGDYYJQNLYXIDOJCRBI45FMWCOU2FTV6FR5GPGJMXKZGBQ] Running shell script
+ bin/s3-sync.sh mdninteractive-b77d14bceaaa9ea4
Completed 3.4 KiB/~38.5 KiB (94.7 KiB/s) with ~12 file(s) remaining (calculating...)
upload: pages/css/align-items.html to s3://mdninteractive-b77d14bceaaa9ea4/pages/css/align-items.html
Completed 3.4 KiB/~38.5 KiB (94.7 KiB/s) with ~11 file(s) remaining (calculating...)
...
Completed 5.1 MiB/5.3 MiB (7.8 MiB/s) with 2 file(s) remaining
upload: media/fonts/FiraSans-Bold.woff2 to s3://mdninteractive-b77d14bceaaa9ea4/media/fonts/FiraSans-Bold.woff2
Completed 5.1 MiB/5.3 MiB (7.8 MiB/s) with 1 file(s) remaining
Completed 5.3 MiB/5.3 MiB (7.7 MiB/s) with 1 file(s) remaining
upload: media/fonts/FiraSans-SemiBoldItalic.woff2 to s3://mdninteractive-b77d14bceaaa9ea4/media/fonts/FiraSans-SemiBoldItalic.woff2
[Pipeline] sh
[interactive-examples_prod-Q2S2J2VGDYYJQNLYXIDOJCRBI45FMWCOU2FTV6FR5GPGJMXKZGBQ] Running shell script
+ bin/irc-notify.sh --stage s3 sync prod --status shipped
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

Both the build and the sync to the S3 bucket were successful. All good here as well. ✅

@escattone
Copy link
Contributor Author

Acceptance Criteria:

  • The mdn/interactive-examples Jenkinsfile runs successfully for the master and prod branches on both the MozMEAO-owned and IT-owned Jenkins services
  • Notifications successfully sent to IRC #mdndev and the embedded link in each notification points to the appropriate Jenkins service (either the MozMEAO-owned or IT-owned service)
  • build stage runs successfully for the prod branch on both Jenkins services
  • s3 sync stage runs successfully for the prod branch on both Jenkins services

@escattone
Copy link
Contributor Author

On the IT-owned Jenkins service, running on an EC2 instance, we're using a ~/.aws/credentials file containing an access/secret key pair under the mdninteractive profile for running the aws s3 sync command (the access/secret key are for the mdn-interactive-uploader IAM user). @limed has rightly suggested that we should instead use an IAM role with the appropriate S3 permissions that we can then associate with the EC2 instance on which the Jenkins service runs (see https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html).

@escattone
Copy link
Contributor Author

On the new IT-owned Jenkins service we just abandoned the approach of installing AWS credentials locally, and moved instead to the AWS-recommended approach of associating IAM roles with EC2 instances (see https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). I submitted mdn/interactive-examples#1137 (needs review, merging, and testing) which allows us to satisfy the MozMEAO-owned Jenkins service (which uses locally installed credentials within profiles) during this interim period until we make the cut-over to the IT-owned services.

@jwhitlock jwhitlock mentioned this issue Sep 10, 2018
Closed
@escattone
Copy link
Contributor Author

mdn/interactive-examples#1137 has been tested, reviewed and merged. I'd like to see the prod branch successfully run on both the IT-owned and MozMEAO-owned Jenkins services before calling this issue done.

@escattone
Copy link
Contributor Author

✅ The prod branch ran successfully on the IT-owned Jenkins service.
✅ The prod branch ran successfully on the MozMEAO-owned Jenkins service.

@limed limed closed this as completed Sep 27, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@escattone escattone

Labels
None yet
Projects
None yet
Milestone
Sprint 4 Q3 2018
Development

No branches or pull requests
3 participants
@limed @jwhitlock @escattone