Refactor to run one workflow per-environment #85
Conversation
Why:
1) Allow for easy identification of which environment smoke test failed.
2) Allow manual running of smoke tests for a single environment, alleviating
the need to run from local workstations.
Changes:
* Modify circleci-run to take a single environment name as an argument
instead of hard coding all environments and running in serial.
* Add -r NUM option to control the number of retries for a smoke test
run. (Previous default of 3 maintained.)
* Refactor CircleCI definition:
- Reusable executor definition to allow changing Ruby image in one place
- Per-environment job definitions
- Per-environment workflows triggered automatically (*:20 - int,
*:25 - staging, *:30 - prod)
.circleci/config.yml
Outdated
| smoketest_staging: | ||
| triggers: | ||
| - schedule: | ||
| cron: "25 * * * *" |
There was a problem hiding this comment.
WDYT of staggering these by 10 minutes instead of 5? Or is 5 minutes plenty of time?
There was a problem hiding this comment.
I think 10 is better. With retries these can take a while.
bin/circleci-run
Outdated
| puts "Smoke tests failed in #{environment}. Retrying..." | ||
| return if system "HEADLESS_BROWSER=true LOWER_ENV=#{environment} bundle exec rspec --only-failure" | ||
| extra_flags = ' --only-failure' |
There was a problem hiding this comment.
WDYT of making the command an array and joining it, rather than having to manage spaces manually?
cmd = ['HEADLESS_BROWSER=true LOWER_ENV=#{environment} bundle exec rspec']
cmd << '--only-failure'
# ...
system cmd.join(' ')
bin/circleci-run
Outdated
|
|
||
| USAGEDOC | ||
|
|
||
| opts.on('-r NUM', Integer, 'Times to retry test. (Default: 3)') do |r| |
There was a problem hiding this comment.
can we add a longer, more descriptive flag name as well?
opts.on('-r NUM', '--retry-count NUM', ....
| filters: | ||
| branches: | ||
| only: | ||
| - master |
There was a problem hiding this comment.
It may make sense to have a branch for each env. Something that matches the IdP code. These frequently break because they are looking for a button label that is deployed to one env but not the other.
There was a problem hiding this comment.
Excellent idea! Going to merge this now and if no one else beats me to it look into that later in the week.
There was a problem hiding this comment.
I think the per-branch stuff might work best if these lived in the same repo as the IDP
Why
the need to run from local workstations.
Changes
instead of hard coding all environments and running in serial.
run. (Previous default of 3 maintained.)
*:25 - staging, *:30 - prod)
Testing
.circleci/config.yamland add your test branch name to one or more of the branch lists under the workflow trigger(s) you wish to test. Example: Runintscheduled test on branchtester/thing:Additional
bin/circleci-runmay also be useful for workstation test runs as it includes retry logic