Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase chamber retries to 20 #1837

Merged
merged 1 commit into from Mar 8, 2019
Merged
Changes from all commits
Commits
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.
+22 −4
Diff settings

Always

Just for now

increase retries for chamber to 20

  • Loading branch information...
pjdufour-truss committed Mar 7, 2019
commit 215f70680b6a01ca6414afefbc5e432da0647556
Copy path View file
@@ -56,6 +56,8 @@ commands:
- run:
name: Run migrations
command: bin/do-exclusively --job-name ${CIRCLE_JOB} bin/ecs-run-app-migrations-container config/app-migrations.container-definition.json ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/app-migrations:git-${CIRCLE_SHA1} $APP_ENVIRONMENT
environment:
CHAMBER_RETRIES: 20
- announce_failure
deploy_app_steps:
parameters:
@@ -76,6 +78,7 @@ commands:
- deploy:
name: Deploy app service
command: bin/do-exclusively --job-name ${CIRCLE_JOB} bin/ecs-deploy-service-container app config/app.container-definition.json ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/app:git-${CIRCLE_SHA1} $APP_ENVIRONMENT FARGATE
no_output_timeout: 20m

This comment has been minimized.

Copy link
@rdhariwal

rdhariwal Mar 8, 2019

Contributor

I was curios what 'no_output_timeout' does, looks like we are overriding the default 10m timeout for the command if it doesn't spit out anything.

This is what circleci docs had to say about it:

Elapsed time the command can run without output. The string is a decimal with unit suffix, such as “20m”, “1.25h”, “5s” (default: 10 minutes)

This comment has been minimized.

Copy link
@pjdufour-truss

pjdufour-truss Mar 8, 2019

Author Contributor

You got it. Some ECS commands timeout right at 10 minutes, so we want a buffer. 20 minutes seems like a good number, but 15 minutes could work too I guess.

- run:
name: Health Check
command: go run cmd/health_checker/main.go --schemes http,https --hosts << parameters.health_check_hosts >> --tries 10 --backoff 3 --log-level info --timeout 15m
@@ -102,6 +105,7 @@ commands:
- deploy:
name: Deploy app-client-tls service
command: bin/do-exclusively --job-name ${CIRCLE_JOB} bin/ecs-deploy-service-container app-client-tls config/app-client-tls.container-definition.json ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/app:git-${CIRCLE_SHA1} $APP_ENVIRONMENT FARGATE
no_output_timeout: 20m
- run:
name: Health Check
command: |
@@ -588,6 +592,7 @@ jobs:
name: Run acceptance tests
command: make webserver_test
environment:
CHAMBER_RETRIES: 20
PWD: /home/circleci/go/src/github.com/transcom/mymove
DOD_CA_PACKAGE: /home/circleci/go/src/github.com/transcom/mymove/config/tls/Certificates_PKCS7_v5.4_DoD.der.p7b
TEST_ACC_ENV: experimental
@@ -650,6 +655,7 @@ jobs:
name: Run acceptance tests
command: make webserver_test
environment:
CHAMBER_RETRIES: 20
PWD: /home/circleci/go/src/github.com/transcom/mymove
DOD_CA_PACKAGE: /home/circleci/go/src/github.com/transcom/mymove/config/tls/Certificates_PKCS7_v5.4_DoD.der.p7b
TEST_ACC_ENV: staging
Copy path View file
3 .envrc
@@ -53,6 +53,9 @@ check_required_variables() {
# directly into a subdirectory.
export MYMOVE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

# Sets the number of retries for chamber to 20.
export CHAMBER_RETRIES=20

# Configuration needed for secure migrations.
export SECURE_MIGRATION_DIR="${MYMOVE_DIR}/local_migrations"
export SECURE_MIGRATION_SOURCE="local"
Copy path View file
@@ -276,13 +276,13 @@ ifndef CIRCLECI
TEST_ACC_CWD=$(PWD) \
DISABLE_AWS_VAULT_WRAPPER=1 \
aws-vault exec $(AWS_PROFILE) -- \
bin/chamber exec app-$(TEST_ACC_ENV) -- \
bin/chamber -r $(CHAMBER_RETRIES) exec app-$(TEST_ACC_ENV) -- \
go test -v -p 1 -count 1 -short $$(go list ./... | grep \\/cmd\\/webserver)
else
@echo "Running acceptance tests for webserver with environment $$TEST_ACC_ENV."
TEST_ACC_DATABASE=0 TEST_ACC_HONEYCOMB=0 \
TEST_ACC_CWD=$(PWD) \
bin/chamber exec app-$(TEST_ACC_ENV) -- \
bin/chamber -r $(CHAMBER_RETRIES) exec app-$(TEST_ACC_ENV) -- \
go test -v -p 1 -count 1 -short $$(go list ./... | grep \\/cmd\\/webserver)
endif
endif
@@ -11,6 +11,8 @@
"essential": true,
"entryPoint": [
"/bin/chamber",
"-r",
"{{ .CHAMBER_RETRIES }}",
"exec",
"app-{{ .environment }}",
"--",
@@ -4,6 +4,8 @@
"essential": true,
"entryPoint": [
"/bin/chamber",
"-r",
"{{ .CHAMBER_RETRIES }}",
"exec",
"app-{{ .environment }}",
"--",
@@ -11,6 +11,8 @@
"essential": true,
"entryPoint": [
"/bin/chamber",
"-r",
"{{ .CHAMBER_RETRIES }}",
"exec",
"app-{{ .environment }}",
"--",
Copy path View file
@@ -8,3 +8,4 @@ DPS_COOKIE_NAME=DPSIVV
GEX_URL=https://gexweba.daas.dla.mil/msg_data/submit/
SEND_PROD_INVOICE=false
SERVE_SWAGGER_UI=true
CHAMBER_RETRIES=20
Copy path View file
@@ -7,4 +7,5 @@ DPS_REDIRECT_URL=https://dps.sddc.army.mil/cust
DPS_COOKIE_NAME=DPS
GEX_URL=https://gexweba.daas.dla.mil/msg_data/submit/
SEND_PROD_INVOICE=true
SERVE_SWAGGER_UI=false
SERVE_SWAGGER_UI=false
CHAMBER_RETRIES=20
Copy path View file
@@ -7,4 +7,5 @@ DPS_REDIRECT_URL=https://dpstest.sddc.army.mil/cust
DPS_COOKIE_NAME=DPSIVV
GEX_URL=https://gexweba.daas.dla.mil/msg_data/submit/
SEND_PROD_INVOICE=false
SERVE_SWAGGER_UI=true
SERVE_SWAGGER_UI=true
CHAMBER_RETRIES=20
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.