Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempt to speed up integration tests using pre-built images #2478

Merged
merged 58 commits into from Aug 23, 2019
Merged
Changes from 6 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
4bcebb1
Attempt to speed up integration tests using pre-built images
chrisgilmerproj Aug 1, 2019
5396b78
Fix docker login
chrisgilmerproj Aug 1, 2019
bd0151a
Fix vars in makefile
chrisgilmerproj Aug 1, 2019
a331b84
More makefile fixes
chrisgilmerproj Aug 1, 2019
db9aeee
Fix
chrisgilmerproj Aug 1, 2019
daeff69
Use docker-compose to bring up server for e2e tests
chrisgilmerproj Aug 1, 2019
8612a90
Remove extra down command
chrisgilmerproj Aug 1, 2019
8b853fe
Default to port 4000
chrisgilmerproj Aug 1, 2019
421ca51
Wait for migrations to complete and then load data
chrisgilmerproj Aug 1, 2019
a985ce0
Change db name and increase timeout for wait script.
chrisgilmerproj Aug 1, 2019
f275b1d
Run commands crom inside docker
chrisgilmerproj Aug 1, 2019
01e6254
Add a new dockerfile
chrisgilmerproj Aug 1, 2019
5e08706
Fix bad make target
chrisgilmerproj Aug 1, 2019
bc30b9e
Ensure swagger is built
chrisgilmerproj Aug 1, 2019
fa2ab49
Fix port
chrisgilmerproj Aug 1, 2019
cdec25e
Correct the db name
chrisgilmerproj Aug 1, 2019
a2f2e69
Properly use entrypoint
chrisgilmerproj Aug 19, 2019
dc17ed3
Use DOCKER_RUN variable
chrisgilmerproj Aug 19, 2019
dd4e0a9
Merge master
chrisgilmerproj Aug 19, 2019
f8d0be1
Fix problem with finding swagger files
chrisgilmerproj Aug 19, 2019
284c766
Ensure testdata fixture is available
chrisgilmerproj Aug 19, 2019
7f5ca1a
Add an extra sleep to try and catch a race condition
chrisgilmerproj Aug 19, 2019
ff149be
List docker processes
chrisgilmerproj Aug 19, 2019
e4086c0
Follow the migration logs and see if anything is broken
chrisgilmerproj Aug 19, 2019
4fcff8f
Attempt to copy in the local migrations
chrisgilmerproj Aug 19, 2019
9125960
Fix docker ps and add some echo statements
chrisgilmerproj Aug 19, 2019
561cc70
Try to only start the containers and then start after copying
chrisgilmerproj Aug 19, 2019
e7b7e5a
Copy to correct location
chrisgilmerproj Aug 19, 2019
68d9a6f
Use docker-compose for local e2e tests
chrisgilmerproj Aug 20, 2019
23e95f3
Merge master
chrisgilmerproj Aug 20, 2019
8052575
Update docker-compose file to match master
chrisgilmerproj Aug 20, 2019
f2000d4
Fix merge problem with docs
chrisgilmerproj Aug 20, 2019
c347e15
Minor fixes after merge from master
chrisgilmerproj Aug 21, 2019
04c9cbe
Output the spec for debugging purposes
chrisgilmerproj Aug 21, 2019
3193213
One line
chrisgilmerproj Aug 21, 2019
d23cdb2
Remove make targets for db test that use docker since e2e tests don't…
chrisgilmerproj Aug 21, 2019
c679bae
Exit script early if migration did not succeed
chrisgilmerproj Aug 21, 2019
9bcefa2
Remove docker layer caching from steps that don't use docker to speed…
chrisgilmerproj Aug 21, 2019
51aa922
Remove docker ps command
chrisgilmerproj Aug 21, 2019
9f86c1c
Move the migration paths to the same folder. Fix some of the error ch…
chrisgilmerproj Aug 21, 2019
ae2acb1
Capture problems with empty paths
chrisgilmerproj Aug 21, 2019
10f9004
Fix remote docker for server tests
chrisgilmerproj Aug 21, 2019
a078acc
Fix docker cp location
chrisgilmerproj Aug 21, 2019
cdd1e0c
Fix chamber path
chrisgilmerproj Aug 21, 2019
2e4de66
Fix some issues with circleci env vars
chrisgilmerproj Aug 21, 2019
0f3a6f2
Debug logging
chrisgilmerproj Aug 21, 2019
e65494b
Fix the defaults for the migration paths
chrisgilmerproj Aug 21, 2019
67ed563
Modify the message to make sure things are changing in the docker con…
chrisgilmerproj Aug 21, 2019
94599b0
Change wording
chrisgilmerproj Aug 21, 2019
15f4910
Remove any docker cache
chrisgilmerproj Aug 21, 2019
f67f38c
Always pull new image
chrisgilmerproj Aug 21, 2019
7b7fcd9
Fix placement of docker pull and also turn off layer caching
chrisgilmerproj Aug 21, 2019
7b64715
Re-install caching and remove pruning
chrisgilmerproj Aug 21, 2019
edb1829
Remove print debugging
chrisgilmerproj Aug 21, 2019
aa92050
Since we're using set +x no need to have so many echo statements
chrisgilmerproj Aug 21, 2019
434f11a
3 spaces is not a standard anywhere
chrisgilmerproj Aug 21, 2019
a789eb1
Update the dockerignore file
chrisgilmerproj Aug 21, 2019
9c406ab
Fix up local dockerfile for e2e testing
chrisgilmerproj Aug 23, 2019
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -816,6 +816,8 @@ workflows:
requires:
- pre_deps_golang
- pre_deps_yarn
- build_app
- build_migrations
- acceptance_tests_local
# if testing on experimental, you can disable these tests by using the commented block below.
# filters:
@@ -2,7 +2,7 @@ export const longPageLoadTimeout = 10000;
export const fileUploadTimeout = 10000;

// Base URLs
const cypressClientPort = 4000; //change this to 3000 to run cypress against dev instance
const cypressClientPort = process.env.CYPRESS_CLIENT_PORT || 4000; //change this to 3000 to run cypress against dev instance
This conversation was marked as resolved by chrisgilmerproj

This comment has been minimized.

Copy link
@chrisgilmerproj

chrisgilmerproj Aug 1, 2019

Author Contributor

I need to figure out how to modify this port. Alternatively we make docker-compose run on 4000.

export const milmoveBaseURL = `http://milmovelocal:${cypressClientPort}`;
export const officeBaseURL = `http://officelocal:${cypressClientPort}`;
export const tspBaseURL = `http://tsplocal:${cypressClientPort}`;
@@ -16,82 +16,106 @@ TSPHOST=tsplocal
ADMINHOST=adminlocal
ORDERSHOST=orderslocal

# Change the port for tests
HTTP_PORT=4000

# When running locally enable TTY
DOCKER_RUN="docker run -t"
if [ -n "${CIRCLECI+x}" ]; then
echo "RUNNING IN CIRCLECI"
DOCKER_RUN="docker run"
fi

# Check that the docker process is running first
docker system info >> /dev/null

# Begin setup
make e2e_clean client_build db_e2e_init_docker server_build_linux

# Download RDS Certificates
# Dockerfile requires them even though not used locally.
# Might as well download them so we don't have to split Dockerfile
make bin/rds-combined-ca-bundle.pem

# Build the docker container
docker build --tag e2e:latest .
if [[ -n "${CIRCLECI+x}" ]]; then
# CI/CD allows us to use pre-built images using the branch name

# Set Defaults
HTTP_PORT=5000
NETWORK="mymove_default" # Docker network

# Begin Setup
make e2e_clean
# TODO: Consider using "--rmi all" when running locally
docker-compose down || true

# Docker Compose Setup
aws ecr get-login --no-include-email --region us-west-2 --no-include-email | sh
scripts/update-docker-compose
docker-compose up -d
CONTAINER=mymove_milmove_1
else
# Locally development requires that we build the container

# Set Defaults
HTTP_PORT=4000
NETWORK="bridge" # Docker network

# Begin setup
make e2e_clean db_e2e_init_docker client_build server_build_linux

# Download RDS Certificates
# Dockerfile requires them even though not used locally.
# Might as well download them so we don't have to split Dockerfile
make bin/rds-combined-ca-bundle.pem

# Build the docker container
docker build --tag e2e:latest .

# Run the image with the ports changed
$DOCKER_RUN \
-e CLIENT_AUTH_SECRET_KEY \
-e CSRF_AUTH_KEY \
-e DB_ENV="test" \
-e DB_HOST="database" \
-e DB_NAME \
-e DB_PASSWORD \
-e DB_PORT \
-e DB_USER \
-e DEVLOCAL_CA="/config/tls/devlocal-ca.pem" \
-e DEVLOCAL_AUTH="true" \
-e DOD_CA_PACKAGE="/config/tls/Certificates_PKCS7_v5.4_DoD.der.p7b" \
-e DPS_AUTH_COOKIE_SECRET_KEY \
-e DPS_COOKIE_EXPIRES_IN_MINUTES \
-e ENVIRONMENT="test" \
-e HERE_MAPS_APP_CODE \
-e HERE_MAPS_APP_ID \
-e HERE_MAPS_GEOCODE_ENDPOINT \
-e HERE_MAPS_ROUTING_ENDPOINT \
-e HTTP_MY_SERVER_NAME="${MILMOVEHOST}" \
-e HTTP_OFFICE_SERVER_NAME="${OFFICEHOST}" \
-e HTTP_TSP_SERVER_NAME="${TSPHOST}" \
-e HTTP_ADMIN_SERVER_NAME="${ADMINHOST}" \
-e HTTP_ORDERS_SERVER_NAME="${ORDERSHOST}" \
-e IWS_RBS_HOST \
-e LOGIN_GOV_CALLBACK_PORT="${HTTP_PORT}" \
-e LOGIN_GOV_CALLBACK_PROTOCOL \
-e LOGIN_GOV_HOSTNAME \
-e LOGIN_GOV_MY_CLIENT_ID \
-e LOGIN_GOV_OFFICE_CLIENT_ID \
-e LOGIN_GOV_SECRET_KEY \
-e LOGIN_GOV_TSP_CLIENT_ID \
-e LOGIN_GOV_ADMIN_CLIENT_ID \
-e MOVE_MIL_DOD_CA_CERT \
-e MOVE_MIL_DOD_TLS_CERT \
-e MOVE_MIL_DOD_TLS_KEY \
-e NO_TLS_ENABLED=1 \
-e NO_TLS_PORT="${HTTP_PORT}" \
-e SECURE_MIGRATION_DIR \
-e SECURE_MIGRATION_SOURCE \
-e STORAGE_BACKEND=memory \
-p "${HTTP_PORT}:${HTTP_PORT}" \
--expose="${HTTP_PORT}" \
--name="${CONTAINER}" \
--link="milmove-db-test:database" \
--read-only \
--detach \
--rm \
"${CONTAINER}:latest"

# If your container is exiting unexpectedly try uncommenting this code to see the output:
# docker logs -f ${CONTAINER}

# Run the image with the ports changed
$DOCKER_RUN \
-e CLIENT_AUTH_SECRET_KEY \
-e CSRF_AUTH_KEY \
-e DB_ENV="test" \
-e DB_HOST="database" \
-e DB_NAME \
-e DB_PASSWORD \
-e DB_PORT \
-e DB_USER \
-e DEVLOCAL_CA="/config/tls/devlocal-ca.pem" \
-e DEVLOCAL_AUTH="true" \
-e DOD_CA_PACKAGE="/config/tls/Certificates_PKCS7_v5.4_DoD.der.p7b" \
-e DPS_AUTH_COOKIE_SECRET_KEY \
-e DPS_COOKIE_EXPIRES_IN_MINUTES \
-e ENVIRONMENT="test" \
-e HERE_MAPS_APP_CODE \
-e HERE_MAPS_APP_ID \
-e HERE_MAPS_GEOCODE_ENDPOINT \
-e HERE_MAPS_ROUTING_ENDPOINT \
-e HTTP_MY_SERVER_NAME="${MILMOVEHOST}" \
-e HTTP_OFFICE_SERVER_NAME="${OFFICEHOST}" \
-e HTTP_TSP_SERVER_NAME="${TSPHOST}" \
-e HTTP_ADMIN_SERVER_NAME="${ADMINHOST}" \
-e HTTP_ORDERS_SERVER_NAME="${ORDERSHOST}" \
-e IWS_RBS_HOST \
-e LOGIN_GOV_CALLBACK_PORT="${HTTP_PORT}" \
-e LOGIN_GOV_CALLBACK_PROTOCOL \
-e LOGIN_GOV_HOSTNAME \
-e LOGIN_GOV_MY_CLIENT_ID \
-e LOGIN_GOV_OFFICE_CLIENT_ID \
-e LOGIN_GOV_SECRET_KEY \
-e LOGIN_GOV_TSP_CLIENT_ID \
-e LOGIN_GOV_ADMIN_CLIENT_ID \
-e MOVE_MIL_DOD_CA_CERT \
-e MOVE_MIL_DOD_TLS_CERT \
-e MOVE_MIL_DOD_TLS_KEY \
-e NO_TLS_ENABLED=1 \
-e NO_TLS_PORT="${HTTP_PORT}" \
-e SECURE_MIGRATION_DIR \
-e SECURE_MIGRATION_SOURCE \
-e STORAGE_BACKEND=memory \
-p "${HTTP_PORT}:${HTTP_PORT}" \
--expose="${HTTP_PORT}" \
--name="${CONTAINER}" \
--link="milmove-db-test:database" \
--read-only \
--detach \
--rm \
${CONTAINER}:latest

# If your container is exiting unexpectedly try uncommenting this code to see the output:
# docker logs -f ${CONTAINER}
fi

# Grab the IP address of the running container
E2E_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ${CONTAINER})
@@ -109,13 +133,15 @@ if [ -z "${DOCKER_PID}" ]; then
fi

$DOCKER_RUN \
-e CYPRESS_baseUrl=http://${MILMOVEHOST}:4000 \
--add-host ${MILMOVEHOST}:"${E2E_IP}" \
--add-host ${OFFICEHOST}:"${E2E_IP}" \
--add-host ${TSPHOST}:"${E2E_IP}" \
--add-host ${ADMINHOST}:"${E2E_IP}" \
-e CYPRESS_baseUrl="http://${MILMOVEHOST}:${HTTP_PORT}" \
-e CYPRESS_CLIENT_PORT="${HTTP_PORT}" \
--add-host "${MILMOVEHOST}:${E2E_IP}" \
--add-host "${OFFICEHOST}:${E2E_IP}" \
--add-host "${TSPHOST}:${E2E_IP}" \
--add-host "${ADMINHOST}:${E2E_IP}" \
--name="${CONTAINER_CYPRESS}" \
--link="${CONTAINER}" \
--net "${NETWORK}" \
--ipc=host \
--detach \
${CONTAINER_CYPRESS}:latest run --spec "${SPEC:-cypress/integration/**/*}"
@@ -126,6 +152,10 @@ docker logs -f ${CONTAINER_CYPRESS}
# Stop the app container to release the DB connection
docker stop ${CONTAINER}

if [[ -n "${CIRCLECI+x}" ]]; then
docker-compose down
fi

# Copy out the results
docker cp ${CONTAINER_CYPRESS}:/cypress/results cypress/ 2>/dev/null || echo "No cypress results copied"
docker cp ${CONTAINER_CYPRESS}:/cypress/screenshots cypress/ 2>/dev/null || echo "No cypress screenshots copied"
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.