Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build tasks inside docker locally for pkcs11 support #3471

Merged
merged 10 commits into from Feb 5, 2020

Conversation

@chrisgilmerproj
Copy link
Contributor

chrisgilmerproj commented Feb 5, 2020

Description

There are several issues this is addressing:

PKCS11 support in our repo means we can't cross compile this binary for linux on macOS to test locally inside Docker. So instead of cross compiling on macOS this adds a builder container for the binary and builds inside Docker.

The assets.go code changes when created inside a linux environment instead of a macOS environment. Having this file checked into the repo doesn't help our build pipeline. To fix this the build steps were changed to rebuild this file from scratch before making the milmove-tasks binary.

The ECS deploy binary incorrectly formatted the execution role, task role, and family name for the ECS scheduled tasks. The CircleCI steps additionally pointed to the wrong docker image. Fixing these steps fixes deploys for the tasks.

Impact

The ECS Schedule Tasks broke sometime between Dec 6, 2019 and Jan 6, 2020. We know this because the fuel prices come out on the 6th of each month and only the January data was missing, meaning the December fuel prices were downloaded correctly.

Part of the debugging issue was the merge of the PKCS11 code in #3395 on Feb 4, 2020. However, this only complicated debugging locally and should not have had any impact on the compiled binaries.

Part of the issue was the merging of #3229 on Jan 15. If this was the original problem the fuel prices would have been downloaded on Jan 6 but were not.

Part of the issue was the compiled assets.go file, which we've been merging since Sept, 2019. However, I'd expect that we would have seen this issue sooner.

Finally, it appears that Fuel Price data has not been needed since removing HHG from the application, but it will be important in the next couple weeks. Also, its unclear that missing emails have caused any issues.

Setup

Try out the tasks:

make db_dev_run db_dev_migrate
make tasks_save_fuel_price_data
make tasks_send_post_move_survey

Also

rm -f bin/ecs-deploy && make bin/ecs-deploy
./bin/ecs-deploy task-def \
  --aws-account-id "${AWS_ACCOUNT_ID}" \
  --aws-region "${AWS_DEFAULT_REGION}" \
  --service app-tasks \
  --environment experimental \
  --image "923914045601.dkr.ecr.us-west-2.amazonaws.com/app-tasks:git-3554bda70a75c75548b1bd47d01b4f852dea5427" \
  --cpu 256 \
  --memory 512 \
  --variables-file "./config/env/experimental.save-fuel-price-data.env" \
  --entrypoint "/bin/milmove-tasks save-fuel-price-data" \
  --dry-run | jq .
@chrisgilmerproj chrisgilmerproj self-assigned this Feb 5, 2020
This reverts commit 3554bda.
@chrisgilmerproj chrisgilmerproj requested review from transcom/truss-infra and tinyels Feb 5, 2020
@chrisgilmerproj chrisgilmerproj marked this pull request as ready for review Feb 5, 2020
@mr337
mr337 approved these changes Feb 5, 2020
Copy link
Contributor

mr337 left a comment

This looks to be working great. 🚢

fuel price

2020-02-05T23:19:18.831Z        DEBUG   milmove-tasks/save_fuel_price_data.go:26        checking config
2020-02-05T23:19:18.831Z        INFO    cli/dbconn.go:257       Connecting to the database      {"url": "postgres://postgres:*****@database:5432/dev_db?sslmode=disable", "db-ssl-root-cert": ""}
2020-02-05T23:19:18.831Z        INFO    cli/dbconn.go:362       Starting database ping....
2020-02-05T23:19:18.838Z        INFO    cli/dbconn.go:369       ...DB ping successful!
2020-02-05T23:19:20.007Z        INFO    fuelprice/diesel_fuel_price_storer.go:131       Fuel Data added
        {"start date month": "February", "pubDate": "2020-02-03T00:00:00.000Z"}
2020-02-05T23:19:20.009Z        INFO    fuelprice/diesel_fuel_price_storer.go:131       Fuel Data added
        {"start date month": "January", "pubDate": "2020-01-06T00:00:00.000Z"}
2020-02-05T23:19:20.011Z        INFO    fuelprice/diesel_fuel_price_storer.go:131       Fuel Data added
        {"start date month": "December", "pubDate": "2019-12-02T00:00:00.000Z"}
2020-02-05T23:19:20.013Z        INFO    fuelprice/diesel_fuel_price_storer.go:131       Fuel Data added
        {"start date month": "November", "pubDate": "2019-11-04T00:00:00.000Z"}
2020-02-05T23:19:20.015Z        INFO    fuelprice/diesel_fuel_price_storer.go:131       Fuel Data added
        {"start date month": "October", "pubDate": "2019-10-07T00:00:00.000Z"}
2020-02-05T23:19:20.016Z        INFO    fuelprice/diesel_fuel_price_storer.go:131       Fuel Data added
        {"start date month": "September", "pubDate": "2019-09-02T00:00:00.000Z"}
2020-02-05T23:19:20.018Z        INFO    fuelprice/diesel_fuel_price_storer.go:131       Fuel Data added
        {"start date month": "August", "pubDate": "2019-08-05T00:00:00.000Z"}
2020-02-05T23:19:20.020Z        INFO    fuelprice/diesel_fuel_price_storer.go:131       Fuel Data added
        {"start date month": "July", "pubDate": "2019-07-01T00:00:00.000Z"}
2020-02-05T23:19:20.022Z        INFO    fuelprice/diesel_fuel_price_storer.go:131       Fuel Data added
        {"start date month": "June", "pubDate": "2019-06-03T00:00:00.000Z"}
2020-02-05T23:19:20.023Z        INFO    fuelprice/diesel_fuel_price_storer.go:131       Fuel Data added
        {"start date month": "May", "pubDate": "2019-05-06T00:00:00.000Z"}

survey move

2020-02-05T23:19:49.144Z        DEBUG   milmove-tasks/send_post_move_survey_email.go:31 checking config
2020-02-05T23:19:49.144Z        INFO    cli/dbconn.go:257       Connecting to the database      {"url": "postgres://postgres:*****@database:5432/dev_db?sslmode=disable", "db-ssl-root-cert": ""}
2020-02-05T23:19:49.144Z        INFO    cli/dbconn.go:362       Starting database ping....
2020-02-05T23:19:49.147Z        INFO    cli/dbconn.go:369       ...DB ping successful!
2020-02-05T23:19:49.147Z        INFO    notifications/notification_sender.go:86 Using local email backend       {"domain": "milmovelocal"}
2020-02-05T23:19:49.164Z        INFO    notifications/move_reviewed.go:88       no emails to be sent    {"date": "2020-01-21 23:19:49.147806486 +0000 UTC"}

@chrisgilmerproj chrisgilmerproj merged commit bb13d85 into master Feb 5, 2020
15 checks passed
15 checks passed
ci/circleci: acceptance_tests_experimental Your tests passed on CircleCI!
Details
ci/circleci: acceptance_tests_local Your tests passed on CircleCI!
Details
ci/circleci: acceptance_tests_staging Your tests passed on CircleCI!
Details
ci/circleci: build_app Your tests passed on CircleCI!
Details
ci/circleci: build_migrations Your tests passed on CircleCI!
Details
ci/circleci: build_storybook_app Your tests passed on CircleCI!
Details
ci/circleci: build_tasks Your tests passed on CircleCI!
Details
ci/circleci: build_tools Your tests passed on CircleCI!
Details
ci/circleci: check_generated_code Your tests passed on CircleCI!
Details
ci/circleci: client_test Your tests passed on CircleCI!
Details
ci/circleci: integration_tests Your tests passed on CircleCI!
Details
ci/circleci: pre_deps_golang Your tests passed on CircleCI!
Details
ci/circleci: pre_deps_yarn Your tests passed on CircleCI!
Details
ci/circleci: pre_test Your tests passed on CircleCI!
Details
ci/circleci: server_test Your tests passed on CircleCI!
Details
@chrisgilmerproj chrisgilmerproj deleted the cg_build_tasks_inside_docker_locally branch Feb 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.