CBA deployment prep #115

chuckwondo · 2023-02-10T19:50:10Z

Modify deployment to allow for deployment to both our "old" AWS accounts as well as our "new" CBA accounts.

@krisstanton and @jsrikish, before approving this PR, I want you both to be able to successfully deploy to the CBA sandbox account, as follows:

First, fetch the code:

git fetch
git checkout cba-deployment-prep
git pull

Next, update your .env file as follows:

Comment out your current values for AWS_PROFILE, TS_ENV, and TF_VAR_cumulus_distribution_uri
Add new lines for each of those same 3 variables:
- Set AWS_PROFILE to the profile you created for the CBA sandbox (perhaps csda-cumulus-sbx-7894)
- Set TS_ENV to something slightly different, perhaps the same as the TS_ENV you just commented out, but with cba as the suffix.
- Initially comment out TF_VAR_cumulus_distribution_url= without a value (you won't know the value until after the first deployment)

Next, setup and deploy everything. You will eventually (after 30-60 minutes?) be prompted to supply your username and password for your account at https://uat.urs.earthdata.nasa.gov/:

make pre-deploy-setup
make all-up-yes

You may have to run make all-up-yes 2 or 3 times for a clean deploy.

Once you get a clean deploy, you have to set TF_VAR_cumulus_distribution_url as follows:

Run make output-cumulus
Find value of cumulus_distribution_api_uri and set that as the value of TF_VAR_cumulus_distribution_url in your .env file (and uncomment the line)
Run make all-up-yes one last time (hopefully)

- Since "all" commands redirect log messages to log files (instead of stdout), automatically tail log files during "all up" commands to avoid the need to open a separate terminal to manually tail them. - Since `make terraform-doctor-STACK` commands parse the Terraform "up" log files, but `make up-STACK` commands do not send log messages to the log files, use `tee` to also send stdout to the log files so that `make terraform-doctor-STACK` can also be used after `make up-STACK` runs. - Since `make pre-deploy-setup` calls `terraspace all init`, which sends output to the log files, automatically tail the log files for convenience.

jsrikish · 2023-02-13T16:07:57Z

Changed the .env file with TS_ENV=jayanthi-sbx
make pre-deploy-setup
bundler: failed to load command: terraspace (/opt/terraspace/embedded/bin/terraspace)
/opt/terraspace/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.168.4/lib/aws-sdk-core/param_validator.rb:35:in `validate!': missing required parameter params[:role_arn] (ArgumentError)
................................
................................
ERROR: No S3 bucket specified

Usage: bin/copy-launchpad-pfx.sh BUCKET

chuckwondo · 2023-02-13T21:52:05Z

Did you also set AWS_PROFILE (in .env) to the CBA sandbox profile you created?

Once you make sure AWS_PROFILE is set correctly, completely rebuild the Docker image:

docker builder prune -af
make docker

Then try again:

make pre-deploy-setup

krisstanton · 2023-02-14T16:39:24Z

I got my sandbox deploy all the way to the smoke test stage and found an error that I think requires a code change.
The Error 'NoSuchBucket' (and MANY similar errors) happens when trying to upload granule samples to this location.
s3://csdap-cumulus-kris-sbx7894-provider/storage-ss-ingest-prod-ingesteddata-uswest2/planet/PSScene3Band-20171201_031958_0f31/20171201_031958_0f31_thumb

I looked in S3 and found that the below bucket exists
s3://csda-cumulus-kris-sbx7894-provider-7894
The bucket that the smoke test is looking for to upload is
s3://csdap-cumulus-kris-sbx7894-provider
Just to note, my .env file has a line that looks like this
TS_ENV=kris-sbx7894

To fix this,
Somewhere in the the smoke test code needs to:
Need to remove a 'p' (character position 4) and need to add '-7894' to the end of the bucket name.

I looked a little bit deeper and found the parts of the code which set this.
Makefile --> (create-test-data) ---> bin/create-test-data.sh

Fixing the code
The exact code that sets this is in the file:
bin/create-test-data.sh

Line: 11: _bucket_prefix=$(test "${TS_ENV}" == "uat" && echo "csdap-uat-" || echo "csdap-${CUMULUS_PREFIX}-")
If my .env file is:
TS_ENV=kris-sbx7894
This state of line 11 sets _bucket_prefix to
csdap-cumulus-kris-sbx7894-
When it should set it to
csda-cumulus-kris-sbx7894-

Line: 14 _provider_bucket=${_bucket_prefix}provider
From the above example '_provider_bucket' gets set to
csdap-cumulus-kris-sbx7894-provider
When it should be set to
csda-cumulus-kris-sbx7894-provider-7894

chuckwondo · 2023-02-14T17:02:33Z

@krisstanton, this ticket is for deployment only, not for dealing with test data.

I'm working on a separate branch for us to deal with test data, which addresses the issue you pointed out, among many other things that we also need.

Approve this PR if your deployment succeeded. Skip creating test data at this point.

krisstanton · 2023-02-14T17:16:21Z

Sorry, I was under the impression that we had to get a smoke test pass before it counted as a complete deploy.
Understood about separate branch for the smoke test issue. Thanks for the clarity.

The only other thing I should mention before clicking approve is that when I run
make output-cumulus
I get a blank space for cumulus_distribution_api_uri (see below).
However, when I go to AWS API Gateway, the distribution API is present and has an address.

`
➜ csdap-cumulus git:(cba-deployment-prep) ✗ make output-cumulus

// OUTPUT
sha256:<LONG_STRING_HERE>
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Building .terraspace-cache/us-west-2/kris-sbx7894/stacks/cumulus
Current directory: .terraspace-cache/us-west-2/kris-sbx7894/stacks/cumulus
=> terraform output
archive_api_redirect_uri = https://t9maohtgsi.execute-api.us-west-2.amazonaws.com:8000/dev/token
archive_api_uri = https://t9maohtgsi.execute-api.us-west-2.amazonaws.com:8000/dev/
cumulus_distribution_api_redirect_uri = login
cumulus_distribution_api_uri =
report_executions_sns_topic_arn = arn:aws:sns:us-west-2:772632177894:cumulus-kris-sbx7894-report-executions-topic
report_granules_sns_topic_arn = arn:aws:sns:us-west-2:772632177894:cumulus-kris-sbx7894-report-granules-topic
report_pdrs_sns_topic_arn = arn:aws:sns:us-west-2:772632177894:cumulus-kris-sbx7894-report-pdrs-topic
stepfunction_event_reporter_queue_url = https://sqs.us-west-2.amazonaws.com/772632177894/cumulus-kris-sbx7894-sfEventSqsToDbRecordsInputQueue
`

So I can just manually grab the URI, which for my deploy is: https://gc86mo52vh.execute-api.us-west-2.amazonaws.com/dev/

Sorry if this issue is also out of scope. Let me know if so and I'll click approve.

Thanks!

chuckwondo · 2023-02-14T17:37:14Z

@krisstanton, if you did your initial deployment with the following line in your .env file, then perhaps that's why the uri in the outputs has no value:

TF_VAR_cumulus_distribution_url=

Perhaps you didn't have that line commented out on the first deploy?

I'm going to update the original description to indicate that the line should be commented out on the first deploy.

krisstanton · 2023-02-14T19:29:52Z

Perhaps you didn't have that line commented out on the first deploy?

You are absolutely right. I had not commented that line out. That makes sense. Ok going to click approve now.
Thanks!

chuckwondo · 2023-02-21T16:27:14Z

@jsrikish, if you're still having a problem with this, let's do the following:

Paste the contents of your .env file in a reply to this comment
Run git log -1 --oneline and paste the output in a reply

If I cannot discern the issue from this information, then let's setup a meeting to debug things.

jsrikish · 2023-02-21T16:28:32Z

Did you also set AWS_PROFILE (in .env) to the CBA sandbox profile you created?

Once you make sure AWS_PROFILE is set correctly, completely rebuild the Docker image:
docker builder prune -af
make docker
Then try again:
make pre-deploy-setup

I tried the above commands but it failed in docker pre-deploy-setup with the same error

jsrikish · 2023-02-21T16:38:08Z

@jsrikish, if you're still having a problem with this, let's do the following:

Paste the contents of your .env file in a reply to this comment

Run git log -1 --oneline and paste the output in a reply

If I cannot discern the issue from this information, then let's setup a meeting to debug things.

These might be the lines you are looking for:

TS_ENV=jayanthi-sbx7894
AWS_REGION=us-west-2
AWS_PROFILE=csda-cumulus-sbx-7894
TF_VAR_cumulus_distribution_url=

jsrikish · 2023-02-21T16:40:40Z

2. git log -1 --oneline

git log -1 --oneline 5dbf0ed (HEAD -> cba-deployment-prep, origin/cba-deployment-prep) Fix use of is_cba? and is_sandbox?

chuckwondo · 2023-02-21T17:14:53Z

I think there's an additional command you need to run, but I missed listing it, so let's do this:

docker builder prune -af
docker system prune -f
DOCKER_BUILD_OPTS= make docker

I think without that 2nd "prune" command, your image was not being fully removed, so the image build was still hanging onto some old layers that might be the cause of the problem.

I think that should get your Docker container sorted out, and you should then try this again:

make pre-deploy-setup

🤞

chuckwondo · 2023-02-21T17:23:57Z

@jsrikish, also comment out this line in your .env file:

TF_VAR_cumulus_distribution_url=

Should be:

#TF_VAR_cumulus_distribution_url=

jsrikish · 2023-02-21T17:42:05Z

I think there's an additional command you need to run, but I missed listing it, so let's do this:
docker builder prune -af
docker system prune -f
DOCKER_BUILD_OPTS= make docker
I think without that 2nd "prune" command, your image was not being fully removed, so the image build was still hanging onto some old layers that might be the cause of the problem.

I think that should get your Docker container sorted out, and you should then try this again:
make pre-deploy-setup
🤞

Didn't work :(

chuckwondo · 2023-02-21T18:06:13Z

OK, let's remove some docker volumes that might be causing problems:

docker volume list | grep -o 'csdap-cumulus[.].*' | xargs -L1 -t docker volume rm

Then try make pre-deploy-setup again.

jsrikish · 2023-02-21T19:48:02Z

docker volume list | grep -o 'csdap-cumulus[.].*' | xargs -L1 -t docker volume rm

Still the same error :(

chuckwondo · 2023-02-21T23:10:49Z

@jsrikish, would you mind pasting into a reply the entire error output your getting?

chuckwondo added 8 commits February 7, 2023 13:05

Be more specific about which stack to destroy

d94b675

Add AWS_PROFILE value to prompt.

0b73649

Inflate launchpad.pfx from secret.

8918b7c

Eliminate prompts for non-dev secrets in CBA sandbox

3c3a907

Support both CBA and non-CBA deployments.

2bbebf8

Tail logs during "all init" command

b61accd

Fix use of is_cba? and is_sandbox?

5dbf0ed

chuckwondo temporarily deployed to uat February 10, 2023 19:50 — with GitHub Actions Inactive

chuckwondo requested review from jsrikish and krisstanton February 10, 2023 19:50

krisstanton approved these changes Feb 14, 2023

View reviewed changes

krisstanton mentioned this pull request Feb 14, 2023

Standup Cumulus development deployment in CBA sandbox account for Kris #92

Closed

2 tasks

chuckwondo merged commit 885cd87 into main Feb 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CBA deployment prep #115

CBA deployment prep #115

chuckwondo commented Feb 10, 2023 •

edited

Loading

jsrikish commented Feb 13, 2023

chuckwondo commented Feb 13, 2023

krisstanton commented Feb 14, 2023

chuckwondo commented Feb 14, 2023

krisstanton commented Feb 14, 2023 •

edited

Loading

chuckwondo commented Feb 14, 2023 •

edited

Loading

krisstanton commented Feb 14, 2023

chuckwondo commented Feb 21, 2023

jsrikish commented Feb 21, 2023

jsrikish commented Feb 21, 2023 •

edited by chuckwondo

Loading

jsrikish commented Feb 21, 2023

chuckwondo commented Feb 21, 2023 •

edited

Loading

chuckwondo commented Feb 21, 2023

jsrikish commented Feb 21, 2023

chuckwondo commented Feb 21, 2023

jsrikish commented Feb 21, 2023

chuckwondo commented Feb 21, 2023

CBA deployment prep #115

CBA deployment prep #115

Conversation

chuckwondo commented Feb 10, 2023 • edited Loading

jsrikish commented Feb 13, 2023

chuckwondo commented Feb 13, 2023

krisstanton commented Feb 14, 2023

chuckwondo commented Feb 14, 2023

krisstanton commented Feb 14, 2023 • edited Loading

chuckwondo commented Feb 14, 2023 • edited Loading

krisstanton commented Feb 14, 2023

chuckwondo commented Feb 21, 2023

jsrikish commented Feb 21, 2023

jsrikish commented Feb 21, 2023 • edited by chuckwondo Loading

jsrikish commented Feb 21, 2023

chuckwondo commented Feb 21, 2023 • edited Loading

chuckwondo commented Feb 21, 2023

jsrikish commented Feb 21, 2023

chuckwondo commented Feb 21, 2023

jsrikish commented Feb 21, 2023

chuckwondo commented Feb 21, 2023

chuckwondo commented Feb 10, 2023 •

edited

Loading

krisstanton commented Feb 14, 2023 •

edited

Loading

chuckwondo commented Feb 14, 2023 •

edited

Loading

jsrikish commented Feb 21, 2023 •

edited by chuckwondo

Loading

chuckwondo commented Feb 21, 2023 •

edited

Loading