TTAHUB System Operations

System Overview

• An overview of our infrastructure can be found in the boundary diagram
• Persistent infrastructure is created, configured, and maintained with terraform scripts
• The File Scanning API is a ClamAV docker container & go API
• Application code is deployed via the last step of our CI/CD pipeline.
• Production pipeline runs

Production Deployments

Deploying the application to the production environment requires the approval of both the Product Owner and Government Technical Monitor, or Vendor Tech Lead in the absence of a GTM role. To achieve this:

1. The first approval is given by one of the PO, GTM, or Vendor tech lead opening a Pull Request to merge the current main branch into the production branch and request a review from the other person.
2. The second approval is given through the PR review mechanism.
3. The PR can then be merged at the time that the deploy is to be done.

Note: if a PR is prematurely marked as approved, the fix depends on whether the merge has occurred yet or not.

• If not merged -> submit a new review with "request changes" state
• If merged already -> have a conversation with Krys in Slack about whether to revert the merge or submit a new PR with any additional changes.

Useful Process Links

• Security Processes and Procedures
• Access Control & Account Management SOP
• PR Workflows
• Configuration Management
  • Information on setting environment variables is found in the Continuous Deployment section of the README
  • Formal Configuration Management Plan

Useful scripts

• There is a script that will dump to the audit logs counts for various models, including breakdowns by region and status for ActivityReports and Grants. See the cli script code for documentation on how to run in each environment.

SSH Access

SSH access is allowed in non-production spaces, and can be enabled using the steps under Production data access for production in case of emergency. It is not allowed by default in production because SSH access allows for actions to be taken that will not be automatically tracked by various audit mechanisms, including data exfiltration.

SSH

cf ssh APP_NAME

SCP

1. Get a one-time code to use as the password: cf ssh-code
2. Get the app's GUID: cf curl /v3/apps/$(cf app APP_NAME --guid)/processes | jq --raw-output '.resources | .[] | select(.type == "web").guid'
3. scp -P 2222 -o User=cf:<<GUID FROM STEP 2>>/0 ssh.fr.cloud.gov:<<PATH TO FILE YOU WANT TO COPY>> <<LOCAL PATH>>
4. You'll be prompted for a password, use the code you got in step 1.

Production Data Access

If access to the production data set is required, pg_dump can be used via cf-connect-service. This only works if ssh access is enabled, which is not the case for the production space. In order to get a production db copy, follow these steps:

1. Create a screensharing video call between at least the Product Owner, GTM, and Vendor Tech Lead. This coworking is to ensure that proper procedures are followed while working in a way that cannot be tracked via existing audit logs
2. While sharing screen and working on Government Furnished Equipment, enable space ssh access cf allow-space-ssh ttahub-prod
3. Follow cloud.gov db backup steps
4. Disable space ssh access cf disallow-space-ssh ttahub-prod
5. Immediately wipe any production data from local storage when the investigation is done

Incident Response & Contingency Plan

• AdHoc Incident Response plan
• Incident Response plan
• Contingency Plan

TLDR;

1. If any issues are suspected, contact the Product Owner and Government Technical Monitor
2. They will activate the appropriate plan in consultation with the Vendor Technical Lead

Helpful Links

• Main system documents folder
• OCIO Governance documents folder
• System help policies

Points of Contact

Position	Name	Email	Phone
Product Owner	Patrice Pascual	patrice.pascual@acf.hhs.gov
Vendor Technical Lead	Krys Wisnaskas	krystyna@adhocteam.us
Vendor Product Manager	Angela Waner	angela.waner@adhocteam.us