OADC demo commons update release to 2022.03

[xritter1]

Link to Jira ticket if there is one:

Environments

OADC

Description of changes

Update releases to 2022.03

[PlanXCyborg]

gen3.datacommons.io/manifest.json

⚠️ Services on branch

• portal

Deployment changes

• fence
  • Requires a Fence DB migration (chore(migration): add authz column to project fence#1003)
  • Requires these cloud-automation updates to revproxy: (PXP-7565): Exclude Fence register-user blueprint from CSRF check cloud-automation#1580 ((PXP-7565): User registration, plus CSRF changes fence#906)
  • By default Fence will allow only visas issued by RAS staging, RAS prod, and itself; if a different set of allowed visa issuers is needed, edit the GA4GH_VISA_ISSUER_ALLOWLIST config value. Otherwise, no action needed. ((PXP-6339): RAS visa validation fence#933)
  • /link/google will DENY requests with "redirect" param values that aren't in the "allowed redirects" for OAuth2/OIDC Clients in Fence (You have to add any valid redirects to the client before deploying this) ((PXP-8286): fix(google): add redirect validation to /link/google fence#927)
  • For existing deployments with "ENABLE_AUDIT_LOGS" turned ON, to DISABLE pushing audit logs to an AWS SQS, the configuration "PUSH_AUDIT_LOGS_CONFIG.type" must be changed to "api" (default: "aws_sqs"). If audit-service has never been deployed, no action required. (PXP-7805 Push audit logs to an AWS SQS fence#923)
  • To ENABLE pushing audit logs to an AWS SQS (REQUIRES audit-service 1.0.0 or more recent), create/update the configuration file's "PUSH_AUDIT_LOGS_CONFIG" field ("type: aws_sqs" (default), and the SQS URL and region can be copied from the audit-service configuration - or run gen3 sqs info $(gen3 api safe-name audit-sqs)), see https://github.com/uc-cdis/fence/blob/fec3dd937d68384e86b2ed6f9b53daf0026c2df9/fence/config-default.yaml#L632-L636. Then run kubectl delete secret fence-config and gen3 kube-setup-fence (PXP-7805 Push audit logs to an AWS SQS fence#923)
• hatchery
  • Add pay-models-dynamodb-table to the Hatchery configuration file to fetch pay model data from DynamoDB (PR fix(dataguids.org): add manifest.json #35) (PXP-8779 HP-347 Feat/ecs hatchery#38)
  • Update the Hatchery configuration file pay-models block to change the user field name to user_id (PR fix(dataguids.org): add manifest.json #35) (PXP-8779 HP-347 Feat/ecs hatchery#38)
  • Add gen3-volume-location to the Hatchery configuration file for each container to overwrite the directory that API key will be saved to (PR fix(account): portal to master #33) (PXP-8779 HP-347 Feat/ecs hatchery#38)
  • Update the Hatchery configuration file pay-models block to add option ecs for enabling ECS decoupled workspace (PXP-8779 HP-347 Feat/ecs hatchery#38)
  • Update the Hatchery configuration file pay-models block to add option subnet for setup transit gateway subnet addresses (PXP-8779 HP-347 Feat/ecs hatchery#38)
  • Add pay-models-dynamodb-table to the Hatchery configuration file to fetch pay model data from DynamoDB (PXP-8341 Fetch pay models from DynamoDB hatchery#35)
  • Update the Hatchery configuration file pay-models block to change the user field name to user_id (PXP-8341 Fetch pay models from DynamoDB hatchery#35)
• indexd
  • Include example secrets to load PostgreSQL settings for indexd (Add files for psql ssl support indexd#317)
• metadata
  • A job has been created to make aggregate MDS population more robust feat(agg-mds-job): add new agg mds job cloud-automation#1722 (feat(populate): improved populate script, docs metadata-service#33)
  • Must set the appropriate AGG_MDS_NAMESPACE value in metadata.env (Feat/mds adapter metadata-service#29)
  • Per chore(metadata): migrate agg MDS configs #3332 a metadata/aggregate_config.json file should be created as part of the manifest config for a commons and the appropriate environment vars for metadata should be defined in the manifest JSON (chore(configs): remove configs from source control metadata-service#30)
  • The new command to populate an aggregate MDS is kubectl exec $(gen3 pod metadata) -- python /src/src/mds/populate.py --config /aggregate_config.json --hostname esproxy-service --port 9200 (chore(configs): remove configs from source control metadata-service#30)
  • Network rules are updated to allow for elasticsearch to work with metadata-service feat(ecosystems): elasticsearch for MDS cloud-automation#1638 (Feat/aggregation metadata-service aggregate APIs metadata-service#27)
  • USE_AGG_MDS must be set to true to enable aggregate metadata APIs (Feat/aggregation metadata-service aggregate APIs metadata-service#27)
• sower
  • PS: This is follow-up on this one: feat(dcf): dcf prod release 2.2.7 #24 ([PXP-8941] [PXP-8859] Feat/update go 1.17 sower#25)
• ssjdispatcher
  • PS: This is follow-up on this one: Kids First ETL Mapping #39 ([PXP-8943] [PXP-8859] Feat/update go 1.17 ssjdispatcher#40)

Breaking changes

• fence
  • /link/google will DENY requests with "redirect" param values that aren't in the "allowed redirects" for OAuth2/OIDC Clients in Fence ((PXP-8286): fix(google): add redirect validation to /link/google fence#927)
• metadata
  • Do not publish GitHub Packages (feat: do not publish GitHub Packages metadata-service#42)
  • ES indices will not be namespaced per environment (Feat/mds adapter metadata-service#29)
• sheepdog
  • Sheepdog now validates formats. If your dictionary uses formats and you have existing data or submission pipelines that do not respect these formats, this might be a breaking change (PXP-9192 Validate format sheepdog#363)