Gen3 Monthly Release 2021.09 va.data-commons.org 1632430540

[themarcelor]

Applying version 2021.09 to va.data-commons.org

[PlanXCyborg]

va.data-commons.org/manifest.json

Deployment changes

• fence
  • Requires these cloud-automation updates to revproxy: (PXP-7565): Exclude Fence register-user blueprint from CSRF check cloud-automation#1580 ((PXP-7565): User registration, plus CSRF changes fence#906)
  • By default Fence will allow only visas issued by RAS staging, RAS prod, and itself; if a different set of allowed visa issuers is needed, edit the GA4GH_VISA_ISSUER_ALLOWLIST config value. Otherwise, no action needed. ((PXP-6339): RAS visa validation fence#933)
  • /link/google will DENY requests with "redirect" param values that aren't in the "allowed redirects" for OAuth2/OIDC Clients in Fence (You have to add any valid redirects to the client before deploying this) ((PXP-8286): fix(google): add redirect validation to /link/google fence#927)
  • For existing deployments with "ENABLE_AUDIT_LOGS" turned ON, to DISABLE pushing audit logs to an AWS SQS, the configuration "PUSH_AUDIT_LOGS_CONFIG.type" must be changed to "api" (default: "aws_sqs"). If audit-service has never been deployed, no action required. (PXP-7805 Push audit logs to an AWS SQS fence#923)
  • To ENABLE pushing audit logs to an AWS SQS (REQUIRES audit-service 1.0.0 or more recent), create/update the configuration file's "PUSH_AUDIT_LOGS_CONFIG" field ("type: aws_sqs" (default), and the SQS URL and region can be copied from the audit-service configuration - or run gen3 sqs info $(gen3 api safe-name audit-sqs)), see https://github.com/uc-cdis/fence/blob/fec3dd937d68384e86b2ed6f9b53daf0026c2df9/fence/config-default.yaml#L632-L636. Then run kubectl delete secret fence-config and gen3 kube-setup-fence (PXP-7805 Push audit logs to an AWS SQS fence#923)
• mariner
  • Removed netpolicy "off" from documentation for Mariner deployment, it has been resolved in this PR. (docs(readme): update mariner readme mariner#48)
  • New pre commit config requires detect-secrets==1.0.3 (docs(readme): update mariner readme mariner#48)
• metadata
  • Network rules are updated to allow for elasticsearch to work with metadata-service feat(ecosystems): elasticsearch for MDS cloud-automation#1638 (Feat/aggregation metadata-service aggregate APIs metadata-service#27)
  • USE_AGG_MDS must be set to true to enable aggregate metadata APIs (Feat/aggregation metadata-service aggregate APIs metadata-service#27)
• portal
  • Requires loginForDownload flag set to true in gitops (PXP-8448 - Register users for download data-portal#912)
  • User registration before download feature depends on (PXP-7565): User registration, plus CSRF changes fence#906 (PXP-8448 - Register users for download data-portal#912)
  • Requires the configuration of the batch-export sower job: Batch export sower job sower-jobs#29 in the environment's manifest. This job also has its own setup in cloud automation: Feat/batch export cloud-automation#1708 (Add UI components and polling for zip downloads data-portal#906)
  • 2 new fields in gitops.json: discoveryconfig.features.exportToWorkspace.[enableDownloadZip (bool) downloadZipButtonText (string)] (Add UI components and polling for zip downloads data-portal#906)
  • Requires PXP-8364 Fix/portal csp cloud-automation#1687, otherwsie Indexing page may not work properly (PXP-8364 fix: strict csp data-portal#893)
  • Portal config field discoveryConfig.features.exportToWorkspaceBETA has been renamed to discoveryConfig.features.exportToWorkspace (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • Note that there are several other new configurations introduced for the Discovery page, please refer to this diff (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • New GEN3_BUNDLE option heal introduced for buliding a HEAL bundle (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • To enable DD RUM, must create a portal-datadog-config k8s secret with datadog_application_id and datadog_client_token in it, see DEVOPS-62 feat: add datadog to portal cloud-automation#1651 (DEVOPS-62 Feat/datadog data-portal#881)

[PlanXCyborg]

va.data-commons.org/manifest.json

Deployment changes

• fence
  • Requires these cloud-automation updates to revproxy: (PXP-7565): Exclude Fence register-user blueprint from CSRF check cloud-automation#1580 ((PXP-7565): User registration, plus CSRF changes fence#906)
• mariner
  • Removed netpolicy "off" from documentation for Mariner deployment, it has been resolved in this PR. (docs(readme): update mariner readme mariner#48)
  • New pre commit config requires detect-secrets==1.0.3 (docs(readme): update mariner readme mariner#48)
• portal
  • Requires loginForDownload flag set to true in gitops (PXP-8448 - Register users for download data-portal#912)
  • User registration before download feature depends on (PXP-7565): User registration, plus CSRF changes fence#906 (PXP-8448 - Register users for download data-portal#912)
  • Requires the configuration of the batch-export sower job: Batch export sower job sower-jobs#29 in the environment's manifest. This job also has its own setup in cloud automation: Feat/batch export cloud-automation#1708 (Add UI components and polling for zip downloads data-portal#906)
  • 2 new fields in gitops.json: discoveryconfig.features.exportToWorkspace.[enableDownloadZip (bool) downloadZipButtonText (string)] (Add UI components and polling for zip downloads data-portal#906)
  • Requires PXP-8364 Fix/portal csp cloud-automation#1687, otherwsie Indexing page may not work properly (PXP-8364 fix: strict csp data-portal#893)
  • Portal config field discoveryConfig.features.exportToWorkspaceBETA has been renamed to discoveryConfig.features.exportToWorkspace (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • Note that there are several other new configurations introduced for the Discovery page, please refer to this diff (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • New GEN3_BUNDLE option heal introduced for buliding a HEAL bundle (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)