Feat/midrc staging

[ajoaugustine]

Link to Jira ticket if there is one:
GPE-143

Environments

MIDRC staging

Description of changes

Updated midrc staging to match MIDRC prod

[PlanXCyborg]

staging.midrc.org/manifest.json

Deployment changes

• indexd
  • Include example secrets to load PostgreSQL settings for indexd (Add files for psql ssl support indexd#317)
• metadata
  • A job has been created to make aggregate MDS population more robust feat(agg-mds-job): add new agg mds job cloud-automation#1722 (feat(populate): improved populate script, docs metadata-service#33)
  • Must set the appropriate AGG_MDS_NAMESPACE value in metadata.env (Feat/mds adapter metadata-service#29)
  • Per chore(metadata): migrate agg MDS configs #3332 a metadata/aggregate_config.json file should be created as part of the manifest config for a commons and the appropriate environment vars for metadata should be defined in the manifest JSON (chore(configs): remove configs from source control metadata-service#30)
  • The new command to populate an aggregate MDS is kubectl exec $(gen3 pod metadata) -- python /src/src/mds/populate.py --config /aggregate_config.json --hostname esproxy-service --port 9200 (chore(configs): remove configs from source control metadata-service#30)
• portal
  • To enable this check, set features.exportToWorkspace.verifyExternalLogins flag to true in gitops (Check all WTS OIDCs are signed in before download/export to WS data-portal#921)
  • Misc: add option for bundle name: ecosystem, which is the same as heal bundle under the hood (Chore/discovery for more bundles data-portal#918)
  • Requires loginForDownload flag set to true in gitops (PXP-8448 - Register users for download data-portal#912)
  • User registration before download feature depends on (PXP-7565): User registration, plus CSRF changes fence#906 (PXP-8448 - Register users for download data-portal#912)
  • Requires the configuration of the batch-export sower job: Batch export sower job sower-jobs#29 in the environment's manifest. This job also has its own setup in cloud automation: Feat/batch export cloud-automation#1708 (Add UI components and polling for zip downloads data-portal#906)
  • 2 new fields in gitops.json: discoveryconfig.features.exportToWorkspace.[enableDownloadZip (bool) downloadZipButtonText (string)] (Add UI components and polling for zip downloads data-portal#906)
  • Requires PXP-8364 Fix/portal csp cloud-automation#1687, otherwsie Indexing page may not work properly (PXP-8364 fix: strict csp data-portal#893)
  • Portal config field discoveryConfig.features.exportToWorkspaceBETA has been renamed to discoveryConfig.features.exportToWorkspace (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • Note that there are several other new configurations introduced for the Discovery page, please refer to this diff (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • New GEN3_BUNDLE option heal introduced for buliding a HEAL bundle (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • To enable DD RUM, must create a portal-datadog-config k8s secret with datadog_application_id and datadog_client_token in it, see DEVOPS-62 feat: add datadog to portal cloud-automation#1651 (DEVOPS-62 Feat/datadog data-portal#881)
  • envs with customized color schemes no longer need to overwrite dictionary button color in their gitops.css ((PXP-6570): refactor/dictionaryBtnColor: change dictionary btn to use defined global colors data-portal#822)
  • New optional config property, tierAccessLevel, in each tab of the explorerConfig block in the portal config. Used to enable index-scoped tiered-access, which is only available starting with Guppy server version 0.11.0. See README for more information. (Feat/index scoped tiered access data-portal#802)
  • To deploy Discovery page: ((PXP-7402) Discovery page data-portal#803)
  • "featureFlags": { "discovery": true } must be set in the portal config. ((PXP-7402) Discovery page data-portal#803)
  • If discoveryConfig.features.authorization.enabled: true, "useArboristUI": true must be set in the portal config. ((PXP-7402) Discovery page data-portal#803)
  • "discoveryConfig": <DiscoveryConfig> must be set in the portal config. DiscoveryConfig documentation can be found at https://github.com/uc-cdis/data-portal/blob/master/docs/portal_config.md ((PXP-7402) Discovery page data-portal#803)
  • To upload data to Discovery page, data must be uploaded to environment's metadata-service instance with guid_type: "discovery_metadata". The metadata must all be under the top-level property gen3_discovery. (No documentation on this process yet, see this gist for right now https://gist.github.com/Avantol13/a53593837503fede2881f315198247c4) ((PXP-7402) Discovery page data-portal#803)
  • Added new entry GWASApp for analysisTools for this new GWAS app for VA. Beaware not to confuse with the old "GWAS PoC" app, which bears the entry vaGWAS (PXP-7296 PXP-7180 Feat/va gwas app data-portal#777)
  • antd modal now by default will use g3-color__base-blue as header background color, this is consistant with our color schemes for other UI components. If an env has customized color scheme, please override .ant-modal-header in gitops.css (PXP-7296 PXP-7180 Feat/va gwas app data-portal#777)
  • The new feature of Portal requires Guppy v0.10.0 ((PXP-6540): Feat/explorer download formats data-portal#780)
  • Added new button types in the Portal config. See portal_config.md for an example. ((PXP-6540): Feat/explorer download formats data-portal#780)
  • At time of writing, qa-midrc is only env using the new feature flag. These changes should be deployed alongside each other to production, but nothing will break if done separately. ((PXP-7097) Add optional alternate TopBar profile menu data-portal#759)
  • New value for GEN3_BUNDLE env var GEN3_BUNDLE=nct (PXP-6895 NIAID CT bundle data-portal#742)
  • New optional portal config variable injectDAPTag (default value is false). If set to true, will inject DAP tag script into page header (this is required for NCT PROD env only) (PXP-6895 NIAID CT bundle data-portal#742)
  • All envs should check if the dataExplorerConfig field in its portal config has ONLY arrangerConfig field but NO guppyConfig field (it is okay to have both). If so, please update dataExplorerConfig to use Guppy (PXP-6409 Remove arranger data-portal#743)
  • A new Export to PFB button can be optionally enabled in the config.json. It is disabled by default. The button requires at least pelican:2020.10 release. (Feat/export pfb to workspace data-portal#681)
  • Add GEN3_BUNDLE=covid19 as portal env var to enable covid bundle (Feat/covid bundle data-portal#737)
  • Limited File PFB Export requires Pelican >= 0.5.0, Tube >= 0.4.1 ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Limited File PFB Export is enabled by setting fileExplorerConfig.enableLimitedFilePFBExport: { sourceNodeField: "source_node" } and by adding buttons of buttonType export-files (Export to Terra), export-files-to-pfb (Download PFB), or export-files-to-seven-bridges (Export to Seven Bridges) to fileExplorerConfig.buttons. Example: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "fileExplorerConfig": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ... ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "buttons": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • .... ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "enabled": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "type": "export-files-to-pfb", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "title": "Export All to PFB", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "rightIcon": "external-link", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "tooltipText": "You have not selected any cases to export. Please use the checkboxes on the left to select specific cases you would like to export." ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "enabled": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "type": "export-files", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "title": "Export All to Terra", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "rightIcon": "external-link", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "tooltipText": "You have not selected any cases to export. Please use the checkboxes on the left to select specific cases you would like to export." ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "enableLimitedFilePFBExport": {"sourceNodeField": "source_node"}, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Limited File PFB Export requires the source_node property to be ETL'd -- source_node should be added to props in the file section of etlMapping.yaml. Example: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: mpingram_file ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • doc_type: file ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • type: collector ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • root: None ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • category: data_file ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • props: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: object_id ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: md5sum ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ... ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: source_node ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Limited File PFB Export requires a new export-files job block to be added to the Sower config in manifest.json. The $ROOT_NODE environment variable must be set to "file", or the name of the Guppy file index if it isn't "file". For BioDataCatalyst, the EXTRA_NODES environment variable must be set to "". (This is for backwards compatibility: pelican-export includes reference_file by default on all BDCat PFB exports unless $EXTRA_NODES is set to an empty string). Example sower config: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "pelican-export-files", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "action": "export-files", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "container": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "job-task", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "image": "quay.io/cdis/pelican-export:0.5.0", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "pull_policy": "Always", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "env": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "DICTIONARY_URL", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "valueFrom": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "configMapKeyRef": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "manifest-global", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "key": "dictionary_url" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "GEN3_HOSTNAME", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "valueFrom": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "configMapKeyRef": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "manifest-global", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "key": "hostname" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "ROOT_NODE", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "value": "file" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "EXTRA_NODES", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "value": "" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "volumeMounts": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "pelican-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "readOnly": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "mountPath": "/pelican-creds.json", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "subPath": "config.json" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "peregrine-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "readOnly": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "mountPath": "/peregrine-creds.json", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "subPath": "creds.json" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "cpu-limit": "1", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "memory-limit": "4Gi" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "volumes": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "pelican-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secret": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secretName": "pelicanservice-g3auto" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "peregrine-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secret": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secretName": "peregrine-creds" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "restart_policy": "Never" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)

validate.midrc.org/manifest.json

Deployment changes

• arborist
  • Needs Fence later than 4.22.2 or 2020.08 (which introduces the new JWT scopes claim). ((PXP-6617): remove scopes from jwt aud claim arborist#130)
• fence
  • Requires these cloud-automation updates to revproxy: (PXP-7565): Exclude Fence register-user blueprint from CSRF check cloud-automation#1580 ((PXP-7565): User registration, plus CSRF changes fence#906)
  • By default Fence will allow only visas issued by RAS staging, RAS prod, and itself; if a different set of allowed visa issuers is needed, edit the GA4GH_VISA_ISSUER_ALLOWLIST config value. Otherwise, no action needed. ((PXP-6339): RAS visa validation fence#933)
  • /link/google will DENY requests with "redirect" param values that aren't in the "allowed redirects" for OAuth2/OIDC Clients in Fence (You have to add any valid redirects to the client before deploying this) ((PXP-8286): fix(google): add redirect validation to /link/google fence#927)
  • For existing deployments with "ENABLE_AUDIT_LOGS" turned ON, to DISABLE pushing audit logs to an AWS SQS, the configuration "PUSH_AUDIT_LOGS_CONFIG.type" must be changed to "api" (default: "aws_sqs"). If audit-service has never been deployed, no action required. (PXP-7805 Push audit logs to an AWS SQS fence#923)
  • To ENABLE pushing audit logs to an AWS SQS (REQUIRES audit-service 1.0.0 or more recent), create/update the configuration file's "PUSH_AUDIT_LOGS_CONFIG" field ("type: aws_sqs" (default), and the SQS URL and region can be copied from the audit-service configuration - or run gen3 sqs info $(gen3 api safe-name audit-sqs)), see https://github.com/uc-cdis/fence/blob/fec3dd937d68384e86b2ed6f9b53daf0026c2df9/fence/config-default.yaml#L632-L636. Then run kubectl delete secret fence-config and gen3 kube-setup-fence (PXP-7805 Push audit logs to an AWS SQS fence#923)
  • This commit includes patches to allow old style refresh tokens and API keys to pass new style validation, while issuing new style refresh tokens and API keys. Since refresh tokens and API keys have a default TTL of 30 days, stay on this tag for 30 days if possible. (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Old access tokens will NOT work once this is deployed so active sessions may be temporarily interrupted. (PXP-6617 Remove scopes from aud claim in tokens fence#839)
• indexd
  • Include example secrets to load PostgreSQL settings for indexd (Add files for psql ssl support indexd#317)
  • When running gen3authz, backoff package needed separate poetry run pip install (PXP-6824: return 401 errors from Arborist indexd#304)
• metadata
  • A job has been created to make aggregate MDS population more robust feat(agg-mds-job): add new agg mds job cloud-automation#1722 (feat(populate): improved populate script, docs metadata-service#33)
  • Must set the appropriate AGG_MDS_NAMESPACE value in metadata.env (Feat/mds adapter metadata-service#29)
  • Per chore(metadata): migrate agg MDS configs #3332 a metadata/aggregate_config.json file should be created as part of the manifest config for a commons and the appropriate environment vars for metadata should be defined in the manifest JSON (chore(configs): remove configs from source control metadata-service#30)
  • The new command to populate an aggregate MDS is kubectl exec $(gen3 pod metadata) -- python /src/src/mds/populate.py --config /aggregate_config.json --hostname esproxy-service --port 9200 (chore(configs): remove configs from source control metadata-service#30)
  • Network rules are updated to allow for elasticsearch to work with metadata-service feat(ecosystems): elasticsearch for MDS cloud-automation#1638 (Feat/aggregation metadata-service aggregate APIs metadata-service#27)
  • USE_AGG_MDS must be set to true to enable aggregate metadata APIs (Feat/aggregation metadata-service aggregate APIs metadata-service#27)
• portal
  • To enable this check, set features.exportToWorkspace.verifyExternalLogins flag to true in gitops (Check all WTS OIDCs are signed in before download/export to WS data-portal#921)
  • Misc: add option for bundle name: ecosystem, which is the same as heal bundle under the hood (Chore/discovery for more bundles data-portal#918)
  • Requires loginForDownload flag set to true in gitops (PXP-8448 - Register users for download data-portal#912)
  • User registration before download feature depends on (PXP-7565): User registration, plus CSRF changes fence#906 (PXP-8448 - Register users for download data-portal#912)
  • Requires the configuration of the batch-export sower job: Batch export sower job sower-jobs#29 in the environment's manifest. This job also has its own setup in cloud automation: Feat/batch export cloud-automation#1708 (Add UI components and polling for zip downloads data-portal#906)
  • 2 new fields in gitops.json: discoveryconfig.features.exportToWorkspace.[enableDownloadZip (bool) downloadZipButtonText (string)] (Add UI components and polling for zip downloads data-portal#906)
  • Requires PXP-8364 Fix/portal csp cloud-automation#1687, otherwsie Indexing page may not work properly (PXP-8364 fix: strict csp data-portal#893)
  • Portal config field discoveryConfig.features.exportToWorkspaceBETA has been renamed to discoveryConfig.features.exportToWorkspace (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • Note that there are several other new configurations introduced for the Discovery page, please refer to this diff (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • New GEN3_BUNDLE option heal introduced for buliding a HEAL bundle (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • To enable DD RUM, must create a portal-datadog-config k8s secret with datadog_application_id and datadog_client_token in it, see DEVOPS-62 feat: add datadog to portal cloud-automation#1651 (DEVOPS-62 Feat/datadog data-portal#881)