BDCAT PREPROD Release 2021.07 preprod.gen3.biodatacatalyst.nhlbi.nih.gov 1631031159

[themarcelor]

Applying version 2021.07 to preprod.gen3.biodatacatalyst.nhlbi.nih.gov

[PlanXCyborg]

preprod.gen3.biodatacatalyst.nhlbi.nih.gov/manifest.json

Deployment changes

• awshelper
  • The new version of the audit service will require running gen3 kube-setup-audit-service and gen3 kube-setup-fence again, and updating the audit-service and fence configuration files (PXP-7805 Audit Service SQS cloud-automation#1603)
• fence
  • For existing deployments with "ENABLE_AUDIT_LOGS" turned ON, to DISABLE pushing audit logs to an AWS SQS, the configuration "PUSH_AUDIT_LOGS_CONFIG.type" must be changed to "api" (default: "aws_sqs"). If audit-service has never been deployed, no action required. (PXP-7805 Push audit logs to an AWS SQS fence#923)
  • To ENABLE pushing audit logs to an AWS SQS (REQUIRES audit-service 1.0.0 or more recent), create/update the configuration file's "PUSH_AUDIT_LOGS_CONFIG" field ("type: aws_sqs" (default), and the SQS URL and region can be copied from the audit-service configuration - or run gen3 sqs info $(gen3 api safe-name audit-sqs)), see https://github.com/uc-cdis/fence/blob/fec3dd937d68384e86b2ed6f9b53daf0026c2df9/fence/config-default.yaml#L632-L636. Then run kubectl delete secret fence-config and gen3 kube-setup-fence (PXP-7805 Push audit logs to an AWS SQS fence#923)
• portal
  • envs with customized color schemes no longer need to overwrite dictionary button color in their gitops.css ((PXP-6570): refactor/dictionaryBtnColor: change dictionary btn to use defined global colors data-portal#822)
  • New optional config property, tierAccessLevel, in each tab of the explorerConfig block in the portal config. Used to enable index-scoped tiered-access, which is only available starting with Guppy server version 0.11.0. See README for more information. (Feat/index scoped tiered access data-portal#802)
  • To deploy Discovery page: ((PXP-7402) Discovery page data-portal#803)
  • "featureFlags": { "discovery": true } must be set in the portal config. ((PXP-7402) Discovery page data-portal#803)
  • If discoveryConfig.features.authorization.enabled: true, "useArboristUI": true must be set in the portal config. ((PXP-7402) Discovery page data-portal#803)
  • "discoveryConfig": <DiscoveryConfig> must be set in the portal config. DiscoveryConfig documentation can be found at https://github.com/uc-cdis/data-portal/blob/master/docs/portal_config.md ((PXP-7402) Discovery page data-portal#803)
  • To upload data to Discovery page, data must be uploaded to environment's metadata-service instance with guid_type: "discovery_metadata". The metadata must all be under the top-level property gen3_discovery. (No documentation on this process yet, see this gist for right now https://gist.github.com/Avantol13/a53593837503fede2881f315198247c4) ((PXP-7402) Discovery page data-portal#803)
  • Added new entry GWASApp for analysisTools for this new GWAS app for VA. Beaware not to confuse with the old "GWAS PoC" app, which bears the entry vaGWAS (PXP-7296 PXP-7180 Feat/va gwas app data-portal#777)
  • antd modal now by default will use g3-color__base-blue as header background color, this is consistant with our color schemes for other UI components. If an env has customized color scheme, please override .ant-modal-header in gitops.css (PXP-7296 PXP-7180 Feat/va gwas app data-portal#777)
  • The new feature of Portal requires Guppy v0.10.0 ((PXP-6540): Feat/explorer download formats data-portal#780)
  • Added new button types in the Portal config. See portal_config.md for an example. ((PXP-6540): Feat/explorer download formats data-portal#780)
  • At time of writing, qa-midrc is only env using the new feature flag. These changes should be deployed alongside each other to production, but nothing will break if done separately. ((PXP-7097) Add optional alternate TopBar profile menu data-portal#759)
  • New value for GEN3_BUNDLE env var GEN3_BUNDLE=nct (PXP-6895 NIAID CT bundle data-portal#742)
  • New optional portal config variable injectDAPTag (default value is false). If set to true, will inject DAP tag script into page header (this is required for NCT PROD env only) (PXP-6895 NIAID CT bundle data-portal#742)
  • All envs should check if the dataExplorerConfig field in its portal config has ONLY arrangerConfig field but NO guppyConfig field (it is okay to have both). If so, please update dataExplorerConfig to use Guppy (PXP-6409 Remove arranger data-portal#743)
  • A new Export to PFB button can be optionally enabled in the config.json. It is disabled by default. The button requires at least pelican:2020.10 release. (Feat/export pfb to workspace data-portal#681)
  • Add GEN3_BUNDLE=covid19 as portal env var to enable covid bundle (Feat/covid bundle data-portal#737)
  • Limited File PFB Export requires Pelican >= 0.5.0, Tube >= 0.4.1 ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Limited File PFB Export is enabled by setting fileExplorerConfig.enableLimitedFilePFBExport: { sourceNodeField: "source_node" } and by adding buttons of buttonType export-files (Export to Terra), export-files-to-pfb (Download PFB), or export-files-to-seven-bridges (Export to Seven Bridges) to fileExplorerConfig.buttons. Example: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "fileExplorerConfig": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ... ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "buttons": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • .... ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "enabled": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "type": "export-files-to-pfb", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "title": "Export All to PFB", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "rightIcon": "external-link", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "tooltipText": "You have not selected any cases to export. Please use the checkboxes on the left to select specific cases you would like to export." ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "enabled": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "type": "export-files", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "title": "Export All to Terra", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "rightIcon": "external-link", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "tooltipText": "You have not selected any cases to export. Please use the checkboxes on the left to select specific cases you would like to export." ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "enableLimitedFilePFBExport": {"sourceNodeField": "source_node"}, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Limited File PFB Export requires the source_node property to be ETL'd -- source_node should be added to props in the file section of etlMapping.yaml. Example: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: mpingram_file ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • doc_type: file ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • type: collector ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • root: None ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • category: data_file ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • props: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: object_id ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: md5sum ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ... ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: source_node ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Limited File PFB Export requires a new export-files job block to be added to the Sower config in manifest.json. The $ROOT_NODE environment variable must be set to "file", or the name of the Guppy file index if it isn't "file". For BioDataCatalyst, the EXTRA_NODES environment variable must be set to "". (This is for backwards compatibility: pelican-export includes reference_file by default on all BDCat PFB exports unless $EXTRA_NODES is set to an empty string). Example sower config: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "pelican-export-files", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "action": "export-files", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "container": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "job-task", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "image": "quay.io/cdis/pelican-export:0.5.0", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "pull_policy": "Always", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "env": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "DICTIONARY_URL", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "valueFrom": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "configMapKeyRef": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "manifest-global", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "key": "dictionary_url" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "GEN3_HOSTNAME", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "valueFrom": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "configMapKeyRef": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "manifest-global", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "key": "hostname" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "ROOT_NODE", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "value": "file" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "EXTRA_NODES", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "value": "" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "volumeMounts": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "pelican-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "readOnly": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "mountPath": "/pelican-creds.json", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "subPath": "config.json" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "peregrine-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "readOnly": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "mountPath": "/peregrine-creds.json", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "subPath": "creds.json" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "cpu-limit": "1", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "memory-limit": "4Gi" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "volumes": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "pelican-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secret": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secretName": "pelicanservice-g3auto" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "peregrine-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secret": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secretName": "peregrine-creds" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "restart_policy": "Never" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Portal should be updated to include this fix if Indexd is above 2.12.0 or 2020.07 and user want to use Arborist UI (fix: data upload auth check data-portal#702)
  • New optional config field in the index section of gitops.json called homepageChartNodeChunkSize. Sets the value for paritioning requests to Peregrine for the homepage chart. Defaults to 15 if not provided. (Feat/peregrine request chunk data-portal#699)