Feat/passport

[Avantol13]

We are merging chunks of this feature into this working branch and will do a merge into master once all features are complete here.

• Update the release versions if they are not fence==6.0.0 and fence==2022.02/04 and uncomment the blocks that automatically setup the cronjob(s), test it again, and get it reviewed when we know the real monthly and semantic release for this feature: Feat/storage expiration cloud-automation#1750 & feat(fence): add new, required ga4gh expiration cronjob (will update … cloud-automation#1806 & feat(jobs): add visa update job for fence (cronjob already exists) cloud-automation#1828
• Revert Fence, cloud-auto, and gitops-qa timeout increases after performance improvements to authz sync
• Check that chore(migration): add authz column to project #996 and chore(migration): add authz column to project #1003 do not result in duplicated code
• Merge Feat/storage expiration cloud-automation#1750 assuming fence==6.0.0 and fence==2022.02 with commented out setup blocks
• Merge GA4GH Cleanup job feat(fence): add new, required ga4gh expiration cronjob (will update … cloud-automation#1806
• Merge Feat/multi passport validation #967
• Merge Google Expiration support feat(storage_expiration): add expires to db for Google access, create… #968
• Merge (PXP-8682): Add visa validation and sync for data download #973
• Merge (PXP-8992): Expire user policies and update Arborist username #986
• Merge (PXP-8994): Populate mapping table for existing users #995
• Merge Fix/visa expiration #997
• Merge feat(passport-exp): handle expiration edge cases [PXP-9235] #988
• Merge (PXP-9480): Improve sync_single_user_visas performance #1005
• Merge Fix/resource deletion #1012
• Merge feat(jobs): add visa update job for fence (cronjob already exists) cloud-automation#1828

New Features

• Add idp_to_user table that maps multiple IdP's to a single user (Currently only using for RAS) (PXP-8679): Add new db table and update login to add entries #963
• Cronjob for removing expired Google Data Access if expiration is set feat(storage_expiration): add expires to db for Google access, create… #968
• Support multi-passport validation Feat/multi passport validation #967
• Visa validation and authz sync for POST /data/download/{guid} endpoint (PXP-8682): Add visa validation and sync for data download #973
• Set expiration for created user policies when a passport is POSTed to DRS endpoint (PXP-8992): Expire user policies and update Arborist username #986
• Change username in Arborist when a user POSTs to DRS endpoint and then logs in (PXP-8992): Expire user policies and update Arborist username #986
• project id to authz resource mapping is now persisted in the database from user.yaml (PXP-9123): feat(authz): persist authz project to resource mapping in database so it can be accessed w/o user.yaml  #987
• Populate iss_sub_pair_to_user table using User table's id_from_idp column (PXP-8994): Populate mapping table for existing users #995
• usersync no longer updates visas and Access Token Polling now both gets updated passport(s) for a user and updates authorization feat(passport-exp): handle expiration edge cases [PXP-9235] #988
• A frequently-running cronjob handles the removal of expired visas feat(passport-exp): handle expiration edge cases [PXP-9235] #988

Breaking Changes

• URL Signing when no_force_sign query param is provided: Previously Fence would make a decision based off if the data was public and no_force_sign provided. Fence will now NEVER sign if no_force_sign is provided (since the concept of "public" data has been abstracted out of Fence. Data access - public or not - is the sole responsibility of the policy engine). In other words, if no_force_sign is provided at the API level, Fence will respect that regardless of whether the resulting URL will actually work. The default Fence behavior should remain the same. feat(passport-exp): handle expiration edge cases [PXP-9235] #988

Bug Fixes

Improvements

• Syncing to Google storage backend supports supplying an expiration for Google Data Access feat(storage_expiration): add expires to db for Google access, create… #968
• Use validate_jwt from Fence instead of authutils Feat/multi passport validation #967
• Passport sync now uses persisted project id to authz resource mapping from the database if available when syncing authz (PXP-9123): feat(authz): persist authz project to resource mapping in database so it can be accessed w/o user.yaml  #987
• Passport cache for substantially increasing speed of subsequent parsing of the same, valid passport feat(passport): add cache [PXP-8838] #999
• Significantly improve sync_single_user_visas performance by use of a single Arborist policy to represent access (PXP-9480): Improve sync_single_user_visas performance #1005
• fix issue where Fence was not finding the newest Service Account for a user b/c of an existing SA under a previous username (this situation occurs when a user uses a Passport and THEN logs in with their eRA Commons) PXP-9386 Fix/google sa username #1001

Dependency updates

• Update gen3authz to 1.5.1 (PXP-9503): Fix access token polling so that it syncs authz info to Arborist #1007 (PXP-9480): Improve sync_single_user_visas performance #1005 (PXP-8992): Expire user policies and update Arborist username #986

Deployment changes

• Requires database migration for a new table to Fence, google Access expiration to add new expires column, and to create iss_sub_pair_to_user table feat(storage_expiration): add expires to db for Google access, create… #968 (PXP-8679): Add new db table and update login to add entries #963 (PXP-8682): Add visa validation and sync for data download #973
• Requires Arborist >= 2022.04 OR >=3.4.0 (PXP-9480): Support merge param for PUT /resource endpoint arborist#143
• To enable Passports -> DRS in an environment you must update Fence Configuration to at a minimum specify GA4GH_PASSPORTS_TO_DRS_ENABLED: true (only do this after coordinating with the environment owner as this has several requirements before it can be enabled, including the need to use Indexd's authz field in ALL records rather than acl)

[github-actions]

The style in this PR agrees with black. ✔️

This formatting comment was generated automatically by a script in uc-cdis/wool.

[coveralls]

Pull Request Test Coverage Report for Build 12534

• 613 of 734 (83.51%) changed or added relevant lines in 19 files are covered.
• 50 unchanged lines in 7 files lost coverage.
• Overall coverage increased (+0.6%) to 72.711%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
fence/blueprints/ga4gh.py	6	7	85.71%
fence/resources/openid/idp_oauth2.py	0	1	0.0%
fence/models.py	58	61	95.08%
fence/sync/passport_sync/ras_sync.py	15	18	83.33%
fence/resources/storage/init.py	0	4	0.0%
fence/blueprints/login/ras.py	12	18	66.67%
fence/blueprints/data/indexd.py	74	81	91.36%
fence/scripting/fence_create.py	45	53	84.91%
fence/utils.py	1	11	9.09%
fence/resources/openid/ras_oauth2.py	50	62	80.65%

Files with Coverage Reduction	New Missed Lines	%
fence/blueprints/login/ras.py	2	80.39%
fence/resources/admin/admin_projects.py	2	79.41%
fence/scripting/fence_create.py	2	57.58%
fence/sync/passport_sync/ras_sync.py	2	86.49%
fence/job/visa_update_cronjob.py	6	87.91%
fence/resources/userdatamodel/userdatamodel_project.py	9	59.26%
fence/sync/sync_users.py	27	74.13%

Totals
Change from base Build 12501:	0.6%
Covered Lines:	6805
Relevant Lines:	9359

---

💛 - Coveralls