DIG-1663: Implement curator ingest in katsu and batch ingest

[mshadbolt]

Incorporates changes in katsu and ingest to allow program curators to ingest into katsu

• Fixes integration tests to work with new changes
• adds program authorization deletion to clean_up methods
• add to non admin integration test
• edit add_program_authorization method so that program curators and team members can be added

Also includes upgrading pip to version 24 because the github action build started failing and this seemed to fix it

[mshadbolt]

We shouldn't merge this until CanDIG/katsu#238 is approved and merged, will need to update the katsu submodule too

[SonQBChau]

Please remove the commits to lib/candigv2-ingest and lib/katsu as they are already merged. This will help us avoid any conflicts.

[SonQBChau]

I also have a couple of comments that are out of scope for this PR:

1. Some test functions, such as test_ingest_not_admin_katsu and test_ingest_admin_katsu, are becoming quite large.
2. The naming of roles is confusing. For example, "curators" and "team members" default to the site_admin value. If we need a convenience value, it should be the least privileges, not full privileges. It's easier to catch bugs this way.

[mshadbolt]

I also have a couple of comments that are out of scope for this PR:

I don't think they are out of scope, if we want to change them we can incorporate it into this PR

1. Some test functions, such as test_ingest_not_admin_katsu and test_ingest_admin_katsu, are becoming quite large.

Do you have a suggestion for how to split them up?

2. The naming of roles is confusing. For example, "curators" and "team members" default to the site_admin value. If we need a convenience value, it should be the least privileges, not full privileges. It's easier to catch bugs this way.

I don't really understand what you mean by this, can you give an example?

[SonQBChau]

Yes, it's out of scope for now, but we can create a ticket to address it during the redesign.

Line 120, it set admin to default user

def add_program_authorization(dataset: str, curators: list = [ENV['CANDIG_SITE_ADMIN_USER']], 
                              team_members: list = [ENV['CANDIG_SITE_ADMIN_USER']]):