[ Feature: document module ] document endpoints

[jerome-white]

Summary

Target issue is #32

Endpoints for manipulating documents meant for RAG interaction.

Checklist

Before submitting a pull request, please ensure that you mark these task.

• Ran poetry run uvicorn src.app.main:app --reload in the repository root and test.
• If you've fixed a bug or added code that is tested and has test cases.

Notes

This is designed to be a "working" PR: it will remain in draft state as it is fleshed out to provide transparency and get ongoing discussion around its practices.

Update on moving from draft

This has become a large PR. It was started before a lot of our development processes were stablized. Notably:

1. Before we confirmed that small progressively merged features were the preference. Rather than single document module endpoints, this PR contains about half of the overall document module specification.
2. Before there was clear delineation between branch purposes and how those branches would work together. The branch against which this code is developed is most certainly out-of-date.

To address the second point, I ask that the reviewers look at this code in isolation, not necessarily how the changes fit with what is currently on main. "Approving" and merging this branch will start the following process:

1. Code from this PR will go back into my out-of-date sub-branch
2. I will work to bring the sub-branch inline with main. This not only means a manual merge, but updates to the code to use current best-practices (see Document Management: Integrate organization/project into endpoints #85 for example). Each non-merge update will proceed as its own PR
3. From there I will proceed with the open sub-issues in Module for managing documents in RAG #26

[jerome-white]

There are several questions I have so far (relevant to 4cbb94f):

AWS

How are we managing AWS credentials across development and production:

1. Are we assuming there will be a single AWS user for all organizations/projects?
2. Is there a set of AWS credentials we are using for development?
3. Where are we storing AWS credentials? The .env seems like the best place, but there is nothing there right now (even though we seem to be testing deployment in AWS)

S3 management

The current version of this PR assumes a bucket per project/organization. Alternatives include:

• a bucket per organization or project, with "directories" for project or organization (respectively)
  s3://org/project_{1,2,3}
• a single bucket for all users, differentiated by "directories" for organization and project
  s3://ai-platform/org/project/

Using buckets per project/org (s3://org-proj/) means we will have several buckets, but in my mind will be much easier to manage; you can clear a bucket, for example, without worry about effecting other users. This becomes particularly easier if the ai-platform has its own AWS sub-account within T4D.

Project and organization information

How are we passing the project and organization to routes? Right now I assume this information is carried in the "current user"; although in my branch the current user has no awareness of a org/project breakdown. I believe #63 will bring this functionality in, but it's not clear whether that will hide the details behind the user facade (like I'm hoping it will)

[sourabhlodha]

@jerome-white I can share you AWS credentials you can add in .env. Let's use same bucket and different directive for all organisation

[jerome-white]

There are several questions...

Based on various discord discussions and meetings:

• For now there will be a single AWS user that all clients will share
• We will use a single bucket for all clients
• Project and organization are now present/available in the main branch; how to replace "user" with them is to me still unknown

[jerome-white]

I can no longer wait for a second review. Merging this