Introduce prototype for upload API, based on S3

[rohanpm]

This API can be used to upload objects.
Rather than inventing our own API for this, let's use a subset of
the true S3 API. This allows usage of existing clients (such as
boto) to upload content through exodus-gw. The examples/s3-upload
script can be used to demonstrate this.

This implementation is prototype-level and comes with a TODO list.
Some major missing pieces are:

• performance testing to verify that this doesn't perform notably
  worse than using S3 directly
• a proper way to handle configuration (it currently allows the boto
  library to use whatever config is discovered by default, meaning
  only a single environment is supported)

This commit also experiments a bit more on how to document these
APIs. There is now a separate API user guide and reference, as it's
not clear if there's a decent way of including general usage info
into the redoc-hosted reference.

[rohanpm]

@nathanegillett

This isn't ready for proper review yet, I need other PRs to be landed first. But I pushed it so you can have a look in case you're also prototyping APIs.

However, the way things are done here in some cases isn't a great example, because it's trying to make an API compatible with S3 which doesn't follow the preferred conventions of FastAPI.

[rohanpm]

Manually published docs at https://rohanpm.github.io/exodus-gw/ so they can be reviewed.

[dichn]

@rohanpm In order to make exdus-gw acess localstack(for integration test), I did some changes like this.

async def object_put(bucket: str, key: str, request: Request):
     # Single-part upload handler: entire object is written via one PUT.
     body = await request.body()
 
-    async with aioboto3.client("s3") as s3:
+    async with aioboto3.client("s3", endpoint_url='http://localhost:4572') as s3:
         response = await s3.put_object(Bucket=bucket, Key=key, Body=body)
 
     return Response(headers={"ETag": response["ETag"]})

It works with the endpoint_url argument added. If we use the origin version, it's kind difficult to configure the integration test.
should this be scoped in the PR?
others look good to me.

[rohanpm]

Added two commits just now:

• enable streaming of requests, after I figured out how to get that working. Performance now seems to be roughly on par with using S3 directly, though I've seen some unexplained spikes at times.
• added an (undocumented) environment variable EXODUS_GW_S3_ENDPOINT_URL which can be used for the localstack testing case mentioned by @dichen16 .

OK to merge this? I think it'd be good to get something in here that others can start building on, even if the API is later expected to change.

[lmilbaum]

@rohanpm This is a large PR in the sense of lots of code changes. Is it possible to break it to little atomic PRs?

[rohanpm]

@lioramilbaum

Is it possible to break it to little atomic PRs?

In general I'm very much in support of that and I do tend to make small PRs with focused goals, but in a case like this - early development of a new project - I'm struggling to find an efficient workflow for it. Maybe I'm missing some option or tool in github, but I don't see any way to make one pull request depend on another.

If I consider this PR as an example then, a way that I could have split it up is:

1. tweaks to the publishing of docs
2. initial implementation of upload APIs
3. add env var to make endpoint configurable
4. support streaming requests

The problem is the dependencies between them... (2) arguably depends on (1), (3) and (4) depend on (2).
I can make those separate PRs, but if (e.g.) PR4 is including some commits from PR2 and PR1, then the UI is going to show all the changes from PR1 and PR2 again in PR4 with no hint to reviewers that the changes will be added by some other PR and so should be ignored here. Reviewers can be told in a comment "please note that this depends on PR 1 & 2, so ignore the changes introduced by those", but I doubt it's practical for reviewers to do that if the UI is not helping them. PR4 wouldn't appear as little & atomic unless the previous PRs are submitted first.

A simple way to fix this at the cost of time is to wait for review of each PR to be completed before asking for review on the next one... but that's extremely inefficient. From my point of view, every review is at least overnight (or more if it doesn't pass on the first attempt). That means making one bigger PR has a chance of getting merged in a day or two, but splitting it into 5 smaller dependent PRs would need at least a full week to get them all merged even if everything goes perfectly.

I'm interested in solutions to this problem (and not only for this project), so if you have some experience or suggestions on how to manage dependencies between PRs please let me know.

[lmilbaum]

@rohanpm You have raised great discussion points. Let's continue the discussion over an email thread or even better, a meeting. For this PR specifically, I do not have the tools (yet) to provide an officiant code review.