-
Notifications
You must be signed in to change notification settings - Fork 11
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Introduce prototype for upload API, based on S3
This API can be used to upload objects. Rather than inventing our own API for this, let's use a subset of the true S3 API. This allows usage of existing clients (such as boto) to upload content through exodus-gw. The examples/s3-upload script can be used to demonstrate this. This implementation is prototype-level and comes with a TODO list. Some major missing pieces are: - performance testing to verify that this doesn't perform notably worse than using S3 directly - a proper way to handle configuration (it currently allows the boto library to use whatever config is discovered by default, meaning only a single environment is supported) This commit also experiments a bit more on how to document these APIs. There is now a separate API user guide and reference, as it's not clear if there's a decent way of including general usage info into the redoc-hosted reference.
- Loading branch information
Showing
16 changed files
with
940 additions
and
14 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,69 @@ | ||
.. _api-guide: | ||
|
||
API User Guide | ||
============== | ||
|
||
This document provides an overview of the APIs available in exodus-gw | ||
along with general usage information. | ||
|
||
For a full reference on the available endpoints, see: | ||
`API Reference <api.html>`_. | ||
|
||
.. warning:: | ||
|
||
exodus-gw is in early stages of development. All APIs are subject | ||
to backwards-incompatible changes without warning. | ||
|
||
|
||
Authentication | ||
-------------- | ||
|
||
Authentication is currently unimplemented in exodus-gw; any clients with | ||
access to the service can perform all operations. | ||
|
||
|
||
Uploading blobs | ||
--------------- | ||
|
||
.. seealso:: `Upload API reference <api.html#tag/upload>`_ | ||
|
||
.. automodule:: exodus_gw.s3.api | ||
|
||
Using boto3 with the upload API | ||
............................... | ||
|
||
As the upload API is partially compatible with S3, it is possible to use | ||
the AWS SDK to perform uploads. This is the recommended method of using the API. | ||
|
||
Use ``endpoint_url`` when creating a boto resource or client to point at exodus-gw. | ||
Region and credentials will be ignored. | ||
|
||
Note that, as the upload API provides only a subset of the S3 API, many boto methods | ||
will not work. Uploading files is supported. | ||
|
||
Usage: | ||
|
||
.. code-block:: python | ||
import boto3 | ||
from botocore.config import Config | ||
# Prepare S3 resource pointing at exodus-gw | ||
s3 = boto3.resource('s3', | ||
endpoint_url='https://exodus-gw.example.com/upload', | ||
# If SSL needs to be configured: | ||
verify='/path/to/bundle.pem', | ||
config=Config(client_cert=('client.crt', 'client.key'))) | ||
# Basic APIs such as upload_file now work as usual | ||
bucket = s3.Bucket('exodus-cdn-dev') | ||
bucket.upload_file('/tmp/hello.txt', | ||
'aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f') | ||
Publishing content | ||
------------------ | ||
|
||
exodus-gw is expected to provide an API for making uploaded content | ||
externally accessible on the exodus CDN. This API is not yet implemented. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
.. _api-reference: | ||
|
||
API Reference | ||
============= | ||
|
||
This page is a placeholder, expected to be overwritten with openapi/api.html | ||
during doc build. | ||
|
||
If you are seeing this page in published docs, something has gone wrong, | ||
please report it! |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,22 @@ | ||
exodus-gw | ||
========= | ||
|
||
Publishing microservice for Red Hat's Content Delivery Network | ||
Publishing microservice for Red Hat's Content Delivery Network. | ||
|
||
.. toctree:: | ||
:maxdepth: 2 | ||
:caption: Contents: | ||
|
||
api-guide | ||
api | ||
|
||
Overview | ||
-------- | ||
|
||
exodus-gw (or "exodus gateway") is a microservice for uploading and publishing content | ||
onto Red Hat's CDN. It exposes an API for use by publishing tools. | ||
|
||
- For general usage information on the exodus-gw API, see :ref:`api-guide`. | ||
- For a technical reference on the exodus-gw API, see :ref:`api-reference`. | ||
- For more general information on the CDN architecture and how exodus-gw fits in, | ||
see `Architecture <https://release-engineering.github.io/exodus-lambda/arch.html>`_. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,60 @@ | ||
#!/usr/bin/env python3 | ||
# Upload a list of files from the command-line to s3, possibly | ||
# using a custom endpoint. | ||
# | ||
# This command is used as a baseline for testing s3 uploads | ||
# via exodus-gw or via s3 directly. It can be used to compare | ||
# performance and compatibility. | ||
# | ||
# Usage: | ||
# | ||
# # Using default S3 | ||
# examples/s3-upload file1 [file2 [...]] | ||
# | ||
# # Testing same thing via exodus-gw | ||
# uvicorn exodus_gw.gateway:app & | ||
# examples/s3-upload --endpoint-url http://localhost:8000/upload file1 [file2 [...]] | ||
# | ||
# It is recommended to test using a mixture of files both smaller | ||
# and larger than 10MB to cover both multipart and single part | ||
# uploads. | ||
|
||
import boto3 | ||
import sys | ||
import hashlib | ||
import argparse | ||
|
||
|
||
def get_object_key(filename): | ||
with open(filename, "rb") as f: | ||
hasher = hashlib.sha256() | ||
while True: | ||
chunk = f.read(1024 * 1024 * 10) | ||
if not chunk: | ||
break | ||
hasher.update(chunk) | ||
return hasher.hexdigest() | ||
|
||
|
||
def main(): | ||
parser = argparse.ArgumentParser() | ||
parser.add_argument("--endpoint-url", default=None) | ||
parser.add_argument("--bucket", default="exodus-cdn-dev") | ||
parser.add_argument("files", nargs="+") | ||
|
||
args = parser.parse_args() | ||
s3 = boto3.resource("s3", endpoint_url=args.endpoint_url) | ||
bucket = s3.Bucket(args.bucket) | ||
|
||
print( | ||
"Using endpoint:", | ||
"[default]" if not args.endpoint_url else args.endpoint_url, | ||
) | ||
|
||
for filename in args.files: | ||
bucket.upload_file(filename, get_object_key(filename)) | ||
print("Uploaded:", filename) | ||
|
||
|
||
if __name__ == "__main__": | ||
main() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
# Any modules which add endpoints onto the app must be imported here | ||
from . import gateway | ||
from . import gateway, s3 | ||
|
||
# Per ASGI conventions, main entry point is accessible as "exodus_gw:application" | ||
from .app import app as application |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
from . import api |
Oops, something went wrong.