A prototype for an API that allows clients to post an images with captions and clients can create a comment on each post.
- 1 post can only have 1 image
- A post can have multiple comments
- Minimum throughput: 100 RPS
- Cater for client's unstable internet connection
- AWS Lambda
- AWS S3
- AWS DynamoDB
- AWS API Gateway
Minimize the number of task a procedure needs to perform. Reduce to a single task if possible.
Why?
- Reduces latency. e.g. An endpoint only needs to perform one thing, either upload to S3 or a single call to DynamoDB to fetch/insert item.
- Minimizes the "transactional behavior" problem that arises when a procedure needs to communicate with different kinds of storages. Transactional behavior needs to make sure that transactions are either success or failed (and not in invalid state) by rollback or compensating transactions whenever one of the task failed.
How?
- The API endpoint for creating post only handles uploading to S3. Inserting the
post
item in dynamodb is done in the background. - The API endpoint for fetching posts, with latest 2 comments, only communicate 1 time in a request because the latest 2 comments are already in the post item in dynamodb. Populating the latest 2 comments in a post is done in the background.
The design prioritizes low latency, high throughput, and could cater for users with unstable internet connections. However, to achieve these constraints, it introduces some tradeoffs which are listed below:
- The text payload (e.g. caption, user) in
POST /posts
is only limited to 2 KB (around ~2k characters). The limitation is on S3 metadata. - Adjusting the number of comments that comes with the
GET /posts
response will not be trivial because the data is being duplicated in the Post entity to reduce latency. e.g. bumping from latest 2 comments to latest 5 comments - Eventual consistency: newly created post will not show immediately in
GET /posts
endpoint. There's a bit of delay since the DynamoDB item is created in the background. Same with comments.
- Finish up missing user stories.
- Distributed tracing and logging.
- Setup alerts for errors and anomalies.
- Implement graceful handling of failures and retries.
- e.g. image uploaded lambda func encountered an error.
- Implement Authorizer. At the time of writing, user is defined via the
user-id
header which is not very secure. - Enable delete protection for the dynamodb table.
- Cloudfront to further improve latency.
- E2e automation tests
- Run
STAGE=prod make deploy
to deploy in production environment.
# run unit tests
make test
# deploy in AWS
make deploy
See Insomnia-collections.yaml
for the endpoint details.