Lambda powertools

[MatMoore]

Idea Description - Build a simplified version of the data platform ingestion api using lambda power tools as a framework. Maybe we could limit it to the create/update/delete metadata/schema endpoints as a proof of concept.

If we have time, we can also consider the Serverless Application Model (SAM) tooling.

Why Should We Do This - we can make progress on this spike ticket https://github.com/ministryofjustice/data-platform/issues/1786. The hope is that this will reduce the size/complexity of the codebase, which would help us to move faster in beta.

Definition of Done - we have learned enough about Lambda power tools to decide whether to use it or not AND we have mapped out any required architecture changes

What skills do we need? AWS / python

Additional information

https://blog.walmsles.io/making-all-your-apis-idempotent
Powertools docs
Powertools tutorial (a good walkthrough)
Another example lambda with Powertools
https://aws.amazon.com/blogs/compute/using-container-image-support-for-aws-lambda-with-aws-sam/
AWS SAM and Terraform

[MatMoore]

Part of this should involve looking at patterns for ensuring operations are idempotent and leave the metadata store in a consistent state.

At the moment, versioning touches multiple distinct systems

• s3
• glue catalogue
• external metadata catalogue

So it's very easy for a broken operation to leave us in a weird state