Key Features:
- Efficient and compact design with minimal resource usage (CPU, storage), optimized for shortening lambda coldstart and improving performance. The only extra requirement is the
requestslibrary. - Ensures the integrity of input data, maintaining the original content without any alterations. This feature facilitates multiple iterations over the development or business logic processes within the data pipeline.
- Support for processing large files:
- Uploads initiated through a presigned S3 URL to ensure security and efficiency.
- The solution employs an S3 Object Lambda for content filtering, implemented in a streaming approach to handle data efficiently.
- Enables the retrieval of processed content directly from the Object Lambda endpoint, ensuring a streamlined and secure download process.
- Infrastructure as code (IaC) with CDK for Terraform managing the AWS resources (shared with a S3 backend for state management)
The following assumes that cdktf, terraform, pipenv and of course python>=3.12 already installed.
See the following links for further information about how to install those
- https://developer.hashicorp.com/terraform/tutorials/cdktf/cdktf-install
- https://github.com/tfutils/tfenv
- https://github.com/pyenv/pyenv
The smoke test uses jq as well
In addition, CDKTF will require an AWS authenticated shell (either through user credentials or SSO with the AWS IAM identity center).
The boto3 and botocore packages are part of the lambda runtime and not listed as project dependencies.
pipenv run pytestNote: the BASE_URL is already set in the file.
./smoke-test.sh ./test-and-deploy.shI did not implement deployment on GitHub via GitHub actions, but it would follow the same logic as described above.
Will have to use a cdktf GitHub action etc. See: