AWS Lambda URI consumer.
Use this generic Lambda service to consume APIs and persist response data in S3. Once the data is in S3, it can be consumed by any number of AWS cloud services.
Experimental
Required
- AWS Lambda as executor of pac-man tasks.
- AWS S3 as Destination for data.
Optional
- GitHub Actions CI/CD to deploy pac-man.
- AWS Secrets Manager for sensitive request or persistence parameters.
- AWS EventBridge to trigger pac-man runs.
AWS Lambda deployments can be triggered in a variety of ways. Configuring events to trigger your Lambda functions involves parameters specific to each use-case.
Uncomfortable with sensitive values in plaintext?
Good, you should be!
AWS Secrets Manager is supported, using a modifier, with the following value expression: {:secrets:<name>:<key>} (see Modifiers).
Caution
Lambda functions are naturally ephemeral or stateless. By design, pac-man expects to be executed as a single-use serverless function. Some modifiers maintain a cache in the event they are utilized more than once for a given request. Using pac-man in stateful applications can have unexpected consequences.
Minima Lambda Event
{
"source": {
"scheme": "https",
"hostname": "example.com"
},
"destination": {
"region": "us-east-1",
"collection": "bucket-name",
"name": "key"
}
}Maxima Lambda Event
{
"mods": [
{
"name": "chunks",
"start": 0,
"end": 100,
"chunk": {
"length": 10
},
"bytes": 100
},
{
"name": "secrets",
"region": "us-east-1"
},
{
"name": "uuid"
}
],
"source": {
"headers": {
"Accepts": [
"text/json"
]
},
"scheme": "https",
"username": "pseudo",
"password": "{:secrets:pac-man:pw}",
"hostname": "example.com",
"port": 8080,
"path": "/follow/the",
"params": {
"from": "{:chunks:chunk:start}",
"to": "{:chunks:chunk:end}"
},
"fragment": "/yellow/brick/road"
},
"destination": {
"region": "us-east-1",
"collection": "bucket-name",
"name": "key-{:uuid}"
}
}Optional Event Fields
delta( minima.fields, maxima.fields )
Modifiers or mods implement functionality that modifies placeholders in pac-man event fields.
By default, mods aren't active and must be toggled respectively within the mods configuration body of the Lambda Event.
{:chunks:chunk:start}
{:chunks:chunk:end}
{:chunks:chunk:index}
{:chunks:chunk:page}
{
"name": "chunks",
"start": 0,
"end": 100,
"chunk": {
"length": 10
},
"bytes": 100
}bytes is a special configuration that denotes the minimum Content-Length header required to continue. It can be used to decide when a last page has been reached.
{:secrets:<name>:<key>}
{
"name": "secrets",
"region": "us-east-1"
}{:uuid}
{:uuid:<key>}
{
"name": "uuid"
}Using a key with the uuid modifier allows reference to the same uuid using a HashMap cache for a given run.
Lambda functions can be executed with the help of Docker.
docker-test.sh launches a Lambda build using Docker.
- AWS_SECRET_ACCESS_KEY
- AWS_ACCESS_KEY_ID
- Build Package
../pac-man$ docker run --rm \
-v ${PWD}:/code \
-v ${HOME}/.cargo/registry:/root/.cargo/registry \
-v ${HOME}/.cargo/git:/root/.cargo/git \
softprops/lambda-rust- Unzip Package
../pac-man$ unzip -o \
target/lambda/release/pac-man.zip \
-d /tmp/lambda- Run Unzipped Package
../pac-man$ docker run \
-i -e DOCKER_LAMBDA_USE_STDIN=1 \
-e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
-e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
--rm \
-v /tmp/lambda:/var/task \
lambci/lambda:provided- Build Lambda Package Using Docker
../pac-man$ docker run --rm -ti -v ${PWD}:/code softprops/lambda-rust- Deploy Lambda Using AWS CLI
../pac-man$ aws lambda create-function --function-name pac-man \
--handler doesnt.matter \
--zip-file fileb://target/lambda/release/pac-man.zip \
--runtime provided \
--region <lambda-region> \
--role arn:aws:iam::<lambda-create-role> \
--timeout <lambda-timeout-seconds> \
--environment Variables={RUST_BACKTRACE=1} \
--tracing-config Mode=Active