An example of using Common Lisp (sbcl) as a custom runtime on AWS lambda
This is an example for using SBCL as a custom runtime on AWS lambda.
- You can write AWS lambda function with Common Lisp.
- Normal lisp files, compiled fasls, and Roswell scripts are ready to use.
- You can use any Lisp libraries (if it built correctly.)
- Docker :: For building binaries on AWS lambda environment (aka Amazon Linux).
- Proper AMI roles :: for deploying and execiting on AWS lambda.
- AWS CLI :: for uploading AWS Lambda function.
Contents of this repository
||Lisp implementation of AWS Lambda custom runtime.|
||Some utilities for building Lisp AWS Lambda function.|
||Libraries not used by
||Scripts for building the runtime to a zip file.|
||Some examples of AWS Lambda functions in Lisp.|
How to use
- Build a custom runtime and publish it as a AWS Lambda layer.
- Write your code as a AWS Lambda function.
Build and publish a custom runtime.
About codes in
(This section explains my cunstom runtime implementation. If you just want to make a AWS Lambda function, please skip!)
/aws-lambda-runtime/ contains codes for using Common Lisp as a AWS custom runtime.
This code uses these libraries:
- uiop :: for
- drakma :: for main HTTP connection.
This code consists of these files:
|aws-lambda-runtime.asd||the ASDF defsystem.|
|package.lisp||a package definition for this code, named
|lambda-env-vars.lisp||variables holding environmental variables of AWS Lambda enviroment.|
|find-handler.lisp||codes for parsing AWS Lambda's 'handler' parameter.|
|bootstrap.lisp||bootstraps and the main loop.|
bootstrap.lisp does almost all tasks as a AWS Lambda's custom runtime.
bootstrapfunction does initialization tasks -- retrieve settings, initializes, and starts
main-loopfunction does processing tasks -- get an event, calls your handler with it, and write its result back.
You can load this code in the following code:
(load "aws-lambda-runtime/aws-lambda-runtime.asd") (asdf:load-system :aws-lambda-runtime)
((At developing, I load this code into my personal machine. It does not work because there are no environmental variables or HTTP endpoints provided by AWS Lambda's environment, but it is suitable for test a small subsystem, like find-handler.lisp))
Build a custom runtime
Let's build a new custom runtime on Amazon Linux environment.
All building process is written in
In this section, I'll explain what the script does.
About the Dockerfile.
build-bootstrap/Dockerfile is a definition of buiding environment.
This Dockerfile does following:
- It starts with the amazonlinux.
- It gets SBCL from the official repository, and installs it. (I've tried
yumof amazonlinux, but it does not have sbcl.)
- It gets Quicklisp and install it.
- It copies runtime sources (
aws-lambda-runtime-additional-libraries), and install them with some libraries fetched via Quicklisp.
Build a Docker VM.
To build a Docker VM named
cl-aws-buildenv, do following:
docker build -t cl-aws-buildenv .
(this is a part of
Build a AWS custom runtime.
Here, what you to do is:
- Start the Docker VM.
- In the VM, build your runtime to a single binary named bootstrap.
- Makes a zip file contains the bootstrap file.
docker run --rm \ -v `pwd`/build-bootstrap-out:/out \ cl-aws-buildenv /out/build_bootstrap_in_vm.sh
(this is a part of
This code starts the VM and calls
This script does following:
- Load required libraries and our custom runtime.
- Make a single binary with
sb-ext:save-lisp-and-diefeature. I named it to bootstrap and restarts with
aws-bootstrap-test:bootstrap, which is our bootstrap function.
aws_lambda_bootstrap.zip will be made.
(Additionaly, this script makes a text file lists built-in libraries. This is only for provide some information to writers of AWS Lambda function.)
(TODO: I will clean build_bootstrap_in_vm.sh by splitting Lisp codes to another file.)
Publish it is as a AWS Lambda's custom function layer.
Upload the zip as a AWS Lambda's layer.
aws lambda publish-layer-version \ --layer-name lisp-layer \ --zip-file fileb://aws_lambda_bootstrap.zip
(this is a part of
This command makes a new AWS custom runtime layer, named lisp-layer.
Makes a AWS Lambda function using Lisp.
Using the lisp-layer, you can run any Lisp codes in AWS Lambda. In this section, I explain this runtime's calling conventions and show some examples.
How to find the entry point.
The entry point of your AWS lambda functions is specified by 'handler' parameter. This custom runtime reads this parameter as following:
standard syntax :
file is loaded by
method is read as a symbol and
funcall'ed for every request.
The symbol is
funcall'ed with two arguments, request data and HTTP headers, and its return value will be returned as AWS Lambda function's return value.
A script file name.
In this case, 'handler' parameter is considered as a file name. The specified file is loaded, and its main function is called with no arguments.
When main is called,
*standard-input* is bound to the request,
aws-lambda-runtime:*HEADER-ALIST* is bound to HTTP headers, and strings written to
*standard-output* are used as AWS-Lambda's result.
For more information, please see
How to get AWS-Lambda contexts.
AWS-Lambda contexts come from two parts.
- Environmental variables.
They are bound to special varibles in
Please see lambda-env-vars.lisp.
They are bound to
aws-lambda-runtime:*HEADER-ALIST*as an alist (because I uses
In addition, when AWS Lambda's standard syntax is used at 'handler', it is passed at the second argument of your handler function.
HTTP header example:
((:CONTENT-TYPE . "application/json") (:LAMBDA-RUNTIME-AWS-REQUEST-ID . "092aafa5-c7cd-46ea-b305-d2e4d3b77b37") (:LAMBDA-RUNTIME-DEADLINE-MS . "1550171924243") (:LAMBDA-RUNTIME-INVOKED-FUNCTION-ARN . "arn:aws:lambda:************:***********:function:ros_script_handler") ; masked... (:LAMBDA-RUNTIME-TRACE-ID . "Root=1-5c65bf11-b8f197fac0ac10e8bfc0b55a;Parent=019bf6fb19d9378c;Sampled=0") (:DATE . "Thu, 14 Feb 2019 19:18:41 GMT") (:CONTENT-LENGTH . "51") (:CONNECTION . "close"))
Example 1 : A simple one.
A simple example is in handler/01_simple_handler/.
simple_handler.lisp file contains
simple-handler function. It will be called.
upload_function.sh upload it with two steps:
- Make a zip file containing the simple_handler.lisp file.
- Upload it as a AWS Lambda function.
At uploading, I specify
--layers parameter. to use the lisp-layer runtime.
And I used
--handler parameter like this:
It is AWS Lambda's standard syntax. This says "Load
simple_handler, and call
cl-user::simple-handler for each request."
(Thanks to the AWS-Lambda console, you can edit your code in it!)
Example 2 : Using scripts.
An example using roswell scripts is in handler/02_01_roswell_script_text/.
I wrote these scripts:
- empty.ros does nothing. (It is only for testing.)
- hello.ros prints "Hello, World" message.
- echo.ros returns the request as-is.
upload_function.sh works like simple-handler's one, except zips these .ros files, and specifies
--handler like this:
It is script-file syntax. This says "Load
hello.ros, and call its
main function for each request."
I run it with the console's test:
And I change 'handler' to
echo.ros and run it again.
echo.ros copies the request to the response, this screenshot looks so.
Example 3 : Ships with other libraries.
Where to place new libraries?
I decided to build this custom runtime with some JSON libs. But there are obviously short of real-world programmings. So, I consider how to ship my code with other libraries. I think some ways:
Build a new runtime with libraries.
Re-building a new runtime with wanted libraries.
It makes handler codes simpler and faster at starting up. But you must manage many AWS custom function layers.
Ships AWS-lambda function codes with a built FASL.
Making a FASL with wanted libraries, and ships a lisp file with the FASL.
In this approach, you can share same runtimes in many AWS Lambda functions.
But this makes startup slower, and the handler file must
load the FASL correctly.
I wrote an example for this case.
Makes a single FASL contains all.
This is a variant of second approach. Making a FASL with wanted libraries AND handler definitions.
I've not tried this, but I think it is good.
A lisp file and one fasl contains libraries.
An example using a fasl is in handler/03_01_load_another_fasl/.
upload_function.sh does three works:
- Makes a monolithic fasl from needed-libs-example.asd, using build_fasl_in_vm.sh in cl-aws-buildenv VM and . (In this phase, you must build fasl with this VM. Without this, FASL error will be raised.)
- Make a zip file containing the fasl and main.lisp file.
- Upload it as a AWS Lambda function.
--handler like this:
Because of this, this runtime does below at start:
- Loads main.lisp file.
- At loading it, the build fasl will be loaded.
test-parse-handlerfor handling requests.
I changed test data for using integers:
And run it:
It seems I successed to use another library.
One big fasl.
I wrote another example, which makes a big fasl with libraries AND handler codes. There are in handler/03_02_one_big_fasl//.
I sometimes saw
EPERM on poll(2) (Screenshot is here).
I don't now what causes this, but I noticed that retrying to get requests works well. This code is written for this, but I want to know how to fix it correctly.
EPERM is caused by IPv6 socket, referring to Erlang's case. In his article (Japanese), he tried to drop IPv6 support and made a working custom runtime with Clozure CL. I wrote codes for this case with SBCL (unused/usocket-patch-drop-ipv6.lisp), but failed to work. I thought that if IPv6 effects,
EPERM should be reported by socket(2), not poll(2).)
Use a keep-alive socket at getting requests.
For speeding up, use a keep-alive socket at getting requests.
At processing requests by a handler, makes a new thread for it.
This reduces latency for getting the next request. If making a thread is too heavy, a kind of producer-consumer pattern can also be considered.
Add a way to change text encoding.
Add a environmental variable to change
Add other calling conventions for handlers.
Currently I get the whole body of request as a string and pass it to
the handler. But, Drakma has a way to get a stream containing the
body of requests. so I thought passing the stream to handlers is one
(If I implement this feature, I'll bind the stream to
*standard-input*, like the convention for scripts.)
Additionally, I think I want to follow @windymelt's lambda-over-lambda convention, it uses
(I'll use environmental variables for selecting conventions;
Add ql-bundle example.
Add cl-launch script support.
Official tutorial on bash. This repo's code mainly derived from it.
An example of Nim language. In Japanese.
Uses Common Lisp on AWS, on pipes of node.js, by @snmsts.
A custom runtime example using ECL. This code uses a shell script for bootstrap, invokes a build Lisp binary.
This code convert a roswell script to another script works as a AWS Lambda custom function.
A new custom runtime implementation.
- Trying to fix weird
EPERMerror. (Drop IPv6 support, restarting poll(2), etc.)
- Picks all env variables.
- invocation error and initialiation error handling (https://docs.aws.amazon.com/lambda/latest/dg/runtimes-api.html)
- Support new handler type -- AWS Lambda's standard format, roswell script, and Lisp forms.
- Include JSON libs into runtime at build time.
- New examples: monolithic fasl examples.
- New examples: roswell script.
First version is here.. This is only experimental -- does not have any error-handling codes. However, quite simple.
About this version, please see This article (japanese)
The MIT License. See LICENSE file.