FaaSMark is a benchmark for serverless compute platforms. It is designed to measure function invocation latency across different clouds and using different invocation methods and function parameters.
The benchmark currently tests latency under different:
- Invocation methods (HTTP vs SDK)
- Programming languages
- Memory sizes
- Load conditions
- FaaS providers
This version supports aspect (invocation memory, programming languages and memory size) benchmarks and concurrency benchmarks only on AWS Lambda.
Benchmark code is divided into three parts:
- Client is the code that drives the benchmark process. It can run on your local machine or any cloud VM/container.
- Initiator is a FaaS function that is deployed on the platform being benchmarked. The initiator is responsible for repeatedly invoking empty functions (see below) and measuring their invocation latencies. There are two types of initiators:
simpleInitiatorandconcurrencyInitiator. The first is used for measuring different aspects of function invocation (method, languages and memory sizes) and comparing different cloud platforms. The second is used for load tests. - Empty is a FaaS function that retuns immediately after invocation. This function is deployed on the FaaS platform being benchmarked. Empty is implemented in JavaScript across all platforms and in multiple languages on AWS Lamda. Concurrency tests use a more complex version of empty called
sleeperthat can sleep for a specified interval before returning. This feature is used for warming up multiple function containers before starting the benchmark itself.
Deploying both initiator and empty on the same FaaS platforms assures that no factors outside of the platform affect invocation latency.
In order to perform the benchmark you first need to deploy functions to the different FaaS provider platforms. See how to below.
Invoke the benchmark using the command
node faasmark.js
Benchmark behavior is control by the file settings.json which has the following fields:
| Name | Values | Description |
|---|---|---|
forever |
false|true |
Test once or loop forever |
interval |
Number | Milliseconds between beginning of tests |
serviceName |
faas-mark |
Change only if you change deployment code |
repeat |
Number | How many times Empty is invoked by Initiator |
concurrencyRepeat |
Number | Same as repeat for concurrency (load) tests |
maxConcurrency |
Number | Maximum concurrent invocations (load level) |
maxConcurrencyPerInitiator |
Number | Maximum concurrent invocations from a single initiator function |
providers.aws.region |
String | AWS region name |
providers.gcloud.region |
String | Google Cloud region name |
providers.gcloud.project |
String | Google Cloud project name |
Note that automatic deployment onto Azure and Bluemix is not supported yet, and that their settings are currently hard coded.
Deployment method varies between clouds.
Use the Serverless Framework (sls) to deploy functions to AWS Lambda. Set your AWS credentials and install install sls.
Now run:
cd providers/aws
sls deploy
Automatic deployment is not yet supported on Azure. To deploy on Azure Functions you can use the web console to create two functions with the contents of
providers/azure/empty.js
providers/azure/simpleInitiator.js
Use the file name (without the extension) as the function name and configure HTTP triggers for the functions.
See Azure above.
Once the gcloud utility is installed and configured, you can use the
providers/gcloud.js
utility to create a project, configure it to run functions and deploy functions to that project.
You can use an existing project or create a new one. Once you have a project name you must set the GCLOUD_REGION and GCLOUD_PROJECT variables accordingly.