Run Jenkinsfiles in AWS Lambda
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
images Initial implementation Jan 27, 2019
src Initial implementation Jan 27, 2019
.dockerignore Initial implementation Jan 27, 2019
.gitignore Initial implementation Jan 27, 2019
Dockerfile Lock jenkinsfile-runner docker image version Feb 11, 2019
LICENSE Initial commit Jan 15, 2019
Makefile Initial implementation Jan 27, 2019
README.md Point to the example repo Jan 27, 2019
plugins.txt Initial implementation Jan 27, 2019
pom.xml Use jenkinsfile-runner 1.0-beta-6 Feb 11, 2019

README.md

Jenkinsfile Runner for AWS Lambda

An AWS Lambda function to run Jenkins pipelines. It will process a GitHub webhook, git clone the repository and execute the Jenkinsfile in that git repository. It allows huge scalability with 1000+ concurrent builds and pay per use with zero cost if not used.

This function allows Jenkinsfile execution without needing a persistent Jenkins master running in the same way as Jenkins X Serverless, but using AWS Lambda instead of Kubernetes. All the logs are stored in AWS CloudWatch and are easily accessible.

Why???

Why not?

I mean, it could make sense to run Jenkinsfiles in Lambda when you are building AWS related stuff, like creating an artifact and uploading it to S3.

Limitations

Lambda limitations:

  • 15 minutes execution time
  • 3008MB of memory
  • git clone and generated artifacts must fit in the 500MB provided

Current implementation limitations:

  • checkout scm does not work, change it to sh 'git clone https://github.com/carlossg/jenkinsfile-runner-lambda-example.git'
  • Jenkinsfile must add /usr/local/bin to PATH and use /tmp for any tool that needs writing files, see the example

Example

See the [jenkinsfile-runner-lambda-example] project for an example that is tested and works.(https://github.com/carlossg/jenkinsfile-runner-lambda-example)

Extending

Three lambda layers are created:

  • jenkinsfile-runner: the main library
  • plugins: minimal set of plugins to build a Jenkinsfile
  • tools: git, openjdk, maven

You can add your plugins in a new layer as a zip file inside a plugins dir to be expanded in /opt/plugins. You could also add the Configuration as Code plugin and configure the Artifact Manager S3 to store all your artifacts in S3.

Other tools can be added as new layers, and they will be expanded in /opt. You can find a list of scripts for inspiration in the lambci project (gcc,go,java,php,python,ruby,rust) and bash, git and zip (git is already included in the tools layer here)

The layers are built with Docker, installing jenkinsfile-runner, tools and plugins under /opt which is where Lambda layers are expanded. These files are then zipped for upload to Lambda.

Installation

Create a lambda function jenkinsfile-runner using Java 8 runtime. Use the layers built in target/layer-* and target/jenkinsfile-runner-lambda-*.jar as function. Could use make publish to create them.

Set

  • handler: org.csanchez.jenkins.lambda.Handler::handleRequest
  • memory: 1024MB
  • timeout: 15 minutes
aws lambda create-function \
    --function-name jenkinsfile-runner \
    --handler org.csanchez.jenkins.lambda.Handler::handleRequest \
    --zip-file fileb://target/jenkinsfile-runner-lambda-1.0-SNAPSHOT.jar \
    --runtime java8 \
    --region us-east-1 \
    --timeout 900 \
    --memory-size 1024 \
    --layers output/layers.json

Exposing the Lambda Function

From the lambda function configuration page add a API Gateway trigger. Select Create a new API and choose the security level. Save the function and you will get a http API endpoint.

Note that to achieve asynchronous execution (GitHub webhooks execution will time out if your webhook takes too long) you would need to configure API Gateway to send the payload to SNS and then lambda to listen to SNS events. See an example.

GitHub events

Add a GitHub json webhook to your git repo pointing to the lambda api gateway url.

Building

make

Publishing

Once the lambda function jenkinsfile-runner is created

make publish

Development

docker build -t csanchez/jenkinsfile-runner-lambda .
docker run -ti --rm -v `pwd`/demo:/workspace -w /workspace csanchez/jenkinsfile-runner-lambda

Testing

curl -v -H "Content-Type: application/json" --data '@src/test/resources/github.json' https://AAAABBBBCCCC.execute-api.us-east-1.amazonaws.com/default/jenkinsfile-runner

Getting the logs

awslogs get /aws/lambda/jenkinsfile-runner ALL --watch