Switch branches/tags
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
README.adoc

README.adoc

JEP-310: Evergreen AWS auto-configuration

Abstract

One of the four pillars of Jenkins Evergreen is that it should be autoconfigured with sane defaults for various cloud platforms. This specification explains what setup is done to autoconfigure Jenkins Essentials to run in the Amazon Web Service environment.

Specification

Global overview

The setup uses an AWS CloudFormation template that instantiates the right resources.

It creates an EC2 instance, to host the Jenkins Evergreen master in a dedicated Instance Profile. The master will be able to provision agents from EC2, and store artifacts directly in S3.

Used plugins and configuration

As a general rule for Evergreen, we are using the configuration-as-code plugin to do the necessary configuration.

No credentials are specified. The permissions are retrieved dynamically through the instance profile used when creating the EC2 instance for the Evergreen master.

Instance Profile and IAM role

A dedicated instance profile is created by the CloudFormation template to allow creating EC2 instances for agents, and access a specific S3 bucket to store artifacts and Pipeline stashes.

EC2 policy

"Statement": [{
    "Sid": "ControlEC2JenkinsAgents",
    "Effect": "Allow",
    "Action": [
        "ec2:DescribeInstances",
        "ec2:TerminateInstances",
        "ec2:RequestSpotInstances",
        "ec2:DeleteTags",
        "ec2:CreateTags",
        "ec2:DescribeRegions",
        "ec2:RunInstances",
        "ec2:DescribeSpotInstanceRequests",
        "ec2:StopInstances",
        "ec2:DescribeSecurityGroups",
        "ec2:GetConsoleOutput",
        "ec2:DescribeImages",
        "ec2:CancelSpotInstanceRequests",
        "ec2:StartInstances",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeSubnets",
        "ec2:DescribeKeyPairs"
    ],
    "Resource": "*"
}]

S3 bucket policy

"Statement": [{
        "Sid": "TodoRefineSecurity-JENKINS-52342",
        "Effect": "Allow",
        "Action": [
            "s3:PutObject",
            "s3:GetObject",
            "s3:ListBucket",
            "s3:DeleteObject"
        ],
        "Resource": "*"
    }
]
🔥
Currently the prototype is configured with the way shown above. We would like to reduce the access to only the dedicated bucket created for Jenkins Evergreen usage. We are tracking this improvement as JENKINS-52342.

Security groups

Master

A dedicated group is created to allow access to the following ports:

22

restricted to the SSHLocation parameter value, passed as CloudFormation template parameter at creation time. The user will be strongly advised to use their own public IP to restrict SSH access to the master to her only (i.e. discourage 0.0.0.0/0). [2]

8080

not restricted.

50000

not restricted (for potentially connecting agents with JNLP).

ℹ️
The port 50000 is not currently enabled inside the Evergreen instance. But we keep it allowed at the security group and Docker container level. This will allow users to connect for instance Windows agents through the Jenkins JNLP protocol by simply enabling the fixed agents port under /configureSecurity/.

Agents

A dedicated group is created to allow only access to port 22.

Motivation

Nothing was existing to provide an autoconfigured setup of Jenkins in a specific Cloud environment.

Reasoning

Instance Profile

Early during the prototype, a dedicated service user was used. This was replaced by instance profile later for various reasons.

This is because it is the recommended path for AWS, but also because doing so makes the configuration leaner: both the EC2 and Artifact Manager on S3 plugins will autodetect their permissions if no explicit credentials are specified.

s3:DeleteObject permission on the S3 bucket

Keeping this permission has been questioned; why allow deletion permission when:

  • pure AWS administrators may prefer to define a lifecycle policy to clean up things;

  • S3 is so cheap that we, at least, should not care about deletions?

We decided to keep it because:

  • Evergreen is critically about simplicity. If a user, or us a bit later, wants to enable to delete checkbox of the artifact-manager-s3 plugin, this should be possible without having to go through the AWS console or CLI to find the right settings;

  • Keeping everything forever is also probably some kind of liability. And if some users have previously been administering a Jenkins instance previously, but they are not AWS experts, their expectation would probably be that artifacts are going to be cleaned up over time, according to the buildDiscarder policy in place.

Backwards Compatibility

There are no backwards compatibility concerns related to this proposal.

Security

Allowed certificates

JEP 307 aggressively restricts the list of certificates that will be allowed to be used from inside Evergreen container.

Two more need to be re-enabled for the Evergreen AWS flavor to be able to auto-detect permissions given by the current instance profile:

  • Baltimore_CyberTrust_Root.crt: for S3.

  • Amazon_Root_CA_1.crt: for EC2.

SSH access to the server

A CloudFormation template parameter SSHLocation is used to define the IPs allowed to access the master using SSH.

Users are advised to pass only their own IP when creating their setup. But SSH is deemed sufficiently secure so that we still allow users to pass 0.0.0.0/0 as a value if they want this.

HTTPS by default

(?) TBD FIXME

Agent port

WIP: FIXME: do we even need to open it. Does the agent actually connect through the SSH pipe?

Infrastructure Requirements

AWS account for testing

If we want to be able to run automated tests, we need some form of AWS account to actually check the CloudFormation works and keep working. See below AWS testing of CloudFormation template.

Testing

AWS testing of CloudFormation template

Given an AWS account for testing, we can use the aws CLI to automatically trigger the provisioning, then retrieve what is neeeded to run automated tests:

  • create the stack using aws cloudformation create-stack …​

  • retrieve the EC2 instance IP using:

    • aws cloudformation list-stack-resources --stack-name <just-created-stack>,

    • and aws ec2 describe-instances --instance-ids i-<retrieved-ID> | grep -i publicIp

  • run some tests, e.g.:

    • is the <masterIp>/login URL reachable, etc..

    • use ssh ec2-user <masterIp> docker exec jenkins-evergreen <some-command> to do additional automated checks from the running instance itself.

Prototype Implementation

The prototype implementation is available at https://github.com/jenkins-infra/evergreen.

More specifically, the AWS part is available under the distribution/environments/aws-ec2-cloud directory.


1. even more important for Jenkins Evergreen which is configured with the sane 0-executor for the master node
2. using curl ident.me is often to help find out one' public IP and pass the value to the CloudFormation template parameter.