Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[4564] Restructure Security section #4612

Open
wants to merge 69 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
9703d7b
[4564] Add conceptual info to security introduction
Oct 5, 2021
6c727c3
move note about "Enable Security" checkbox back to "Managing Security"
Oct 5, 2021
8163c74
delete bogus link to a PR
Oct 5, 2021
a604ac6
Merge branch 'master' into 1004-security-overview
Oct 6, 2021
708997b
Moved info about who affects Pipeline from controller-isolation.adoc
Oct 7, 2021
e68d0ad
Merge branch '1004-security-overview' of github.com:StackScribe/jenki…
Oct 7, 2021
dcad943
Merge branch 'master' into 1004-security-overview
Oct 7, 2021
d568df4
Merge branch 'master' into 1004-security-overview
Oct 8, 2021
721f907
minor edits
Oct 13, 2021
0ffb510
Pipelines -> jobs
Oct 14, 2021
7fb49ce
Add concepts-security page
Oct 19, 2021
bdf825b
moving files in _chapter.yml
Oct 20, 2021
033a397
Add rewritten controller-isolation from PR4635
Oct 20, 2021
db5a03e
Move TCP port info into its own page
Oct 20, 2021
6639fec
Removed a bit of crud
Oct 20, 2021
5e0b503
Rename securing-jenkins to be a placeholder for those wiki links
Oct 20, 2021
5f4e5d4
Remove "Agents and Security" info from index.ado
Oct 20, 2021
557c68d
Move Access Control info from managing-security to access-control
Oct 20, 2021
0eeeb0b
Add "story" (outline of chapter) to index.adoc
Oct 20, 2021
70098e6
Consolidate access control info in one page
Oct 23, 2021
da01b25
fix link to security advisories
Oct 26, 2021
6f81a40
fix link to security advisories
Oct 26, 2021
b8d7c1e
fix link to managing-security
Oct 26, 2021
28e443a
remove solopsistic link
Oct 26, 2021
e7f5165
delete extraneous title
Oct 26, 2021
f7d4100
an user -> an user
Oct 26, 2021
b1519f3
Draft "Configure Global Security" page
Oct 26, 2021
6109f6a
Refine definitions of nodes and executors
Oct 26, 2021
52b9f64
remove details about nodes
Dec 9, 2021
b750b91
remove details about agents/exectors on controller
Dec 9, 2021
228fb0f
remove some details about number of executors
Dec 9, 2021
1a08475
one sentence per line
Jan 10, 2022
c6eff48
one sentence per line
Jan 10, 2022
23c078f
one sentence per line
Jan 10, 2022
27429db
one sentence per line
Jan 10, 2022
9f431b8
one sentence per line
Jan 10, 2022
184cd91
one sentence per line
Jan 10, 2022
cd21cad
one sentence per line
Jan 10, 2022
35b4ed8
Update content/doc/book/security/index.adoc
Jan 10, 2022
87c2ab8
one sentence per line
Jan 10, 2022
808d2aa
one sentence per line
Jan 10, 2022
b061d8d
one sentence per line
Jan 10, 2022
7f61450
one sentence per line
Jan 10, 2022
0bc82cd
Pare down description of distributed components
Jan 11, 2022
f6096fd
Clarify that only 0 executors on controller guarantees no builds
Jan 11, 2022
384fe9d
Merge branch 'master' into 1004-security-overview
MarkEWaite Jan 13, 2022
ca895cd
Resolving some dbeck comments
Jan 20, 2022
ba6730b
Remove admonition to not configure TCP port if you don't need it
Jan 21, 2022
9849ee6
rephrase material about jobs being able to do anything
Jan 21, 2022
27afddd
one sentence per line -- part 1
Jan 21, 2022
9ad1afa
Merge branch '1004-security-overview' of github.com:StackScribe/jenki…
Jan 21, 2022
c545ed7
restore TODO about Agent -> Controller version
Jan 21, 2022
dd7768a
details about agent->controller in that page; disabling is history
Jan 21, 2022
bc3212c
Advice to revisit security configuration moved from index to concepts
Jan 21, 2022
aec1a09
one sentence per line
Jan 21, 2022
2c46b5a
delete securing-jenkins.adoc -- dbeck says these wiki links are obsolete
Jan 21, 2022
d512ff7
InBound -> Inbound
Jan 21, 2022
1609c80
Merge branch '1004-security-overview' of github.com:StackScribe/jenki…
Jan 21, 2022
2549d42
fix indentation
Jan 21, 2022
fb96a9e
Merge branch '1004-security-overview' of github.com:StackScribe/jenki…
Jan 21, 2022
8b9af00
"Jenkins Access Control" -> "Access Control"
Jan 21, 2022
ed0e597
How Jenkins executes Pipeline -> job
Jan 23, 2022
89adbc6
Deleted securing-jenkins from _chapter.yml
Jan 23, 2022
d5b6325
new file for "How Jenkins Executes a Job"; rename "Security Concepts"
Jan 26, 2022
1d349e8
Remove links to pages for fields on Configure Security page from inde…
Jan 26, 2022
4b4ced3
Move Agent -> Controller Security into controller-isolation.adoc
Jan 26, 2022
e92c6f7
fix xref
Jan 26, 2022
b886850
Merge branch 'master' into 1004-security-overview
MarkEWaite Aug 5, 2022
be0f0aa
Merge branch 'master' into 1004-security-overview
MarkEWaite Aug 12, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
25 changes: 13 additions & 12 deletions content/doc/book/security/_chapter.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,20 @@
sections:
# index - implied overview

- access-control

# Legacy sections
- securing-jenkins
- managing-security
# General information
- concepts
- how-jobs-execute
- controller-isolation
- managing-security

# Individual how-to pages
- controller-isolation
- csrf-protection
- user-content
- build-authorization
- environment-variables
- markup-formatter
- access-control
- tcp-port
- markup-formatter
- csrf-protection
- user-content
- build-authorization
- environment-variables

# Further references
- services
- services
225 changes: 197 additions & 28 deletions content/doc/book/security/access-control.adoc

Large diffs are not rendered by default.

35 changes: 35 additions & 0 deletions content/doc/book/security/concepts.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
title: Security Concepts
layout: section
---

Security concepts should guide the practices and tools used to fight and prevent threats.
The major principles for security are:

* *Least privilege:*
Give people the privileges required to do their jobs but do not give everyone permission to do everything and do not open ports that are not required for your work.

* *Know the system:*
The more you understand about how your system works, the more you are prepared to protect the integrity of your system.
See link:/doc/book/security/how-jobs-execute/[How Jenkins Executes Jobs] for a summary of the Jenkins execution sequence.

* *Defense in depth:*
Systems are layered.
Put security on all layers.

* *Prevention is good, but detection is better:*
Monitor your Jenkins installation constantly so that you quickly detect signs of a security breach.

Good practices that help keep your instance secure include:

* *Keep your system current:*
Pay attention to Jenkins link:/security/advisories/[Security Advisories] and apply link:/security/for-administrators/#how-quickly-should-i-apply-security-updates[Security Updates] as soon as possible.
Keeping the Jenkins software and all plugins current also helps ensure that your system is secure.
Also be sure to keep all other software up to date, including the underlying operating system software, build tools, and test tools that you use.

* *Revisit your security configuration periodically:*
Most Jenkins environments grow over time, requiring their trust models to evolve as the environment grows.
Schedule regular "check-ups" of your security settings to ensure that they are still appropriate for your instance.
In particular, if you had to disable some of the default protections that Jenkins provides,
perhaps because you were using a plugin that had not been updated to support that protection, you may be able to re-enable that protection to increase the security of your instance.

137 changes: 98 additions & 39 deletions content/doc/book/security/controller-isolation.adoc
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: Controller Isolation
title: Distributed Builds
layout: section
---
ifdef::backend-html5[]
Expand All @@ -9,67 +9,126 @@ ifndef::env-github[:imagesdir: ../../resources]
:hide-uri-scheme:
endif::[]

What exactly happens during a build is often controlled by people less trusted than a Jenkins administrator:
Never run builds on the Jenkins controller in production environments.
Out of the box, Jenkins is set up to run builds on the built-in controller node.
This makes it easier to get started with Jenkins, but, on production instances, builds should only be executed on nodes that are external to the controller.

* Jenkins users with Job/Configure permission
* Build script authors (`pom.xml`, `Makefile`, etc.)
* Code authors (for example test code executed during a build)
A build job that uses an agent running on the built-in controller node may be able to access Jenkins configuration files and the workspaces of other builds.
An agent could also request information that belongs to other teams or organizations that share the controller.
Configuring Jenkins as a link:/doc/book/scaling/architecting-for-scale/#distributed-builds-architecture[distributed build system] enhances the security of your Jenkins instance as well as improving its performance and making it more stable.
This is because builds that run on the built-in node have the same level of access to the controller file system as the Jenkins process itself.
Configuring your Jenkins instance so that builds execute on agents running on separate nodes protects the Jenkins controller from malicious (or just broken) build scripts.

They all have some control over commands executed during a build.
This does not even consider issues like supply chain attacks on build dependencies, whereby attackers take over control of NPM or Maven packages and insert malicious code.
When distributed builds are implemented, the Jenkins controller serves HTTP requests and stores all important information related to the builds.
Nodes are separate physical or virtual machines that are configured. The Jenkins controller manages these nodes, but the agents manage the task execution on behalf of the Jenkins controller and supply most of the computing power required to build and test the software.
Different nodes can run different operating systems and different tools, enabling one Pipeline to build and test the same software for a variety of platforms.

To ensure the stability of the Jenkins controller, builds should be executed on other nodes than the built-in node.
This concept is called _distributed builds_ in Jenkins, and you can learn more about this https://wiki.jenkins.io/display/JENKINS/Distributed+builds[here].
Setting up distributed builds in Jenkins is a great start for protecting the Jenkins controller from malicious (or just broken) build scripts, but care needs to be taken for protections to be effective.

[CAUTION]
[NOTE]
====
Most Jenkins environments grow over time requiring their trust models to evolve as the environment grows. Please consider scheduling regular "check-ups" to review whether any disabled security settings should be re-enabled.
A job that performs administrative tasks such as backups may run on the controller.
To reduce the chance of someone using this executor to run a job, label the executor with a hard-to-guess label and only allow it to be used by jobs that specify that label.

An executor can be temporarily configured on the controller long enough to run the administrative task but no executors should be configured on the controller at other times
This ensures that no builds execute on the controller since a clever Pipeline author may be able to work around the label restriction.

You can install the plugin:job-restrictions[Job Restrictions Plugin] to ensure that builds do not run on the controller even if executors are configured.
====

An agent can also write malicious code to its local disk so that the node is tainted and can infect a later build.
For maximum security, run all builds on ephemeral agents in the cloud so that they are destroyed at the end of each build job.

== Not building on the built-in node
By extension, you also should not run agents on the same Docker host as the controller.

Out of the box, Jenkins is set up to run builds on the built-in node.
This is to make it easier to get started with Jenkins, but is inadvisable longer term:
Any builds running on the built-in node have the same level of access to the controller file system as the Jenkins process.
== Distributed Components

It is therefore highly advisable to not run any builds on the built-in node, instead using agents (statically configured or provided by _clouds_) to run builds.
A Jenkins distributed builds instance includes the following components.
For more details about these components, see link:/doc/book/managing/nodes/[Managing Nodes].

// TODO Fix this once https://github.com/jenkinsci/jenkins/pull/5425 is merged and released:
Jenkins controller::

To prevent builds from running on the built-in node directly, navigate to _Manage Jenkins » Manage Nodes and Clouds_.
The Jenkins controller is a webserver where Jenkins is installed.
It schedules tasks, executes management tasks (configuration, authorization, and authentication).
Files written when a Pipeline executes are written to the filesystem on the controller unless they are off-loaded to an artifact repository such as Nexus or Artifactory.

To prevent builds from running drectly on the built-in node, navigate to _Manage Jenkins » Manage Nodes and Clouds_.
Select _Built-In Node_ in the list, then select _Configure_ in the menu.
Set the number of executors to 0 and save.
Make sure to also set up clouds or build agents to run builds on, otherwise builds won't be able to start.
Configure clouds or build agents where builds can execute.

Alternatively, use a plugin such as plugin:job-restrictions[Job Restrictions Plugin] to limit which jobs can be run on certain nodes (like the built-in node), independent of what your less trusted users may use as label expression in their jobs' configurations.
Nodes::

[NOTE]
====
If you do not have any other computers to run agents on, you can also run an agent process as a different operating system user on the same system to achieve a similar isolation effect.
In this case, ensure that the agent process has no file system access (neither read nor write) to the Jenkins home directory, and that the agent process cannot use `sudo` or similar methods to elevate its own permissions.
====
Nodes are the "machines" on which build agents run.
The Jenkins controller itself runs on a special _built-in node_.

Agents::

Agents are small Java client processes that use executors to manage the task execution on behalf of the Jenkins controller.
In practice, nodes and agents are essentially the same but it is good to remember that they are conceptually distinct.

Executors::

An executor is a slot for execution of tasks; effectively, it is a thread in the agent.
The number of executors on a node defines the number of concurrent tasks that can be executed on that node at one time.

== Implementing Distributed Builds

To implement distributed builds on your Jenkins instance, you must do the following:

* Create nodes and agents where the builds can run.
* Prevent builds from running on the controller.

//== Infrastructure
// TODO Don't run agents on the same Docker host as the controller etc.
This section also includes other tips and tricks you can use.

== Agent → Controller Access Control
=== Create Nodes and Agents for Builds

The Jenkins controller and agents can be thought of as a distributed process which executes across multiple discrete processes and machines.
This allows an agent to ask the controller process for information available to it, for example, the contents of files, etc., and even to have the controller run certain commands when requested by the agent.
To implement a distributed builds architecture for your Jenkins instance, you must configure nodes and agents to use for your builds.
See link:/doc/book/managing/nodes/[Managing Nodes] for instructions.

So while not building on the built-in node is a good general practice to protect from bugs and less sophisticated attackers, an agent process taken over by a malicious user would still be able to obtain data or execute commands on the Jenkins controller.
To prevent this, the Agent → Controller Access Control system prevents agent processes from being able to send malicious commands to the Jenkins controller.
=== Prevent Builds from Running on the Controller

// TODO Also mention first LTS once it's known
StackScribe marked this conversation as resolved.
Show resolved Hide resolved
This system is always enabled since Jenkins 2.326 (see link:jep-235[Agent → Controller Security Changes in 2.326]).
In Jenkins 2.325 and earlier, it is enabled by default, but can be disabled in the web UI by un-checking the box on the _Manage Jenkins » Configure Global Security_ page.
You must also prevent builds from running on the built-in controller node.
To do this:

IMPORTANT: It is strongly recommended that you not disable the Agent → Controller Access Control system.
. Navigate to _Manage Jenkins » Manage Nodes and Clouds_.
. Select _master_ in the list, then select _Configure_ in the menu.
. Set the number of executors to 0.
. Save the configuration.

NOTE: A job that performs administrative tasks such as backup may run on the controller, but be sure to label the executor with a hard-to-guess label and only allow it to be used by jobs that use that label.
Jenkins should normally be configured to have 0 executors on the controller, and then be temporarily reconfigured to have 1 executor long enough to run the backup or other administrative task.
Unless the number of executors is 0, it is possible for a build to run on the controller.

The link:/doc/book/security/agent-controller-access/[Agent → Controller Access Control system] provides additional protections for the controller.

Another way to prevent builds from running on the controller is to use a plugin such as the plugin:job-restrictions[Job Restrictions Plugin] to limit which jobs can be run on certain nodes (like the built-in controller node), independent of what your less trusted users may use as label expression in their job configurations.

== Agent → Controller Security

The Jenkins controller and agents can be thought of as a distributed process that executes across multiple discrete processes and machines.
Not building on the built-in controller node is an essential practice to protect the controller from bugs and less sophisticated attackers.
However, an agent can stil ask the controller process for information such as the contents of files, and can even request that the controller run certain commands.
This means that an agent process taken over by a malicious user could still obtain data or execute commands on the Jenkins controller.
To prevent this, Jenkins implements the **Agent → Controller Access Control** filter that prevents agent processes from sending malicious commands to the Jenkins controller.

This filter may prevent your builds from executing legitimate operations.

* If builds are failing because of the agent-to-controller access control, you should first ensure that you have upgraded to the latest version of all plugins, since older versions of plugins may not work correctly with this feature.

* You can also tweak the filter's rules by adding `allow/deny` rules to specify the commands that you need and the type of access that is allowed for the agent.
See link:/doc/book/security/controller-isolation/agent-to-controller/[Customizing Agent -> Controller Security] for more information.

[NOTE]
====
// TODO Add mention of first LTS once it's known
In releases prior to Jenkins 2.326 (see link:jep-235[Agent → Controller Security Changes in 2.326]), this protection could be
disabled in the web UI by un-checking the box on the _Manage Jenkins » Configure Global Security_ page, although this was strongly discouraged.

image::security/configure-global-security-agent-controller-toggle.png["Configure Global Security - Enable Agent => Controller Access Control", role=center]

As an alternative to disabling Agent → Controller Access Control, in Jenkins 2.325 and earlier, administrators can selectively allow greater access.
See link:/doc/book/security/controller-isolation/agent-to-controller/[the documentation] for details.
====

== Other Tips and Tricks

If you do not have any other computers on which to run agents, you can run an agent process as a different operating system user on the same system to achieve a similar isolation effect.
In this case, be sure that the agent process has neither read nor write file system access to the Jenkins home directory, and that the agent process cannot use `sudo` or other practices to elevate its own permissions.
41 changes: 41 additions & 0 deletions content/doc/book/security/how-jobs-execute.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
title: How Jenkins Executes Jobs
layout: section
---

A simple overview of how Jenkins executes a Freestyle or Pipeline job helps to understand the security considerations.
The execution sequence is similar for Freestyle and Pipeline jobs, with the following differences:

* Pipelines are controlled by a _Jenkinsfile_ that is written in a scripting language whereas Freestyle jobs are controlled from a configuration defined in the UI.
The scripting language supports features such as loops and allows more flexibility about when and where a build script executes;
these features also provide additional opportunities to do mischief.
* The _Jenkinsfile_ is stored under SCM, which can initiate its execution, so securing the SCM is also an important part of protecting the integrity of your Jenkins installation.

By default, a job executes with the full privileges of the Jenkins administrator, although you can configure Jenkins to execute jobs with fewer privileges.

All of the job's logic as well as a Pipeline's Groovy conditionals, loops, and so forth execute on the controller.

When the job runs:

* Jenkins creates a _workspace_ on the controller for each build that runs.
Files for that build are stored in the workspace.
* The job calls a series of _steps_, each of which is a script or command that does the real work and mostly executes using an _executor_ on an _agent_.

The agent:

* Writes some files to the local node.
* Sends data back to the controller.
* May also request information from the controller.

Many different people have some control over the commands that are executed during a build:

* Jenkins users with Job/Configure permission
* Authors of build scripts such as `pom.xml` and `Makefile`
* Authors of code, such as test suites that are executed during a build

Any of these could introduce security issues, either deliberately or accidentally.
In addition, supply chain attacks can occur on build dependencies, whereby attackers take over control of NPM or Maven packages and insert malicious code.

You can see the complexity of keeping your Jenkins instance secure.
In the following sections we discuss specific protections that Jenkins provides and practices that you can implement to protect Jenkins from intrusions.