JEP |
200 |
Title |
Switch Remoting/XStream denylist to an allowlist |
Sponsor |
|
Status |
Final 🔒 |
Type |
Standards |
Created |
2017-10-30 |
JIRA |
|
BDFL-Delegate |
Since its early days, Jenkins has used two object serialization frameworks: Java’s built-in API, from the Remoting layer; and XStream, for persisting configuration and settings. Both allow any Java class to be loaded merely by specifying its name in the input.
This proposal switches the denylist to an allowlist of a few dozen entries: for Jenkins to process a third-party class, it must be explicitly approved. Classes defined in Jenkins core or plugins are exempt, as are a few special categories.
The main aspect of the change is the introduction of a ClassFilterImpl in Jenkins core which supplants the simple denylist defined in Remoting.
This filter is applied to all Java or XStream deserialization performed by the Jenkins controller (as well as XStream serialization, to "fail fast").
A crude denylist extension API in Remoting introduced in
2017-04-26 is deprecated
in favor of the more general ClassFilter.setDefault, as is the system property hudson.remoting.ClassFilter.DEFAULTS_OVERRIDE_LOCATION.
The new filter has several levels of rules:
-
A
CustomClassFilterimplementation may override any behavior. This is discussed in the compatibility section. -
Classes mentioned in the original denylist continue to be rejected.
-
Java arrays and enums are accepted, as these are special formats.
-
Java
Throwabletypes are accepted, as they are commonly used in Remoting responses and it is impractical to enumerate every exception that might be thrown. -
A class defined in Jenkins core, Remoting, or a Jenkins module/plugin is accepted. This does not apply to classes defined in other JARs packaged in
WEB-INF/lib/*.jar. -
Some classes defined in test utilities, by mocking frameworks, etc., are accepted when running inside tests.
-
Classes defined in a
RemoteClassLoader(so, already sent to this JVM from a presumably trusted source) are accepted. This is needed at least by CloudBees Jenkins Operations Center, and clients of themaster-to-master-apiplugin if there are any. -
Any other class is accepted if it is explicitly mentioned in a list
whitelisted-classes.txtbundled in Jenkins core. These consist of:-
Basic Java utility types (see the reference implementation).
-
Some other collection types defined in Google Guava.
-
A handful of safe types defined in some libraries used by Jenkins core.
-
A handful of types used by some plugins known to otherwise fail: Git, Pipeline, Maven, Job DSL, etc.
-
The allowlist has been built up by repeatedly running a large array of tests against a patched core and collecting relevant errors. It is expected that the standard allowlist will be expanded somewhat as field reports arrive of less common code paths needing new entries.
When a class is (first) rejected, a warning is printed to the system log. This is particularly useful when scanning large automated test results for regressions caused by this change. Typically the rejection will also trigger an exception which gets displayed in some other manner.
Plugin developers will be able to allow the classes in their libraries/plugins using the following approaches:
-
By providing a custom
META-INF/hudson.remoting.ClassFilterresource file in their plugin -
By adding the
Jenkins-ClassFilter-Whitelisted=truemanifest entry (example for Maven)-
Old Hudson plugins may need an update to be processed as Jenkins plugins by the extensions mentioned above.
-
The manifest should have
Short-Name, and any ofPlugin-Version/Jenkins-Versionentries -
The entries are known to be fine for Jenkins/Hudson plugin POM versions newer than 1.320 (with default maven-hpi-plugin settings)
-
-
By implementing the
CustomClassFilterextension point, for dynamic allowlists—for example, all implementations of some interface.
Target date: Jan 10, 2017
JEP-200 requires extra API in Remoting, so pull request #208 needs to be delivered in advance. @oleg-nenashev will release the following components once the rollout plan is confirmed with the JEP sponsor:
-
Remoting
-
docker-slave (including the
latesttag) -
docker-jnlp-slave images (including the
latesttag)
Target date: Jan 13, 2017
-
Once this JEP is approved, the https://jenkins.io/redirect/class-filter/ will be created on the Jenkins website
-
This document should provide a custom guide for creating JIRA issues with the
jep-200label
-
-
The JEP sponsor will write an announcement blogpost, which will describe the change and provide links to mitigation guidelines
-
This blog-post will be used as part of the upgrade guideline for LTS
-
Upgrade guidelines should explicitly recommend backing up the instance before the upgrade
-
Upgrade guidelines will also provide allowlist management guidelines to plugin developers
-
The blog post will include a reference to a Plugins affected by JEP-200 Wiki page, which will be providing info to Jenkins administrators about new discoveries if any.
-
-
Once the blog post draft is approved, Jenkins PR #3120 will be integrated towards the next weekly release
After the weekly release the JEP sponsor (or a group of people nominated by him, JEP-200 maintainer(s)) will be responsible to provide an extra support for the issues:
-
JEP-200 maintainer(s) will regularly review open defects and triage them
-
JEP-200 maintainer(s) may request additional information from the reporter. Finally, they are expected to communicate the triage outcome.
-
Possible triage outcomes:
-
Accepted - patch in the plugin. Patch to be proposed by JEP-200 maintainer(s)
-
Accepted - update allowlist in the core (similar to these cases), patch to be proposed by JEP-200 maintainer(s)
-
Rejected - functional defect. JEP-200 maintainer(s) are not responsible to fix any issue, the reporter can use the suggested workarounds. The issue remains open as a common bug.
-
Rejected - security risk. In such case the issue will be moved to the SECURITY bugtracker and then handled by the Jenkins Security team
-
-
For accepted issues JEP-200 maintainer(s) schedule the fix and communicate ETAs to the reporter
Target Date: Mar 14, 2018 (if the weekly gets accepted to LTS)
There is no plan to backport the proposed change to the 2.89.x LTS baseline. The change will be integrated into the LTS if the governance meeting selects a weekly with the integrated change. Estimated meeting date - Feb 14, 2018.
Notes:
-
The change will be referenced in the upgrade guidelines based on the announcement blog post
-
These guidelines will be updated by the weekly rollout results
-
If there are any unresolved known issues, they will be referenced in the Known Issues section
-
The change may cause regressions in plugins on updating instances. In order to mitigate them, we define an extra support policy in the community.
-
Before May 01, 2018 - JEP-200 maintainer(s) will be responsible to review/triage issues. It means there will be an extra month of active support. The process is similar to the one described in the Phase 2 section.
-
After May 01, 2018 - Issues labeled with
jep-200will not be regularly reviewed by JEP-200 maintainer(s), so the maintainers will be the entry point.
For years, the Jenkins project has received reports of remote code execution (RCE) attacks involving these frameworks. Typically the attacks involve fairly exotic classes in the Java Platform, or sundry libraries such as Groovy. The Jenkins CERT has responded to such reports reactively, by prohibiting the affected classes or packages. That approach has proven unmaintainable, as there is a constant threat of further exploits using unexamined classes.
This proposal switches the denylist to an allowlist of a few dozen entries.
In practice it seems that very few plugins actually need to serialize any (third-party) types outside the allowlist. Many such cases point to dubious design decisions, but to retain compatibility a few such entries are bundled in core. Plugins or administrators can also expand the allowlist if regressions arise.
The past few years have seen a flurry of activity by security researchers regarding Java deserialization vulnerabilities.
The ysoserial attack library has been created to host standard "gadgets";
Moritz Bechler has
published a survey of the field.
While none of the Jenkins CERT team members are experts in this area,
various parties have reported remote code execution (RCE) attacks targeting Jenkins.
In just the past two years, the CERT team has had to issue five security advisories including fixes for deserialization vulnerabilities:
first in
2015-11-11,
when a new ClassFilter denylist was introduced as a defense; then in
2016-02-24,
2016-11-16,
2017-02-01, and
2017-04-26.
At this point it is difficult to have any confidence that the ever-growing denylist in fact covers every dangerous class
bundled in the Java Platform, Jenkins core, or commonly used plugins.
Any newly discovered exploit could be a critical breach in Jenkins security, and it may not be responsibly disclosed.
The exploit in the last (2017-04-26) advisory, like many of the others, was reported against the Jenkins CLI tool. Since this historically used Jenkins Remoting, it allowed remote attackers—often even with no authentication—to run code inside the Jenkins controller. The fallout from this exploit led the CERT team to deprecate use of Remoting in CLI and switch to a safer protocol: JENKINS-41745. Thus Java deserialization exploits are no longer a threat to users of the recommended CLI modes.
Similarly, after 2017-02-01 a potential attack vector involving console notes (markup in Jenkins build logs) was closed: these must now be signed by a key available only inside Jenkins, and deserialization is only performed after successful signature verification.
However, deserialization is still performed on data an attacker could control in two cases. Messages sent from an agent to the Jenkins controller (unprompted, or responses to requests) are normally passed through a "callable allowlist" as of 2014-10-30. This allowlist is only applied after deserializing the message, though, at which point it may be too late. Since an agent JVM is assumed to be compromisable with a little effort by a rogue build (for example, of a malicious pull request), the controller must apply a filter on incoming classes.
XStream deserialization is also performed when loading job (agent, …) definitions from several REST or CLI commands.
These commands require some authentication and authorization,
but it is worrisome that XStream does not require that a class implement the Serializable interface,
so the reserve of potentially exploitable classes is far broader.
Thus any denylist which hopes to be exhaustive must include many more classes than typical gadgets attempt to use.
(Note: Pipeline builds based on the Groovy CPS engine use yet another serialization framework, JBoss Marshalling, to save state.
This is not considered a security issue since the program.dat files are never read from user data.)
The CERT team could continue to expand the denylist in response to newly reported vulnerabilities. This has proven to be a significant maintenance burden, and there is little trust in the result. Outside security authorities have repeatedly urged the Jenkins team to switch to an allowlist.
Jenkins could theoretically switch to other designs that do not involve Java object deserialization. In practice this would be wildly incompatible, requiring a rewrite of much of Jenkins core and most plugins.
Every single class used in serial form by Remoting or XStream could be listed. This would be a gigantic list, however, and would consist mostly of types defined in plugins (thus being antimodular): it is perfectly common to define callables, settings, or nested "structs" in a plugin for purposes of communication or persistence. It seems a reasonable compromise to expect that classes defined specifically for use in Jenkins not expose unsafe deserialization behaviors.
In the other direction, it would be possible to reduce the size of the allowlist
by automatically approving any third-party class which does not define a custom deserialization method such as readResolve.
(There are some tricky points here involving subclasses, since the Serialization specification allows some inheritance of behaviors.)
This would defend against the most obvious attacks which involve unexpected code execution during deserialization of the exploited class itself.
However, some more subtle gadgets rely on a combination of behaviors:
custom deserialization methods in quite standard classes (usually some kind of collection) which call methods like equals or hashCode on elements;
and unusual classes which have unsafe implementations of these methods.
Some experimentation was done on this strategy,
but in fact the allowlist size increase needed to handle third-party classes with no deserialization methods is not dramatic,
and this seems well worth the added measure of safety and transparency.
JDK Enhancement Proposal (JEP) 290 provides a standard way to apply deserialization filters in Java.
This is not particularly helpful for Jenkins.
There are two kinds of filters in JEP 290: declarative and programmatic.
The programmatic filters would allow the full flexibility that Jenkins’ ClassFilter requires.
However, this is only available in Java 9 and later, and anyway we already control the ObjectInputStream construction, so it would be functionally equivalent.
(But with no XStream support.)
The declarative filters are available in Java 8, but are too limited
(for example, we cannot automatically approve types defined in Jenkins code);
these have the advantage of applying to any ObjectInputStream in the system,
but that is only really helpful when defending against attacks like the SignedObject exploit in 2017-04-26,
which was already covered by a denylist entry (and now a lack of allowing as well).
@oleg-nenashev raised a concern about using the same allowlist for Remoting and XStream:
-
With the reference implementation in Jenkins PR #3120 there is no way to approve serialization only for a single serialization type (e.g. only XStream).
-
Possible attack vectors in Remoting and XStream differ, especially when Remoting CLI is enabled due to the features missing in other CLI modes (multiple file parameters, etc.).
-
For attack via XML you usually need
Item.CONFIGUREpermissions -
For attack over Remoting -
Computer.CONFIGUREor write access to Remoting/Swarm Client JAR files on an agent. If Remoting CLI is enabled… then there is no special permissions required.
-
Feedback from the JEP Sponsor:
-
Remoting CLI is not a concern since we are going to consider it as insecure and deprecated option even after integrating
-
Current implementation can be extended in the future if needed. Jenkins core patch may be required to pass information about the serialization type to the
CustomClassFilterimplementations -
CustomClassFilterextension point is restricted now, so any required adjustements can be made by API users when needed.
The BDFL delegate agreed with the provided feedback (Jan 03, 2017). He would like to see better extensibility in the future, but it does not block delivery of JEP-200. It is NOT a deferred task, JEP Sponsor has no plan to implement it. If a need arises, it can been contributed by somebody else.
There is an obvious risk that some plugins will have a legitimate need to serialize and deserialize third-party types not covered in the allowlist. In fact it is expected that there will be some such cases; this is simply the cost of having a tighter security policy.
To ameliorate the risk we can check automated test results against the patched core,
specifically scanning for the term class-filter which appears in logs whenever a violation is encountered.
Some runs of acceptance-test-harness (ATH) were already performed in this mode.
plugin-compat-tester (PCT) was also run against an array of plugins.
See the list in Appendix A for more details.
If new allowlist entries are needed after release, they can be added to core in weekly updates. Plugins can also contribute their own allowlist (or even denylist) entries for third-party libraries they bundle, as described in Extensibility above.
Finally, an individual administrator can define site-specific allowlist (or denylist) entries with a system property hudson.remoting.ClassFilter.
This could be useful as an emergency measure, allowing functionality to be restored while awaiting a new plugin release.
(Such a command-line option could be noted as a workaround in a JIRA bug report by someone familiar with the Jenkins security architecture.)
jenkins.security.ClassFilterImpl.SUPPRESS_WHITELIST disables the allowlist, logging violations, but keeps the denylist;
jenkins.security.ClassFilterImpl.SUPPRESS_ALL disables them both (which is very dangerous).
This proposal is expected to strictly improve Jenkins security, as the existing denylist is retained as a fallback unless deliberately overridden.
A new redirect https://jenkins.io/redirect/class-filter/ has been offered, pointing to documentation for this feature.
This permalink is printed to log messages appearing when a allowlist violation is encountered;
in these cases plugin developers or administrators are likely to need instructions on how to proceed.
|
ℹ️
|
This section is listed as described by the JEP Sponsor. Additional testing has been performed during the JEP-200 review in order to evaluate the proposal. Testing notes for the JEP review phase can be found in Appendix A and the linked documents. |
The reference implementation includes test coverage for the essential aspects of the newly added filter: for example, that an example library class not currently included in the allowlist is rejected under the expected conditions.
A number of core tests had already been added during various advisories as mentioned in the motivation. When the fallback to the original denylist is disabled, these continue to pass, indicating that the allowlist alone is a good defense. (In a few cases, some technical changes had to made to these tests to ensure that they exercised a realistic code path.)
The interesting testing is however driven by scanning ATH and PCT results for failures mentioning certain keywords, as detailed in the discussion on backwards compatibility. The broader the set of plugins which can be included in these test runs, the more regressions will be caught early.
For example, a mistake in the dockerhub-notification plugin (that would have caused errors under this proposal)
was already detected by an automated test run, and a simple fix proposed and merged.
Testing against this proposal also rediscovered JENKINS-47158, though sufficient reasonable allowlist entries were added to not cause regressions for Blue Ocean even if that were not fixed.
In several cases, test failures and consequent allowlist additions highlighted poor design decisions in existing code.
For example, as of
PR 497
the git plugin does a lot of tricky things with the Eclipse JGit library.
That is true even if you have specified the CLI implementation of Git for use in the build!
In this case, GitSCM.printCommitMessageToLog asks the agent to return a RevCommit (a JGit type),
which is serialized and deserialized, and then the controller calls getShortMessage() on that structure.
It would be simpler, faster, and safer to do this processing on the agent and send back a String,
but the deceptive ease of Remoting tempts developers to do the wrong thing.
Enforcing an allowlist in the baseline version of Jenkins might help guide them to the simpler solution.
Functional tests (using JenkinsRule) which employ mocking frameworks (Mockito / PowerMock)
force the new filter to be disabled, as the changes to class loading prevent normal operation.
Thus any plugin functionality covered only by mock-based tests might quietly regress.
Fortunately these tests generally check only unit functionality to begin with,
and are not likely to be exercising interesting code paths such as settings storage or remote calls to agents.
For similar reasons, certain tests written in Groovy rather than Java prevent normal filter operation and may fail spuriously.
During JEP-200 review an extra testing has been performed. Testing steps and discovered issues are being tracked in link: JEP-200 Testing Notes.
-
Jenkins Acceptance Test Harness (ATH) has been executed with the patched components, several plugins were fixed (see the Testing section)
-
Jenkins WAR from Jenkins PR #3120 has been tested with a custom core
-
After reviewing of existing ATH tests we concluded that usage of a custom WAR is not a problem
-
We agreed that testing against obsolete dependencies could be a problem. During the PR merge procedure the JEP Sponsor and the BDFL Delegate will rerun ATH to confirm there is no issues with stock Jenkins WAR
-
-
Plugin Compatibility Tester (PCT)
-
Originally PCT has been executed for a limited plugin set for a sponsor’s custom Jenkins WAR.
-
It was decided that it is not enough (not recent plugin versions, potential impact on the plugin behavior by the custom logic), so the BDFL delegate and the JEP Sponsor re-run PCT with a standard Jenkins WAR
-
During testing the BDFL Delegate discovered issues which prevent him from running PCT in particular cases. These issues are listed in the JENKINS-48734 EPIC. The blocker issues have been resolved.
-
All plugins recommended in the Jenkins Installation Wizard have been covered as well as many other plugins
-
BDFL delegate has not tested Pipeline and Blue Ocean plugins, because they are being maintained by employees of the JEP Sponsor’s company. According to the JEP sponsor, they were covered by their internal testing procedure.
-
Although there will be extra testing performed before the release of the change in the Weekly, BDFL confirms that the current test coverage is good enough to accept this Jenkins Enhancement Proposal (Jan 08, 2017).
-
Jenkins PR #3120 contains the bulk of the change and links to related PRs.
-
Remoting PR #208 introduces the new API required to deliver the change.