Permalink
Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
432 lines (339 sloc) 25.6 KB

JEP-200: Switch Remoting/XStream blacklist to a whitelist

Abstract

Since its early days, Jenkins has used two object serialization frameworks: Java’s built-in API, from the Remoting layer; and XStream, for persisting configuration and settings. Both permit any Java class to be loaded merely by specifying its name in the input.

This proposal switches the blacklist to a whitelist of a few dozen entries: for Jenkins to process a third-party class, it must be explicitly approved. Classes defined in Jenkins core or plugins are exempt, as are a few special categories.

Specification

The main aspect of the change is the introduction of a ClassFilterImpl in Jenkins core which supplants the simple blacklist defined in Remoting. This filter is applied to all Java or XStream deserialization performed by the Jenkins master (as well as XStream serialization, to "fail fast"). A crude blacklist extension API in Remoting introduced in 2017-04-26 is deprecated in favor of the more general ClassFilter.setDefault, as is the system property hudson.remoting.ClassFilter.DEFAULTS_OVERRIDE_LOCATION.

The new filter has several levels of rules:

  • A CustomClassFilter implementation may override any behavior. This is discussed in the compatibility section.

  • Classes mentioned in the original blacklist continue to be rejected.

  • Java arrays and enums are accepted, as these are special formats.

  • Java Throwable types are accepted, as they are commonly used in Remoting responses and it is impractical to enumerate every exception that might be thrown.

  • A class defined in Jenkins core, Remoting, or a Jenkins module/plugin is accepted. This does not apply to classes defined in other JARs packaged in WEB-INF/lib/*.jar.

  • Some classes defined in test utilities, by mocking frameworks, etc., are accepted when running inside tests.

  • Classes defined in a RemoteClassLoader (so, already sent to this JVM from a presumably trusted source) are accepted. This is needed at least by CloudBees Jenkins Operations Center, and clients of the master-to-master-api plugin if there are any.

  • Any other class is accepted if it is explicitly mentioned in a list whitelisted-classes.txt bundled in Jenkins core. These consist of:

    • Basic Java utility types (see the reference implementation).

    • Some other collection types defined in Google Guava.

    • A handful of safe types defined in some libraries used by Jenkins core.

    • A handful of types used by some plugins known to otherwise fail: Git, Pipeline, Maven, Job DSL, etc.

The whitelist has been built up by repeatedly running a large array of tests against a patched core and collecting relevant errors. It is expected that the standard whitelist will be expanded somewhat as field reports arrive of less common code paths needing new entries.

When a class is (first) rejected, a warning is printed to the system log. This is particularly useful when scanning large automated test results for regressions caused by this change. Typically the rejection will also trigger an exception which gets displayed in some other manner.

Extensibility

Plugin developers will be able to whitelist the classes in their libraries/plugins using the following approaches:

  1. By providing a custom META-INF/hudson.remoting.ClassFilter resource file in their plugin

  2. By adding the Jenkins-ClassFilter-Whitelisted=true manifest entry (example for Maven)

    • Old Hudson plugins may need an update to be processed as Jenkins plugins by the extensions mentioned above.

    • The manifest should have Short-Name, and any of Plugin-Version/Jenkins-Version entries

    • The entries are known to be fine for Jenkins/Hudson plugin POM versions newer than 1.320 (with default maven-hpi-plugin settings)

  3. By implementing the CustomClassFilter extension point, for dynamic whitelists—for example, all implementations of some interface.

Rollout plan

Phase 1. Remoting release

Target date: Jan 10, 2017

JEP-200 requires extra API in Remoting, so pull request #208 needs to be delivered in advance. @oleg-nenashev will release the following components once the rollout plan is confirmed with the JEP sponsor:

Phase 2. Weekly Release

Target date: Jan 13, 2017

  1. Once this JEP is approved, the https://jenkins.io/redirect/class-filter/ will be created on the Jenkins website

    • This document should provide a custom guide for creating JIRA issues with the jep-200 label

  2. The JEP sponsor will write an announcement blogpost, which will describe the change and provide links to mitigation guidelines

    • This blog-post will be used as part of the upgrade guideline for LTS

    • Upgrade guidelines should explicitly recommend backing up the instance before the upgrade

    • Upgrade guidelines will also provide whitelisting guidelines to plugin developers

    • The blog post will include a reference to a Plugins affected by JEP-200 Wiki page, which will be providing info to Jenkins administrators about new discoveries if any.

  3. Once the blog post draft is approved, Jenkins PR #3120 will be integrated towards the next weekly release

After the weekly release the JEP sponsor (or a group of people nominated by him, JEP-200 maintainer(s)) will be responsible to provide an extra support for the issues:

  • JEP-200 maintainer(s) will regularly review open defects and triage them

  • JEP-200 maintainer(s) may request additional information from the reporter. Finally, they are expected to communicate the triage outcome.

  • Possible triage outcomes:

    • Accepted - patch in the plugin. Patch to be proposed by JEP-200 maintainer(s)

    • Accepted - update whitelist in the core (similar to these cases), patch to be proposed by JEP-200 maintainer(s)

    • Rejected - functional defect. JEP-200 maintainer(s) are not responsible to fix any issue, the reporter can use the suggested workarounds. The issue remains open as a common bug.

    • Rejected - security risk. In such case the issue will be moved to the SECURITY bugtracker and then handled by the Jenkins Security team

  • For accepted issues JEP-200 maintainer(s) schedule the fix and communicate ETAs to the reporter

Phase 3. LTS availability

Target Date: Mar 14, 2018 (if the weekly gets accepted to LTS)

There is no plan to backport the proposed change to the 2.89.x LTS baseline. The change will be integrated into the LTS if the governance meeting selects a weekly with the integrated change. Estimated meeting date - Feb 14, 2018.

Notes:

  • The change will be referenced in the upgrade guidelines based on the announcement blog post

    • These guidelines will be updated by the weekly rollout results

    • If there are any unresolved known issues, they will be referenced in the Known Issues section

Phase 4. Post-release

The change may cause regressions in plugins on updating instances. In order to mitigate them, we define an extra support policy in the community.

  • Before May 01, 2018 - JEP-200 maintainer(s) will be responsible to review/triage issues. It means there will be an extra month of active support. The process is similar to the one described in the Phase 2 section.

  • After May 01, 2018 - Issues labeled with jep-200 will not be regularly reviewed by JEP-200 maintainer(s), so the maintainers will be the entry point.

Motivation

For years, the Jenkins project has received reports of remote code execution (RCE) attacks involving these frameworks. Typically the attacks involve fairly exotic classes in the Java Platform, or sundry libraries such as Groovy. The Jenkins CERT has responded to such reports reactively, by blacklisting the affected classes or packages. That approach has proven unmaintainable, as there is a constant threat of further exploits using unexamined classes.

This proposal switches the blacklist to a whitelist of a few dozen entries.

In practice it seems that very few plugins actually need to serialize any (third-party) types outside the whitelist. Many such cases point to dubious design decisions, but to retain compatibility a few such entries are bundled in core. Plugins or administrators can also expand the whitelist if regressions arise.

The past few years have seen a flurry of activity by security researchers regarding Java deserialization vulnerabilities. The ysoserial attack library has been created to host standard "gadgets"; Moritz Bechler has published a survey of the field.

While none of the Jenkins CERT team members are experts in this area, various parties have reported remote code execution (RCE) attacks targeting Jenkins. In just the past two years, the CERT team has had to issue five security advisories including fixes for deserialization vulnerabilities: first in 2015-11-11, when a new ClassFilter blacklist was introduced as a defense; then in 2016-02-24, 2016-11-16, 2017-02-01, and 2017-04-26. At this point it is difficult to have any confidence that the ever-growing blacklist in fact covers every dangerous class bundled in the Java Platform, Jenkins core, or commonly used plugins. Any newly discovered exploit could be a critical breach in Jenkins security, and it may not be responsibly disclosed.

The exploit in the last (2017-04-26) advisory, like many of the others, was reported against the Jenkins CLI tool. Since this historically used Jenkins Remoting, it allowed remote attackers—often even with no authentication—to run code inside the Jenkins master. The fallout from this exploit led the CERT team to deprecate use of Remoting in CLI and switch to a safer protocol: JENKINS-41745. Thus Java deserialization exploits are no longer a threat to users of the recommended CLI modes.

Similarly, after 2017-02-01 a potential attack vector involving console notes (markup in Jenkins build logs) was closed: these must now be signed by a key available only inside Jenkins, and deserialization is only performed after successful signature verification.

However, deserialization is still performed on data an attacker could control in two cases. Messages sent from an agent to the Jenkins master (unprompted, or responses to requests) are normally passed through a "callable whitelist" as of 2014-10-30. This whitelist is only applied after deserializing the message, though, at which point it may be too late. Since an agent JVM is assumed to be compromisable with a little effort by a rogue build (for example, of a malicious pull request), the master must apply a filter on incoming classes.

XStream deserialization is also performed when loading job (agent, …) definitions from several REST or CLI commands. These commands require some authentication and authorization, but it is worrisome that XStream does not require that a class implement the Serializable interface, so the reserve of potentially exploitable classes is far broader. Thus any blacklist which hopes to be exhaustive must include many more classes than typical gadgets attempt to use.

(Note: Pipeline builds based on the Groovy CPS engine use yet another serialization framework, JBoss Marshalling, to save state. This is not considered a security issue since the program.dat files are never read from user data.)

Reasoning

The CERT team could continue to expand the blacklist in response to newly reported vulnerabilities. This has proven to be a significant maintenance burden, and there is little trust in the result. Outside security authorities have repeatedly urged the Jenkins team to switch to a whitelist.

Jenkins could theoretically switch to other designs that do not involve Java object deserialization. In practice this would be wildly incompatible, requiring a rewrite of much of Jenkins core and most plugins.

Every single class used in serial form by Remoting or XStream could be listed. This would be a gigantic list, however, and would consist mostly of types defined in plugins (thus being antimodular): it is perfectly common to define callables, settings, or nested "structs" in a plugin for purposes of communication or persistence. It seems a reasonable compromise to expect that classes defined specifically for use in Jenkins not expose unsafe deserialization behaviors.

In the other direction, it would be possible to reduce the size of the whitelist by automatically approving any third-party class which does not define a custom deserialization method such as readResolve. (There are some tricky points here involving subclasses, since the Serialization specification allows some inheritance of behaviors.) This would defend against the most obvious attacks which involve unexpected code execution during deserialization of the exploited class itself. However, some more subtle gadgets rely on a combination of behaviors: custom deserialization methods in quite standard classes (usually some kind of collection) which call methods like equals or hashCode on elements; and unusual classes which have unsafe implementations of these methods. Some experimentation was done on this strategy, but in fact the whitelist size increase needed to handle third-party classes with no deserialization methods is not dramatic, and this seems well worth the added measure of safety and transparency.

JDK Enhancement Proposal (JEP) 290 provides a standard way to apply deserialization filters in Java. This is not particularly helpful for Jenkins. There are two kinds of filters in JEP 290: declarative and programmatic. The programmatic filters would allow the full flexibility that Jenkins’ ClassFilter requires. However, this is only available in Java 9 and later, and anyway we already control the ObjectInputStream construction, so it would be functionally equivalent. (But with no XStream support.) The declarative filters are available in Java 8, but are too limited (for example, we cannot automatically approve types defined in Jenkins code); these have the advantage of applying to any ObjectInputStream in the system, but that is only really helpful when defending against attacks like the SignedObject exploit in 2017-04-26, which was already covered by a blacklist entry (and now a lack of whitelisting as well).

Concern about the single whitelist approach

@oleg-nenashev raised a concern about using the same whitelist for Remoting and XStream:

  • With the reference implementation in Jenkins PR #3120 there is no way to approve serialization only for a single serialization type (e.g. only XStream).

  • Possible attack vectors in Remoting and XStream differ, especially when Remoting CLI is enabled due to the features missing in other CLI modes (multiple file parameters, etc.).

    • For attack via XML you usually need Item.CONFIGURE permissions

    • For attack over Remoting - Computer.CONFIGURE or write access to Remoting/Swarm Client JAR files on an agent. If Remoting CLI is enabled…​ then there is no special permissions required.

Feedback from the JEP Sponsor:

  • Remoting CLI is not a concern since we are going to consider it as insecure and deprecated option even after integrating

  • Current implementation can be extended in the future if needed. Jenkins core patch may be required to pass information about the serialization type to the CustomClassFilter implementations

  • CustomClassFilter extension point is restricted now, so any required adjustements can be made by API users when needed.

The BDFL delegate agreed with the provided feedback (Jan 03, 2017). He would like to see better extensibility in the future, but it does not block delivery of JEP-200. It is NOT a deferred task, JEP Sponsor has no plan to implement it. If a need arises, it can been contributed by somebody else.

Backwards Compatibility

There is an obvious risk that some plugins will have a legitimate need to serialize and deserialize third-party types not covered in the whitelist. In fact it is expected that there will be some such cases; this is simply the cost of having a tighter security policy.

To ameliorate the risk we can check automated test results against the patched core, specifically scanning for the term class-filter which appears in logs whenever a violation is encountered. Some runs of acceptance-test-harness (ATH) were already performed in this mode. plugin-compat-tester (PCT) was also run against an array of plugins. See the list in Appendix A for more details.

If new whitelist entries are needed after release, they can be added to core in weekly updates. Plugins can also contribute their own whitelist (or even blacklist) entries for third-party libraries they bundle, as described in Extensibility above.

Finally, an individual administrator can define site-specific whitelist (or blacklist) entries with a system property hudson.remoting.ClassFilter. This could be useful as an emergency measure, permitting functionality to be restored while awaiting a new plugin release. (Such a command-line option could be noted as a workaround in a JIRA bug report by someone familiar with the Jenkins security architecture.) jenkins.security.ClassFilterImpl.SUPPRESS_WHITELIST disables the whitelist, logging violations, but keeps the blacklist; jenkins.security.ClassFilterImpl.SUPPRESS_ALL disables them both (which is very dangerous).

Security

This proposal is expected to strictly improve Jenkins security, as the existing blacklist is retained as a fallback unless deliberately overridden.

Infrastructure Requirements

A new redirect https://jenkins.io/redirect/class-filter/ has been offered, pointing to a wiki page. This permalink is printed to log messages appearing when a whitelist violation is encountered; in these cases plugin developers or administrators are likely to need instructions on how to proceed.

Testing

ℹ️
This section is listed as described by the JEP Sponsor. Additional testing has been performed during the JEP-200 review in order to evaluate the proposal. Testing notes for the JEP review phase can be found in Appendix A and the linked documents.

The reference implementation includes test coverage for the essential aspects of the newly added filter: for example, that an example library class not currently included in the whitelist is rejected under the expected conditions.

A number of core tests had already been added during various advisories as mentioned in the motivation. When the fallback to the original blacklist is disabled, these continue to pass, indicating that the whitelist alone is a good defense. (In a few cases, some technical changes had to made to these tests to ensure that they exercised a realistic code path.)

The interesting testing is however driven by scanning ATH and PCT results for failures mentioning certain keywords, as detailed in the discussion on backwards compatibility. The broader the set of plugins which can be included in these test runs, the more regressions will be caught early.

For example, a mistake in the dockerhub-notification plugin (that would have caused errors under this proposal) was already detected by an automated test run, and a simple fix proposed and merged.

Testing against this proposal also rediscovered JENKINS-47158, though sufficient reasonable whitelist entries were added to not cause regressions for Blue Ocean even if that were not fixed.

In several cases, test failures and consequent whitelist additions highlighted poor design decisions in existing code. For example, as of PR 497 the git plugin does a lot of tricky things with the Eclipse JGit library. That is true even if you have specified the CLI implementation of Git for use in the build! In this case, GitSCM.printCommitMessageToLog asks the agent to return a RevCommit (a JGit type), which is serialized and deserialized, and then the master calls getShortMessage() on that structure. It would be simpler, faster, and safer to do this processing on the agent and send back a String, but the deceptive ease of Remoting tempts developers to do the wrong thing. Enforcing a whitelist in the baseline version of Jenkins might help guide them to the simpler solution.

Functional tests (using JenkinsRule) which employ mocking frameworks (Mockito / PowerMock) force the new filter to be disabled, as the changes to class loading prevent normal operation. Thus any plugin functionality covered only by mock-based tests might quietly regress. Fortunately these tests generally check only unit functionality to begin with, and are not likely to be exercising interesting code paths such as settings storage or remote calls to agents. For similar reasons, certain tests written in Groovy rather than Java prevent normal filter operation and may fail spuriously.

Tested Plugins

During JEP-200 review an extra testing has been performed. Testing steps and discovered issues are being tracked in link: JEP-200 Testing Notes.

  • Jenkins Acceptance Test Harness (ATH) has been executed with the patched components, several plugins were fixed (see the Testing section)

    • Jenkins WAR from Jenkins PR #3120 has been tested with a custom core

    • After reviewing of existing ATH tests we concluded that usage of a custom WAR is not a problem

    • We agreed that testing against obsolete dependencies could be a problem. During the PR merge procedure the JEP Sponsor and the BDFL Delegate will rerun ATH to confirm there is no issues with stock Jenkins WAR

  • Plugin Compatibility Tester (PCT)

    • Originally PCT has been executed for a limited plugin set for a sponsor’s custom Jenkins WAR.

    • It was decided that it is not enough (not recent plugin versions, potential impact on the plugin behavior by the custom logic), so the BDFL delegate and the JEP Sponsor re-run PCT with a standard Jenkins WAR

    • During testing the BDFL Delegate discovered issues which prevent him from running PCT in particular cases. These issues are listed in the JENKINS-48734 EPIC. The blocker issues have been resolved.

    • All plugins recommended in the Jenkins Installation Wizard have been covered as well as many other plugins

    • BDFL delegate has not tested Pipeline and Blue Ocean plugins, because they are being maintained by employees of the JEP Sponsor’s company. According to the JEP sponsor, they were covered by their internal testing procedure.

Although there will be extra testing performed before the release of the change in the Weekly, BDFL confirms that the current test coverage is good enough to accept this Jenkins Enhancement Proposal (Jan 08, 2017).

Reference Implementation