[JENKINS-72325] Define an executor and scheduler for SandboxResolvingClassLoader
#543
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
basil
force-pushed
the
cacheExecutor2
branch
3 times, most recently
from
4683a5a
to
c275367
Compare
basil
changed the title
SandboxResolvingClassLoaderSandboxResolvingClassLoader
basil
added a commit
to basil/bom
that referenced
this pull request
Nov 17, 2023
basil
changed the title
SandboxResolvingClassLoaderSandboxResolvingClassLoader
basil
force-pushed
the
cacheExecutor2
branch
from
c275367
to
648a16b
Compare
jglick
approved these changes
Nov 27, 2023
Comment on lines
+73
to
+78
| * <p>In the medium term, once {@link Thread#inheritedAccessControlContext} is removed upstream, we could possibly | ||
| * switch to a combination of {@link Executors#newCachedThreadPool} and {@link ClassLoaderSanityThreadFactory}. | ||
| * | ||
| * <p>In the long term, a listener should be added to inform this class when dynamically installed plugins become | ||
| * available, as described in the comments to {@link #makeParentCache(boolean)}, in which case the use of Caffeine | ||
| * could possibly be removed entirely. |
There was a problem hiding this comment.
Actually the listener approach could probably be implemented by us at any time, whereas the removal of Thread.inheritedAccessControlContext may take years I guess.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Environment
Apache Maven 3.9.5 (57804ffe001d7215b5e7bcb531cf83df38f93546)
Maven home: /usr/share/apache-maven-3.9.5
Java version: 21.0.1, vendor: Eclipse Adoptium, runtime: /opt/jdk-21
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.10.197-186.748.amzn2.x86_64", arch: "amd64", family: "unix"
Steps to reproduce
See JENKINS-72325. Run a
jenkinsci/bomjob with the following patch:This patch runs
org.jenkinsci.plugins.workflow.libs.LibraryMemoryTestin a loop against the latest Jenkins weekly release on Java 21.Expected results
The test runs in a loop without failure. This is the actual result when running the test on Java 17.
Actual results
The test eventually fails after 1-25 iterations with
From standard error:
The failure cannot be easily reproduced locally; however, it failed in CI (with the above patch) after about 1-25 iterations against commit 2d062012 (Nov 13), commit 7ae60b1 (Nov 5), commit 430707e (October 29), commit c300386 (October 16), commit 155e2e7 (October 9), and commit 2798690 (October 2). So this is not a very recent flake.
Evaluation
Care must be taken to avoid leaking instances of
GroovyClassLoaderwhen computing cached values inSandboxResolvingClassLoader. In its default configuration, Caffeine usesForkJoinPool#commonPoolas itsExecutor. As of recent Java versions,ForkJoinPoolcan capture a reference toGroovyClassLoaderby creating aForkJoinWorkerThreadwhoseThread#inheritedAccessControlContextrefers to anAccessControlContextwhoseProtectionDomainrefers toGroovyClassLoader.This is arguably a Java Platform regression, likely caused by JEP-425. However, as of JEP-411,
Thread#inheritedAccessControlContextis deprecated for removal. Therefore, we explicitly decline to report this issue upstream, although we must contend with it in the short to medium term in Jenkins.Rejected Solution
We initially tried configuring Caffeine with the
Executorreturned byExecutors#newCachedThreadPool. However, this merely traded one set of problems for another. ThisExecutorwas seen to also capture a reference toGroovyClassLoaderby creating aThreadwhoseThread#inheritedAccessControlContextrefers to anAccessControlContextwhoseProtectionDomainrefers toGroovyClassLoader:Additionally, when the thread pool's
ThreadFactoryis not wrapped byClassLoaderSanityThreadFactory(see JENKINS-49206), theExecutorcan sometimes createThreadinstances whoseThread#contextClassLoaderrefers toGroovyClassLoader:Given the problems with this approach, we rejected a solution of
Executors#newCachedThreadPool.Solution
We create a dedicated
Executors#newSingleThreadExecutor, which is safe for use with Caffeine from a memory perspective because:ForkJoinPool#commonPool, the thread is eagerly created and avoids references toGroovyClassLoaderinThread#inheritedAccessControlContext.Executors#newCachedThreadPool, the thread is eagerly created and avoids references toGroovyClassLoaderinThread#inheritedAccessControlContext.Executors#newCachedThreadPool, the thread is eagerly created and avoids references toGroovyClassLoaderinThread#contextClassLoader, thereby avoiding the need forClassLoaderSanityThreadFactory.A single-threaded
Executoris safe for use with Caffeine from a CPU perspective because the cache's work is implemented with cheap O(1) algorithms.Future work
In the medium term, once
Thread#inheritedAccessControlContextis removed upstream, we could possibly switch to a combination ofExecutors#newCachedThreadPoolandClassLoaderSanityThreadFactory.The comments state:
If this were done, then this two-level Caffeine cache could possibly be replaced by something based on ClassValue. See, for example, org.kohsuke.stapler.ClassLoaderValue. We explicitly decline to perform this cleanup, but we have updated the existing wish list comment to add this cleanup to the list.
Implementation notes
See the code comments.
Testing done
We tested by running the test in a loop in CI (the only environment where the problem can be reliably reproduced in under 50 iterations) in four configurations:
In configuration 3, the test always failed within 50 iterations with the error shown above. In configuration 1, the test passed reliably after two hours of iterations. After this PR, configurations 2 and 4 are passing reliably after two hours of iterations on both Java 17 and Java 21.
In addition to the above tests, I also ran a successful
jenkinsci/bomtest with theweekly-testlabel to test PCT for all plugins against the weekly release line.