Skip to content
Browse files
[JENKINS-39370,JENKINS-39369] - Support of work directories in Remoti…
…ng (#129)

* [JENKINS-39370,JENKINS-39369] - Add workDir parameter and automatically create logs there if specified

* Save the progress

* [JENKINS-39370] - Generalize the workspace manager for Java Web Start

* [JENKINS-39370] - WiP - Save progress in the test suite

* [JENKINS-39370] - Add tests for WorkDirManager

* [JENKINS-39370] - Restrict the range of supported symbols in the remoting work directory

* [JENKINS-39370] - Generalize the workspace initialization checks

* [JENKINS-39130] - Allow specifying flag for failing initialization if workdir is missing

* [JENKINS-39370] - Reference DirType in WorkDirManager Javadocs

* [JENKINS-39370] - @stephenc noticed that workDir may be null, Intellij IDEA adoption fun

* [JENKINS-39370] - Seems this message breaks our CI

* [JENKINS-39370] - Another message, which likely breaks the CI instance

* [JENKINS-39370] - Simplify the log handling logic

* [JENKINS-39817] - Introduce the agentLog parameter in remoting.jnlp.Main

@stephenc suggested doing it in the PR, so I decided to address it as a part of JENKINS-39370.
But the code still has initialization in hudson.remoting.Launcher for other logging modes.

* Enable JUL logging to a log-rotated file by default

* [JENKINS-18578] - If workspace manager is defined, use JAR Cache within its interbal directory

* [JENKINS-39370] - Fix the workDirManager's log initialization in Launcher

* [JENKINS-39370] - Draft the documentation

* [JENKINS-39369] - Make JUL logging system configurable via property file

* [JENKINS-39369] - Fixes in logging management after the manual testing

* [JENKINS-39369] - Add tests for the logging subsystem

* [JENKINS-39369] - Respect configuration being passed from java.util.logging.config.file system property
  • Loading branch information
oleg-nenashev committed May 7, 2017
1 parent 3df4ce6 commit 76c9b8ccf14f7def1141565b0dee2e4d1c5508d4
@@ -53,6 +53,8 @@ User documentation:
* [Remoting 3 Compatibility Notes](docs/
* [Remoting Protocols](docs/ - Overview of protocols integrated with Jenkins
* [Remoting Configuration](docs/ - Configuring remoting agents
* [Logging](docs/ - Logging
* [Work Directory](docs/ - Remoting work directory (new in Remoting `TODO`)
* [Jenkins Specifics](docs/ - Notes on using remoting in Jenkins
* [Troubleshooting](docs/ - Investigating and solving common remoting issues

@@ -0,0 +1,46 @@

In Remoting logging is powered by the standard `java.util.logging` engine.
The default behavior depends on the [Work Directory]( mode.

### Configuration

In order to configure logging it is possible to use an external property file, path to which can be defined using the `-loggingConfig` CLI option or the `java.util.logging.config.file` system property.

If logging is configured via `-loggingConfig`, some messages printed before the logging system initialization may be missing in startup logs configured by this option.

See details about the property file format
in [Oracle documentation](
and [this guide](
Note that `ConsoleHandler` won't be enabled by default if this option is specified.

### Default behavior with work directory

With work directory Remoting automatically writes logs to the disk.
This is a main difference from the legacy mode without workDir.

Logging destinations:

* Logs include `java.util.logging` and messages printed to _STDOUT/STDERR_ directly.
* Files - `${workDir}/${internalDir}/logs` directory
* File base name - `remoting.log`
* Logs are being automatically rotated.
By default, Remoting keeps 5 10MB files
* Default logging level - `INFO`
* If the legacy `-agentLog` or `-slaveLog` option is enabled, this file logging will be disabled.

If `-agentLog` or `-slaveLog` are not specified, `${workDir}/${internalDir}/logs` directory will be created during the work directory initialization (if required).

<!--TODO: Mention conflict with early initialization by java.util.logging.config.file?-->

### Default behavior without work directory (legacy mode)

* By default, all logs within the system are being sent to _STDOUT/STDERR_ using `java.util.logging`.
* If `-agentLog` or `-slaveLog` option is specified, the log will be also forwarded to the specified file
* The existing file will be overridden on startup
* Remoting does not perform automatic log rotation of this log file

Particular Jenkins components use external features to provide better logging in the legacy mode.
E.g. Windows agent services generate logs using features provided by [Windows Service Wrapper (WinSW)](
@@ -0,0 +1,45 @@
Remoting Work directory

In Remoting work directory is a storage

Remoting work directory is available starting from Remoting `TODO`.
Before this version there was no working directory concept in the library itself;
all operations were managed by library users (e.g. Jenkins agent workspaces).

### Before Remoting TODO

* There is no work directory management in Remoting itself
* Logs are not being persisted to the disk unless `-slaveLog` option is specified
* JAR Cache is being stored in `${user.home}/.jenkins` unless `-jarCache` option is specified

### After Remoting TODO

Due to compatibility reasons, Remoting retains the legacy behavior by default.
Work directory can be enabled using the `-workDir` option in CLI or via the `TODO` [system property](

Once the option is enabled, Remoting starts using the following structure:

|_ ${INTERNAL_DIR} - defined by '-internalDir', 'remoting' by default
|_ jarCache - JAR Cache
|_ logs - Remoting logs
|_ ... - Other directories contributed by library users

Structure of the `logs` directory depends on the logging settings.
See [this page]( for more information.

### Migrating to work directories in Jenkins

:exclamation: Remoting does not perform migration from the previous structure,
because it cannot identify potential external users of the data.

Once the `-workDir` flag is enabled in Remoting, admins are expected to do the following:

1. Remove the `${user.home}/.jenkins` directory if there is no other Remoting instances running under the same user.
2. Consider upgrading configurations of agents in order to enable Work Directories
* SSH agents can be configured in agent settings.
* JNLP agents should be started with the `-workDir` parameter.
* See [JENKINS-TODO](TODO) for more information about changes in Jenkins plugins, which enable work directories by default.
@@ -29,9 +29,12 @@
import java.nio.file.Path;
@@ -68,6 +71,7 @@
import org.jenkinsci.remoting.engine.JnlpConnectionStateListener;
import org.jenkinsci.remoting.engine.JnlpProtocolHandler;
import org.jenkinsci.remoting.engine.JnlpProtocolHandlerFactory;
import org.jenkinsci.remoting.engine.WorkDirManager;
import org.jenkinsci.remoting.protocol.IOHub;
import org.jenkinsci.remoting.protocol.cert.BlindTrustX509ExtendedTrustManager;
import org.jenkinsci.remoting.protocol.cert.DelegatingX509ExtendedTrustManager;
@@ -150,7 +154,62 @@ public void run() {
private boolean keepAlive = true;

private JarCache jarCache = new FileSystemJarCache(new File(System.getProperty("user.home"),".jenkins/cache/jars"),true);

* Default JAR cache location for disabled workspace Manager.
private static final File DEFAULT_NOWS_JAR_CACHE_LOCATION =
new File(System.getProperty("user.home"),".jenkins/cache/jars");

private JarCache jarCache = null;

* Specifies a destination for the agent log.
* If specified, this option overrides the default destination within {@link #workDir}.
* If both this options and {@link #workDir} is not set, the log will not be generated.
* @since TODO
private Path agentLog;

* Specified location of the property file with JUL settings.
* @since TODO
private Path loggingConfigFilePath = null;

* Specifies a default working directory of the remoting instance.
* If specified, this directory will be used to store logs, JAR cache, etc.
* <p>
* In order to retain compatibility, the option is disabled by default.
* <p>
* Jenkins specifics: This working directory is expected to be equal to the agent root specified in Jenkins configuration.
* @since TODO
public Path workDir = null;

* Specifies a directory within {@link #workDir}, which stores all the remoting-internal files.
* <p>
* This option is not expected to be used frequently, but it allows remoting users to specify a custom
* storage directory if the default {@code remoting} directory is consumed by other stuff.
* @since TODO
public String internalDir = WorkDirManager.DirType.INTERNAL_DIR.getDefaultLocation();

* Fail the initialization if the workDir or internalDir are missing.
* This option presumes that the workspace structure gets initialized previously in order to ensure that we do not start up with a borked instance
* (e.g. if a filesystem mount gets disconnected).
* @since TODO
public boolean failIfWorkDirIsMissing = WorkDirManager.DEFAULT_FAIL_IF_WORKDIR_IS_MISSING;

private DelegatingX509ExtendedTrustManager agentTrustManager = new DelegatingX509ExtendedTrustManager(new BlindTrustX509ExtendedTrustManager());

@@ -165,12 +224,68 @@ public Engine(EngineListener listener, List<URL> hudsonUrls, String secretKey, S

* Configures JAR caching for better performance.
* Starts the engine.
* The procedure initializes the working directory and all the required environment
* @throws IOException Initialization error
* @since TODO
public synchronized void startEngine() throws IOException {

@CheckForNull File jarCacheDirectory = null;

// Prepare the working directory if required
if (workDir != null) {
final WorkDirManager workDirManager = WorkDirManager.getInstance();
if (jarCache != null) {
// Somebody has already specificed Jar Cache, hence we do not need it in the workspace.

if (loggingConfigFilePath != null) {

final Path path = workDirManager.initializeWorkDir(workDir.toFile(), internalDir, failIfWorkDirIsMissing);
jarCacheDirectory = workDirManager.getLocation(WorkDirManager.DirType.JAR_CACHE_DIR);
workDirManager.setupLogging(path, agentLog);
} else if (jarCache != null) {
LOGGER.log(Level.WARNING, "No Working Directory. Using the legacy JAR Cache location: {0}", DEFAULT_NOWS_JAR_CACHE_LOCATION);

if (jarCache == null){
if (jarCacheDirectory == null) {
// Should never happen in the current code
throw new IOException("Cannot find the JAR Cache location");
LOGGER.log(Level.FINE, "Using standard File System JAR Cache. Root Directory is {0}", jarCacheDirectory);
jarCache = new FileSystemJarCache(jarCacheDirectory, true);
} else {
LOGGER.log(Level.INFO, "Using custom JAR Cache: {0}", jarCache);

// Start the engine thread

* Configures custom JAR Cache location.
* Starting from TODO, this option disables JAR Caching in the working directory.
* @param jarCache JAR Cache to be used
* @since 2.24
public void setJarCache(JarCache jarCache) {
public void setJarCache(@Nonnull JarCache jarCache) {
this.jarCache = jarCache;

* Sets path to the property file with JUL settings.
* @param filePath JAR Cache to be used
* @since TODO
public void setLoggingConfigFile(@Nonnull Path filePath) {
this.loggingConfigFilePath = filePath;

* Provides Jenkins URL if available.
@@ -198,6 +313,44 @@ public void setNoReconnect(boolean noReconnect) {
this.noReconnect = noReconnect;

* Sets the destination for agent logs.
* @param agentLog Path to the agent log.
* If {@code null}, the engine will pick the default behavior depending on the {@link #workDir} value
* @since TODO
public void setAgentLog(@CheckForNull Path agentLog) {
this.agentLog = agentLog;

* Specified a path to the work directory.
* @param workDir Path to the working directory of the remoting instance.
* {@code null} Disables the working directory.
* @since TODO
public void setWorkDir(@CheckForNull Path workDir) {
this.workDir = workDir;

* Specifies name of the internal data directory within {@link #workDir}.
* @param internalDir Directory name
* @since TODO
public void setInternalDir(@Nonnull String internalDir) {
this.internalDir = internalDir;

* Sets up behavior if the workDir or internalDir are missing during the startup.
* This option presumes that the workspace structure gets initialized previously in order to ensure that we do not start up with a borked instance
* (e.g. if a filesystem mount gets disconnected).
* @param failIfWorkDirIsMissing Flag
* @since TODO
public void setFailIfWorkDirIsMissing(boolean failIfWorkDirIsMissing) { this.failIfWorkDirIsMissing = failIfWorkDirIsMissing; }

* Returns {@code true} if and only if the socket to the master will have {@link Socket#setKeepAlive(boolean)} set.
@@ -239,6 +392,7 @@ public void removeListener(EngineListener el) {

public void run() {
// Create the engine
try {
IOHub hub = IOHub.create(executor);
try {
@@ -37,11 +37,15 @@ public class FileSystemJarCache extends JarCacheSupport {
private final Map<String, Checksum> checksumsByPath = new HashMap<>();

* @param rootDir
* Root directory.
* @param touch
* True to touch the cached jar file that's used. This enables external LRU based cache
* eviction at the expense of increased I/O.
* @throws IllegalArgumentException
* Root directory is {@code null} or not writable.
public FileSystemJarCache(File rootDir, boolean touch) {
public FileSystemJarCache(@Nonnull File rootDir, boolean touch) {
this.rootDir = rootDir;
this.touch = touch;
if (rootDir==null)
@@ -54,6 +58,11 @@ public FileSystemJarCache(File rootDir, boolean touch) {

public String toString() {
return String.format("FileSystem JAR Cache: path=%s, touch=%s", rootDir, Boolean.toString(touch));

protected URL lookInCache(Channel channel, long sum1, long sum2) throws IOException, InterruptedException {
File jar = map(sum1, sum2);

0 comments on commit 76c9b8c

Please sign in to comment.