Configurable number of executors

[javierlopezfernandez]

Our test jobs normally chain other jobs. For that reason we require more than one executor per slave. Looking at the code I see numExecutors is hardcoded to 1 and I wonder if this is temporal or permanent feature

[magnayn]

Probably permanent, because I can't see the use of >1 executor per docker slave.

Since docker is an extremely lightweight "process wrapper", it's (basically) no more expensive to instantiate a new slave than it is to try to run a second executor within the first. Just bump the number of allowable up in the cloud configuration

There's also no general guarantee that the test executor will be in the same slave as the upstream job (well, you could tie it with labels, but... yuck).

If you're trying to 'share' outputs to downstream jobs, tag the build job, then use a linked container for the downstream jobs.

[horaceheaven]

@magnayn Since the number of executors per slave is restricted to one, is it possible to run multiple slaves of the same image in the same job?

Also, I see that you indicated above to "Just bump the number of allowable up in the cloud configuration", I'm not sure exactly which option you're referring to in the configurations, can you shed some light on this?

[thomassuckow]

I am not sure what you are trying to accomplish with "run multiple slaves of the same image in the same job".

If you mean can the same image be used simultaneously, absolutely. I have many matrix jobs that spawn 8 instances of the same image.

If you are just wanting to use multiple CPU cores, just use them. Though be aware of what jobs are running and whether you will be over provisioning your system if the maximum number of jobs runs.

If you mean something else, please elaborate.

[horaceheaven]

@thomassuckow Ultimately, what I'm trying to accomplish is to run concurrent builds using this plugin on Jenkins. When I click "Build Now" while a previous build is running on a docker slave, I get the following message "(pending—Waiting for next available executor on docker-slave-test-0)". I was wondering if its possible to get some form of concurrency using this plugin?

As @magnayn mentioned earlier, the default number of executors for each docker slave is 1. He also made reference to "Just bump the number of allowable up in the cloud configuration", I'm not sure what exactly he meant by that.

[thomassuckow]

Unfortunately that is controlled by Jenkins based on the information in /load-statistics. When you only have two jobs, it may just wait for the first to complete. If the job takes sufficiently long Jenkins will go ahead and spin up another instance for the second job. Just make sure that you haven't set the max instances for Docker too low that it isn't allowed to spin up another.

Edit: Note that often Jenkins will wait if it has no estimated time for the Job (For example, the first time you run it).

[horaceheaven]

It seems as if the issue I'm facing is related to this https://issues.jenkins-ci.org/browse/JENKINS-24752

[maafy6]

I'm going to throw in my lot by saying this would be pretty useful, although I don't claim the same use case that the OP has, I just want to be able to run multiple builds on the same slave.

It's not entirely true that spinning up multiple containers is the same as (or even some epsilon less than delta than) multiple executors in the same container. There is a decent overhead in launching a container, having the jar copied over, doing the handshake with the server, yada yada yada. If you have the same issue as several others have reported of slave containers being stuck in suspended mode, you can have a fair amount of churn waiting for a container that works, and when that happens, you want to milk it for all it's worth.

Even if you do have mostly successful containers, I've found it pretty easy to DOS myself with a job that has about 20 sub-jobs, each of which were spinning up containers to do their build. Even with caps (instance and total), it was about a 50-50 shot as to whether or not I'd have to give up and restart Jenkins because it couldn't handle the churn.

I hacked the changes in the above commit together to test on our system here and it's working much, much smoother. I'm not 100% interested in opening a pull request right now because the version that that's based off of is fairly old, and I haven't been bothered to figure out what I need to do to build with the new docker-java jar the pom mentions because it can't seem to find it, but that's the base. It mostly merges straight into the current master (one files had some imports rearranged, but that was the only real conflict).

[KostyaSha]

experimental feature in master, try follow references... too many duplicated issues..

[mikemol]

Probably permanent, because I can't see the use of >1 executor per docker slave.

I realize that this may or may not be particularly relevant to this specific issue, but in Openshift, when namespace quotas are enabled, you can only get so much CPU or memory available for your pods and containers. And, for some setups, you must set up quotas and CPU/memory commits for each pod, which means you can't rely on an idle container leaving cycles or resources for another container to use; a container effectively has 100% utilization of its quota at all times.

I realize this is a terrible setup, and systems probably shouldn't be set up that way, but it happens, and it's not always under our control. As such, in order to be able to do effective resource sharing, the only way to do it in such as case is inside the container, and the only way to do that here is with multiple executors to an agent.

[pjdarton]

If containers are run with the "docker once" strategy where they are discarded immediately after completing any form of build activity then you can't have more than one executor (because the first one to finish would kill all of them).

If you were to implement this feature, you'd have to implement a different retention-strategy, e.g. a "until it's been left idle for too long" strategy (as used by the OpenStack plugin etc) and then you could have more than one executor ... although that would be removing most of the advantages to having "one-use disposable" containers because you then couldn't be sure your builds were starting with a clean environment (just as with static slaves).

It would be much better to run more than one container. Docker is very good at allowing memory re-use so you should find that N containers each having 1 executor have a very similar resource usage to 1 container with N executors.

[mikemol]

To be clear, I deal with the clean-workspace issue with a

node() {
  cleanWs(disableDeferredWipeout: true, deleteDirs: true)
  try {
    // do work
  } finally {
    cleanWs(disableDeferredWipeout: true, deleteDirs: true)
  }
}

pattern, implemented as a utility function that takes a closure as an argument. So, roughly (you'd need to modify to pass in parameters managing node()'s arguments):

def wsWrap(Closure body() ) {
  def res
  node() {
    cleanWs(disableDeferredWipeout: true, deleteDirs: true)
    try {
      res = body()
    } finally {
      cleanWs(disableDeferredWipeout: true, deleteDirs: true)
    }
  }
  return res
}

It only leaks occasionally, and then only when communications between the agent and the master are totally fouled up. But idleMinutes usually steps in and cleans things up when that happens...and the initial cleanWs() prevents it from being anything more than a storage leakage problem.

[pjdarton]

In simple cases, sure, having a clean workspace is all you need to worry about. In my experience, I've found that my developers can get "inventive" when it comes to making a mess that they can't tolerate starting with 😁
That's why I like one-use disposable containers.

However, regardless of the reasons, the underlying issue remains the same - this plugin (currently) only supports having a "one-use" retention strategy and, if you want multiple executors, this plugin would need something more complex. If you're interested in doing that, take a look at the CONTRIBUTING.md guidelines.