[JENKINS-682] Clone workspace between jobs

[jenkins-infra-bot]

Seems that the usual way how jobs work on hudson is that there is one job for
building a project and then if I want to have a subsequent job to run all unit
tests, that job needs to download artifacts produced by that build job from
Hudson. As a consequence, people usually add separate targets to their build
scripts for running unit tests from hudson which download the build artifacts
from the network.
I think it would be nice if things could work the way how users would do it when
running tests locally - i.e. use usual targets for building and then simply run
a target for unit tests in the same workspace. This could be achieved by
providing the subsequent jobs an exact snapshot of the workspace at the time
when the parent job finished running.

---

Originally reported by mmatula, imported from: Clone workspace between jobs

• assignee: abayer
• status: Closed
• priority: Critical
• component(s): other
• resolution: Fixed
• resolved: 2010-03-14T08:44:51+00:00
• votes: 42
• watchers: 30
• imported: 2025-12-08

Raw content of original issue

Seems that the usual way how jobs work on hudson is that there is one job for
building a project and then if I want to have a subsequent job to run all unit
tests, that job needs to download artifacts produced by that build job from
Hudson. As a consequence, people usually add separate targets to their build
scripts for running unit tests from hudson which download the build artifacts
from the network.
I think it would be nice if things could work the way how users would do it when
running tests locally - i.e. use usual targets for building and then simply run
a target for unit tests in the same workspace. This could be achieved by
providing the subsequent jobs an exact snapshot of the workspace at the time
when the parent job finished running.

• environment: Platform: All, OS: All

[jenkins-infra-bot]

mmatula:

• Original comment link
• Raw content of original comment:

  Oops, this an enhancement request, not a defect...
  

Oops, this an enhancement request, not a defect...

[jenkins-infra-bot]

kohsuke:

• Original comment link
• Raw content of original comment:

  The problem with doing this is that then you won't be able to run the main build
  and tests in parallel, so it increases the turn-around time.
  
  What's the difference between setting up two jobs with the same workspace, as
  opposed to setting up one job that does both the build and the test?
  

The problem with doing this is that then you won't be able to run the main build
and tests in parallel, so it increases the turn-around time.

What's the difference between setting up two jobs with the same workspace, as
opposed to setting up one job that does both the build and the test?

[jenkins-infra-bot]

mmatula:

• Original comment link
• Raw content of original comment:

  I see. I was not sure how it is implemented - i.e. whether different slaves
  access the same physical workspace. I thought each slave checks out its own
  workspace for a given job in which case tests and a new build could still run in
  parallel.
  I thought the difference between running it as a single job would be exactly
  that possibility to start another build before tests (and possibly other
  subsequent jobs such as coverage, findbugs, javadoc generation, etc.) finished
  running.
  

I see. I was not sure how it is implemented - i.e. whether different slaves
access the same physical workspace. I thought each slave checks out its own
workspace for a given job in which case tests and a new build could still run in
parallel.
I thought the difference between running it as a single job would be exactly
that possibility to start another build before tests (and possibly other
subsequent jobs such as coverage, findbugs, javadoc generation, etc.) finished
running.

[jenkins-infra-bot]

kohsuke:

• Original comment link
• Raw content of original comment:

  Every slave uses a workspace on local file system, so different slaves get
  different workspaces.
  
  The problem I was trying to point out in sharing a workspace is that if a new
  build starts while a test run is in progress, it will most likely mess up. For
  example, a build might try to overwrite the file that a test is using. So in
  general, I don't see how you can reliably run multiple tasks over the same
  workspace in parallel.
  
  Or maybe what you are really saying is a slightly different model, where the
  execution will go like:
  
  
   ws #0   <--- build #N --><-- test #N --->
   ws #1                     <--- build #N+1 --><--test #N+1 --->
  
  ?
  

Every slave uses a workspace on local file system, so different slaves get
different workspaces.

The problem I was trying to point out in sharing a workspace is that if a new
build starts while a test run is in progress, it will most likely mess up. For
example, a build might try to overwrite the file that a test is using. So in
general, I don't see how you can reliably run multiple tasks over the same
workspace in parallel.

Or maybe what you are really saying is a slightly different model, where the
execution will go like:

ws #0 <--- build #N --><-- test #N --->
ws #1 <--- build #N+1 --><--test #N+1 --->

?

[jenkins-infra-bot]

gradopado:

• Original comment link
• Raw content of original comment:

  Have a look at Buildbot, where each build is composed of several build steps.
  Steps are tied to one another, no overlapping.
  
  The first one is mostly cvs-checkout and then you are free to do in steps what
  you think is necessary. Each step has its own result and derective how to
  proceed on WARNING and ERROR cases. This concept is nice.
  

Have a look at Buildbot, where each build is composed of several build steps.
Steps are tied to one another, no overlapping.

The first one is mostly cvs-checkout and then you are free to do in steps what
you think is necessary. Each step has its own result and derective how to
proceed on WARNING and ERROR cases. This concept is nice.

[jenkins-infra-bot]

mmatula:

• Original comment link
• Raw content of original comment:

  Kohsuke, what I would like is to be able to configure my job in such a way that
  it could assume the workspace is in the same state as it was after it's parent
  job. So, if some job is triggered by other job, then I thought it would be good
  if there was a way for this triggered job to inherit the workspace of that
  triggering job. So that people would not have to do additional "magic" specific
  to hudson (and maintain additional hudson-specific targets) in their build
  scripts. Hudson does know that a given job is triggered by another job and most
  likely it is going to use the artifacts that other job produces. So I thought it
  would be nice if those could be provided to these subsequent jobs automatically.
  
  As a user that does not know much about hudson internals, I cannot suggest how
  to implement it. I thought one way (the basic one) it could be done is what you
  showed in your ASCII diagram. Maybe other possible way could be to have a
  mechanism to push a snapshot of the workspace from a job to other jobs it
  triggers (potentially to other machines if those jobs run on other systems). I
  guess this would be least limiting when it comes to parallel execution.
  

Kohsuke, what I would like is to be able to configure my job in such a way that
it could assume the workspace is in the same state as it was after it's parent
job. So, if some job is triggered by other job, then I thought it would be good
if there was a way for this triggered job to inherit the workspace of that
triggering job. So that people would not have to do additional "magic" specific
to hudson (and maintain additional hudson-specific targets) in their build
scripts. Hudson does know that a given job is triggered by another job and most
likely it is going to use the artifacts that other job produces. So I thought it
would be nice if those could be provided to these subsequent jobs automatically.

As a user that does not know much about hudson internals, I cannot suggest how
to implement it. I thought one way (the basic one) it could be done is what you
showed in your ASCII diagram. Maybe other possible way could be to have a
mechanism to push a snapshot of the workspace from a job to other jobs it
triggers (potentially to other machines if those jobs run on other systems). I
guess this would be least limiting when it comes to parallel execution.

[jenkins-infra-bot]

kohsuke:

• Original comment link
• Raw content of original comment:

  A note to myself — recent introduction of Resource interface can be used as a
  lock mechanism for multiple jobs to use the same workspace.
  
  This is convenient for some use cases, like occasionally running lengthy-task
  (like "mvn site") where normally a quick CI build runs.
  

A note to myself — recent introduction of Resource interface can be used as a
lock mechanism for multiple jobs to use the same workspace.

This is convenient for some use cases, like occasionally running lengthy-task
(like "mvn site") where normally a quick CI build runs.

[jenkins-infra-bot]

akostadinov:

• Original comment link
• Raw content of original comment:

  I want to mention the case where workspace is on NFS.
  
  By many of the jobs we lose most of the time checking out the sources so being
  able to share workspace saves considerable amount time. 
  
  That is in the case of a multiple configuration project or the job can be
  separated into few tasks.
  
  I think that by being able to mark arbitrary jobs as sequential that will work
  for us.
  

I want to mention the case where workspace is on NFS.

By many of the jobs we lose most of the time checking out the sources so being
able to share workspace saves considerable amount time.

That is in the case of a multiple configuration project or the job can be
separated into few tasks.

I think that by being able to mark arbitrary jobs as sequential that will work
for us.

[jenkins-infra-bot]

akostadinov:

• Original comment link
• Raw content of original comment:

  adding myself to the cc
  

adding myself to the cc

[jenkins-infra-bot]

kohsuke:

• Original comment link
• Raw content of original comment:

  this is a commonly asked feature, and this is from my colleague, so bumping up
  the priority a bit.
  
  See another recent discussion at
  http://www.nabble.com/Efficiently-using-Hudson-tf4649823.html
  

this is a commonly asked feature, and this is from my colleague, so bumping up
the priority a bit.

See another recent discussion at
http://www.nabble.com/Efficiently-using-Hudson-tf4649823.html

[jenkins-infra-bot]

kohsuke:

• Original comment link
• Raw content of original comment:

  Another recent discussion about this:
  http://www.nabble.com/Multiple-jobs-per-project---to14341510.html
  

Another recent discussion about this:
http://www.nabble.com/Multiple-jobs-per-project---to14341510.html

[jenkins-infra-bot]

kohsuke:

• Original comment link
• Raw content of original comment:

  
  	
  
  	
  
  		
  
  		
  
  			
  Issue 398 has been marked as a duplicate of this issue. ***
  		
  		
  	
  	
  
  
  

• • • Issue 398 has been marked as a duplicate of this issue. ***

[jenkins-infra-bot]

hoshposh:

• Original comment link
• Raw content of original comment:

  Adding myself on the CC list for this issue.
  

Adding myself on the CC list for this issue.

[jenkins-infra-bot]

lloydchang:

• Original comment link
• Raw content of original comment:

  Adding myself as a cc
  

Adding myself as a cc

[jenkins-infra-bot]

jameslivingston:

• Original comment link
• Raw content of original comment:

  adding myself as cc
  

adding myself as cc