New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shared data context and data passing between steps #2482

Merged
merged 236 commits into from Jun 20, 2017

Conversation

Projects
None yet
5 participants
@gschueler
Member

gschueler commented May 3, 2017

Work-in-progress. (target release: 2.9.0)

Goal: add ability for workflow steps to output data variables which can be used in later steps.

(This is a precursor to adding "data capture plugins" which can parse command output and extract useful data values for later steps.)

  • workflow engine refactoring to support a shared data context between steps
  • add a "scoped" shared data context for collating data in global, node step, and node contexts
  • step context should allow Output data to specify a scope, e.g. export to global scope instead of local scope.
  • add mechanism for expanding variable references like ${data.key@node} and other scoped expansion (syntax work-in-progress)
  • variable expansion should resolve values by widening search scope if scope is not specified, otherwise do not widen scope if scope is explicit
  • finalize variable syntax
  • use shared context variable expansion in place of previous mechanism in all step contexts
@jtobard

This comment has been minimized.

Show comment
Hide comment
@jtobard

jtobard Jun 15, 2017

Contributor

Issue using a workflowstep plugin running a local execution or to a single node:
Having this definition:
screenshot at jun 15 13-21-04
Using this regex: ^.+?: (.+?)\s*=\s*(.+)$

The output is captured as can be seen here:
screenshot at jun 15 13-21-50

But the data is not available to the next job.

Same behavior using a command job its ok:
screenshot at jun 15 13-22-18
screenshot at jun 15 13-22-34

Contributor

jtobard commented Jun 15, 2017

Issue using a workflowstep plugin running a local execution or to a single node:
Having this definition:
screenshot at jun 15 13-21-04
Using this regex: ^.+?: (.+?)\s*=\s*(.+)$

The output is captured as can be seen here:
screenshot at jun 15 13-21-50

But the data is not available to the next job.

Same behavior using a command job its ok:
screenshot at jun 15 13-22-18
screenshot at jun 15 13-22-34

@gschueler gschueler merged commit 866af8a into rundeck:master Jun 20, 2017

1 check was pending

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
@gschueler

This comment has been minimized.

Show comment
Hide comment
@gschueler

gschueler Jun 20, 2017

Member

premature merge to master

Member

gschueler commented Jun 20, 2017

premature merge to master

@DarwinJS

This comment has been minimized.

Show comment
Hide comment
@DarwinJS

DarwinJS Aug 3, 2017

Is this work still active?

DarwinJS commented Aug 3, 2017

Is this work still active?

@gschueler

This comment has been minimized.

Show comment
Hide comment
@gschueler

gschueler Aug 3, 2017

Member

@DarwinJS this has been merged to master and will be 2.9.0 release

Member

gschueler commented Aug 3, 2017

@DarwinJS this has been merged to master and will be 2.9.0 release

@DarwinJS

This comment has been minimized.

Show comment
Hide comment
@DarwinJS

DarwinJS Aug 3, 2017

@gschueler - thanks - missed that.

Does this only apply to workflow steps or to all step types? (The latter would bring it more on par with TeamCity)

DarwinJS commented Aug 3, 2017

@gschueler - thanks - missed that.

Does this only apply to workflow steps or to all step types? (The latter would bring it more on par with TeamCity)

@gschueler

This comment has been minimized.

Show comment
Hide comment
@gschueler
Member

gschueler commented Aug 3, 2017

@gschueler

This comment has been minimized.

Show comment
Hide comment
@gschueler

gschueler Aug 3, 2017

Member

@DarwinJS to answer your question, yes it applies to workflow or node steps. The data captured in a Node context is associated with the node, and is available in later node steps on the same node, but can be referenced explicitly if necessary. It is not supported for Job Reference steps

Member

gschueler commented Aug 3, 2017

@DarwinJS to answer your question, yes it applies to workflow or node steps. The data captured in a Node context is associated with the node, and is available in later node steps on the same node, but can be referenced explicitly if necessary. It is not supported for Job Reference steps

@dhoffi

This comment has been minimized.

Show comment
Hide comment
@dhoffi

dhoffi Mar 30, 2018

is there a "how to use" example somewhere?

dhoffi commented Mar 30, 2018

is there a "how to use" example somewhere?

@blajoie55

This comment has been minimized.

Show comment
Hide comment
@blajoie55

blajoie55 Oct 11, 2018

@gschueler I'm not understanding something and hoping you can help

If passing data generated in a step (like an inline script step) to a job reference step isn't supported, then I don't understand why the issue here:
#116
was closed. The description you wrote states such type of data passing as a requirement:

"Given the following job :
Job A :

step1 : script

step2 : job reference to Job B

It would be very useful to be able to pass a value generated in step1 as commandline argument value for Job B."

I ask because we really need such a feature for our rundeck jobs at my company and without it we may be forced to find a rundeck alternative :-(

blajoie55 commented Oct 11, 2018

@gschueler I'm not understanding something and hoping you can help

If passing data generated in a step (like an inline script step) to a job reference step isn't supported, then I don't understand why the issue here:
#116
was closed. The description you wrote states such type of data passing as a requirement:

"Given the following job :
Job A :

step1 : script

step2 : job reference to Job B

It would be very useful to be able to pass a value generated in step1 as commandline argument value for Job B."

I ask because we really need such a feature for our rundeck jobs at my company and without it we may be forced to find a rundeck alternative :-(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment