New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2.9.0 Changes #2577

Closed
gschueler opened this Issue Jun 21, 2017 · 7 comments

Comments

Projects
None yet
4 participants
@gschueler
Member

gschueler commented Jun 21, 2017

(Note: This issue is for preliminary feature documentation, and for discussing 2.9.0 changes.)

New Features

Data Passing

Passing data between steps (Data-capture plugins). Trello card

We've added this feature with a combination of several new pieces under the hood:

  1. A new plugin type: "LogFilterPlugin"
  2. A new "shared data context" used by the Workflow execution system to manage the captured data
  3. A new "Data output" feature used by the Log Filter plugins to add new data to the shared context

Log Filter plugins can be used to observe and modify the log output from indvidual steps in the workflow, or from all steps for a Job when applied as a global Log filter.

Built in LogFilter plugins:

  • Key Value Data: use a regular expression to capture key/value data from the log output of steps
  • Render Formatted Data: mark some or all output of a step as having a certain datatype, e.g. JSON, CSV, HTML, Markdown so it can be rendered in the GUI using "Log Rendering" (see below)
  • Highlight Output: use a regular expression to colorize matching text in the output log
  • Quiet Output: use a regular expression to reduce or increase the log level of log output
  • Mask Passwords: mask any value that matches the Secure Input option values entered by the user (useful to apply at the Global workflow filter level)

Using Key Value Data filter plugin to capture data

The "Key Value Data" Log Filter plugin lets you can specify a regular expression to match against the log output, and will capture key/value pairs from
the log data. Its default regular expression is ^RUNDECK:DATA:(.+?)\s*=\s*(.+)$. This will capture output in the form RUNDECK:DATA:MyKey=MyValue and add it to the "shared data context" for the step and node that emitted it. Later steps can reference the data as ${data.MyKey} (for step or command arguments) or @data.MyKey@ (for inline scripts).

If you have a Node step, such as a command or script, the data will be captured and associated with the particular Node the command executed on. Later when you dereference the value, it will automatically use the value for the current node. If you want to specify a node you can use ${data.MyKey@nodename}. You can reference data captured for a step number X as ${X:data.MyKey} or a specific step and node as ${X:data.MyKey@nodename}.

In addition, you can collect all node values captured as ${data.MyKey*} (with an asterisk). This will expand to a comma-separated string of all node
values captured for that key.

Inside inline scripts, the syntax is @X:data.MyKey/nodename@, (using / instead of @ for node name) but otherwise works the same way.

Useful Stub plugin Steps

The "Stub" plugin bundled with Rundeck adds these new workflow step types:

  • Log Data Step: Log all the data from the "Shared Data Context" as JSON data. This is useful for debugging/viewing all of the data that has been captured and is available in your workflow.
  • Data Step/Data Node Step: Enter json/yaml/properties formatted text to add to the "Shared Data Context" directly, either as a Step context, or a Node Step Context.

Log Rendering

A new plugin type "ContentConverterPlugin" allows output logs which have been marked as a certain content type to be rendered in the GUI in a nicer fashion.

For example, applying the "Render Formatted Data" LogFilter plugin to a step will mark some or all output from the step as being of a certain data type, such as JSON.

When it is marked as JSON, Rundeck can use a ContentConverterPlugin that knows how to parse JSON and render it as HTML to display the content
in the output log.

Content Converter Plugins will be applied automatically to any log data that is marked as having a supported data type. This means that as long as you
mark the log output with a data type (using the Render Formatted Data Log Filter plugin, for example), then any Content Converter plugin that
supports that data type will be used when rendering the log in the GUI.

Built in Content Converter Plugins

  • Json Data Converter: converts JSON text to structured data
  • Tabular Data Converter: converts CSV/TSV text to structured data
  • Properties Data Converter: converts Java Properties text to structured data
  • Markdown View Converter: Renders Markdown text to display as HTML
  • HTML View Converter: Allows HTML to display in the log
  • HTML Table View Converter: Renders structured data as an HTML table

Additional features

Using Jobs across projects:

  • Job Reference Steps can reference Jobs in another project
  • Job State Conditional Step can reference Jobs in another project
  • "Copy Job to another project" action for jobs

Project Passive Mode:

  • You can disable Executions and/or Schedules for Jobs in each project.

No Matched Nodes as Success:

  • You can set a Job to treat "no matched nodes" as successful instead of failure

Improved Time Zone support:

  • Activity page and Job detail use the same time zone for display
  • Specify the Time Zone used in the Job Schedule
@qicaihuoyan

This comment has been minimized.

Show comment
Hide comment
@qicaihuoyan

qicaihuoyan Jun 22, 2017

Very much looking forward to

Very much looking forward to

@gschueler

This comment has been minimized.

Show comment
Hide comment
@gschueler

gschueler Jun 23, 2017

Member
  • new plugin types should be listed in plugins page
Member

gschueler commented Jun 23, 2017

  • new plugin types should be listed in plugins page
@gschueler

This comment has been minimized.

Show comment
Hide comment
@Ciberth

This comment has been minimized.

Show comment
Hide comment
@Ciberth

Ciberth Jul 13, 2017

@gschueler

I just installed the 2.9.0 preview to test a bit with the global log filters. This feature is a real life-saver but there are some small adjustments that would make it perfect.

HTML
I have a script that prints to stdout a html file with a table in it. First of all I lose my border (?) ; I don't know if that is intentional but I wanted to let you know. This is probably because the output is loaded with javascript or something? More importantly the elements next to each other so same row but different column are really glued next to each other making it rather difficult to read. Is this fixable with some whitespace?

CSV
I think there is a small bug with the CSV parsing. When there are empty cells (followed only by empty cells) we get a nice 'Null Value'. So this is fine! But when there are empty cells in between filled in cells then the filled in cells (at the right) get pushed to the left, resulting that the data is in wrong column

An example

HEADER1, HEADER2, HEADER3, HEADER4 
normal, sentence, each, cell has a thing
this is also fine, if you only, have empties at the end,
but this,,,is not fine

Furthermore I took a small peak at your commited code and I saw code that would fetch the header but I have no clue how to tell rundeck that my first row should be used as header (?) maybe I'm missing something here!

This more a feature request but it should be a small thing to fix. It would be awesome to have a direct download link to the formatted file with the correct extension. So meaning if we format to csv then it would be cool to have an url next to the "Text, HTML Download" that is CSV for CSV or MD for markdown that would fetch the output with the correct file extension. If this is implemented you directly implement #1787

Thanks in advance!

Ciberth commented Jul 13, 2017

@gschueler

I just installed the 2.9.0 preview to test a bit with the global log filters. This feature is a real life-saver but there are some small adjustments that would make it perfect.

HTML
I have a script that prints to stdout a html file with a table in it. First of all I lose my border (?) ; I don't know if that is intentional but I wanted to let you know. This is probably because the output is loaded with javascript or something? More importantly the elements next to each other so same row but different column are really glued next to each other making it rather difficult to read. Is this fixable with some whitespace?

CSV
I think there is a small bug with the CSV parsing. When there are empty cells (followed only by empty cells) we get a nice 'Null Value'. So this is fine! But when there are empty cells in between filled in cells then the filled in cells (at the right) get pushed to the left, resulting that the data is in wrong column

An example

HEADER1, HEADER2, HEADER3, HEADER4 
normal, sentence, each, cell has a thing
this is also fine, if you only, have empties at the end,
but this,,,is not fine

Furthermore I took a small peak at your commited code and I saw code that would fetch the header but I have no clue how to tell rundeck that my first row should be used as header (?) maybe I'm missing something here!

This more a feature request but it should be a small thing to fix. It would be awesome to have a direct download link to the formatted file with the correct extension. So meaning if we format to csv then it would be cool to have an url next to the "Text, HTML Download" that is CSV for CSV or MD for markdown that would fetch the output with the correct file extension. If this is implemented you directly implement #1787

Thanks in advance!

@gschueler

This comment has been minimized.

Show comment
Hide comment
@gschueler

gschueler Jul 13, 2017

Member

@Ciberth thanks for the feedback!

If you could attach any sample output you have, e.g. the rdlog file if possible, it would help me reproduce the exact behavior you are seeing.

For the HTML tables, you might try applying the bootstrap css e.g. <table class="table table-bordered">

Maybe we should add css to apply a style refresh to anything within the log output, and use some standard formatting such as bordered tables.

The CSV behavior needs updating for a few things, including conforming to the rfc better, and handling those empty values better.

Member

gschueler commented Jul 13, 2017

@Ciberth thanks for the feedback!

If you could attach any sample output you have, e.g. the rdlog file if possible, it would help me reproduce the exact behavior you are seeing.

For the HTML tables, you might try applying the bootstrap css e.g. <table class="table table-bordered">

Maybe we should add css to apply a style refresh to anything within the log output, and use some standard formatting such as bordered tables.

The CSV behavior needs updating for a few things, including conforming to the rfc better, and handling those empty values better.

@Ciberth

This comment has been minimized.

Show comment
Hide comment
@Ciberth

Ciberth Jul 14, 2017

@gschueler thanks for the quick reply!

HTML
The HTML bootstrap class worked like a charm and makes reading from it easy, thanks for the tip! I'm still unsure about how it all gets processed but it works perfectly now!

CSV
I don't know if it will be that usefull but I remade the example I gave in my post before. All information can be found on this codeshare

In addition to my previous request for the "file format export". I tried looking at the API to ask for a certain filetype when downloading the output but without much success. It would be cool to have something like execution/downloadOutput/40?view=inline&formatted=false&filetype=csv or something. A direct downloadlink would still be better though!

Another small feature request or rather a question since I didn't find anything in the configuration files or the documentation is the view options. By default Time, Node, Step and Ansi Color are enabled. Is there a way to change this default behaviour to disable the checks? It would be a nice little enhancement to be able to change this per job.

Anyway thanks again for the fast reply and the development on rundeck!

Ciberth commented Jul 14, 2017

@gschueler thanks for the quick reply!

HTML
The HTML bootstrap class worked like a charm and makes reading from it easy, thanks for the tip! I'm still unsure about how it all gets processed but it works perfectly now!

CSV
I don't know if it will be that usefull but I remade the example I gave in my post before. All information can be found on this codeshare

In addition to my previous request for the "file format export". I tried looking at the API to ask for a certain filetype when downloading the output but without much success. It would be cool to have something like execution/downloadOutput/40?view=inline&formatted=false&filetype=csv or something. A direct downloadlink would still be better though!

Another small feature request or rather a question since I didn't find anything in the configuration files or the documentation is the view options. By default Time, Node, Step and Ansi Color are enabled. Is there a way to change this default behaviour to disable the checks? It would be a nice little enhancement to be able to change this per job.

Anyway thanks again for the fast reply and the development on rundeck!

@opm22

This comment has been minimized.

Show comment
Hide comment
@opm22

opm22 Jul 27, 2017

really nice!! looking forward for this release!

opm22 commented Jul 27, 2017

really nice!! looking forward for this release!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment