transform_ex.rst

• @transform(.., add_inputs(...)| inputs(...), ...) <new_manual.inputs> in the Ruffus Manual
• Decorators <decorators> for more decorators

transform

@transform( input_, filter_, replace_inputs_ | add_inputs_, output_, [extras_,...] )

Purpose:

Applies the task function to transform data from input_ to output_ files.

This variant of @transform allows additional inputs or dependencies to be added dynamically to the task.

Output file names are specified from input_, i.e. from the output_ of specified tasks, or a list of file names, or a _ matching pattern.

This variant of @transform allows additional or replacement input file names to be derived in the same way.

String replacement occurs either through suffix matches via suffix<decorators.suffix> or the formatter<decorators.formatter> or regex<decorators.regex> indicators.

It is a one to one operation.

add_inputs(...)<decorators.add_inputs> nests the the original input parameters in a list before adding additional dependencies.

inputs(...)<decorators.inputs> replaces the original input parameters wholescale.

Only out of date tasks (comparing input and output files) will be run

Example of add_inputs(...)<decorators.add_inputs>

A common task in compiling C code is to include the corresponding header file for the source.

To compile *.c to *.o, adding *.h and the common header universal.h:

@transform(["1.c", "2.c"], suffix(".c"), add_inputs([r"\1.h", "universal.h"]),  ".o")
def compile(infile, outfile):
    pass

This will result in the following functional calls:

compile(["1.c", "1.h", "universal.h"], "1.o")
compile(["2.c", "2.h", "universal.h"], "2.o")

Example of inputs(...)<decorators.inputs>

inputs(...)<decorators.inputs> allows the original input parameters to be replaced wholescale.

This can be seen in the following example:

@transform(input          = [  ["1.c", "A.c", 2]
                               ["2.c", "B.c", "C.c", 3]],
           filter         =    suffix(".c"),
           replace_inputs = inputs([r"\1.py", "docs.rst"]),
           output         = ".pyc")
def compile(infile, outfile):
    pass

This will result in the following functional calls:

compile(["1.py", "docs.rst"], "1.pyc")
compile(["2.py", "docs.rst"], "2.pyc")

Parameters:

• input = tasks_or_file_names

  can be a:

  1. Task / list of tasks (as in the example above).

     File names are taken from the output_ of the specified task(s)

  2. (Nested) list of file name strings.

     File names containing *[]? will be expanded as a _.

     E.g.:"a.*" => "a.1", "a.2"

• filter = suffix(suffix_string)

  must be wrapped in a suffix<decorators.suffix> indicator object. The end of each input_ file name which matches suffix_string will be replaced by output_. Thus:

  @transform(input = ["a.c", "b.c"],
             filter = suffix(".c"),
             output = ".o")
  def compile(infile, outfile):
      pass

  will result in the following function calls:

  compile("a.c", "a.o")
  compile("b.c", "b.o")

  File names which do not match suffix_string will be ignored

• filter = regex(matching_regex)

  is a python regular expression string, which must be wrapped in a regex<decorators.regex> indicator object See python regular expression (re) documentation for details of regular expression syntax Each output file name is created using regular expression substitution with output

• filter = formatter(...)

  a formatter<decorators.formatter> indicator object containing optionally a python regular expression (re).

• add_inputs = add_inputs(...) or replace_inputs = inputs(...)

  Specifies the resulting input_(s) to each job.

  Positional parameters must be disambiguated by wrapping the values in inputs(...)<decorators.inputs> or an add_inputs(...)<decorators.add_inputs>.

  Named parameters can be passed the values directly.

  Takes:

  1. Task / list of tasks.

     File names are taken from the output of the specified task(s)

  2. (Nested) list of file name strings.

     Strings will be subject to substitution. File names containing *[]? will be expanded as a _. E.g. "a.*" => "a.1", "a.2"

• output = output

  Specifies the resulting output_ file name(s) after string substitution

• extras = extras

  Any extra parameters are passed verbatim to the task function

  If you are using named parameters, these can be passed as a list, i.e. extras= [...]

  Any extra parameters are consumed by the task function and not forwarded further down the pipeline.

  If the regex(...) or formatter(...) parameter is used, then substitution is first applied to (even nested) string parameters. Other data types are passed verbatim.

  For example:

  @transform(["a.c", "b.c"], regex(r"(.*).c"), inputs(r"\1.c", r"\1.h", "universal.h"),  r"\1.o", r"\1")
  def compile(infiles, outfile, file_name_root):
      # do something here
      pass

  will result in the following function calls:

  compile(["1.c", "1.h", "universal.h"], "1.o", "1")
  compile(["2.c", "2.h", "universal.h"], "2.o", "2")

See here <decorators.transform> for more straightforward ways to use transform.