Authoring Analysis Modules

Creating an analysis module is pretty straightforward.

Create a new class that extends ThackTech.Pipelines.PipelineModule

from ThackTech.Pipelines import PipelineModule
class HelloWorld(PipelineModule):
	def __init__(self, **kwargs):
		super_args = dict(name='HelloWorld', short_description='Say "Hello World".')
		super_args.update(**kwargs)
		super(HelloWorld, self).__init__(**super_args)

Notes:

• It is important to call the super constructor in the way shown so that consumers of the module may override parameters such as the module name

Implement some methods

At the minimum, an analysis module must implement the run() method:

def run(self, cxt):
	cxt.log.write("Hello World from:\n%s\n" % (platform.uname(),))
	cxt.log.flush()
#end run()

The run method is passed a context object derived from ThackTech.Pipelines.ModuleRunContext and contains references to the sample, logging utilities, and information regarding the pipeline state. All computational work should occur within the run() method or methods directly called by run().

Return Values

If the module generates any output files, these should be returned from the run() method as a list of ThackTech.Pipelines.FileInfo objects.

Alternatively, an older style method of returning output files from the module is supported by returning a dict with keys representing file role and values representing the file fullpath. This gets translated by the pipeline runtime into FileInfo objects using the appropriate module run context to generate FileContext objects with the role provided in the returned dict. This method is deprecated, and my be fully removed in future versions.

Module Failure

The pipeline runtime monitors the execution of the a module and reports failure of a module. Essentially any exceptions thrown by the module are caught by the runtime. The action taken by the runtime depends on the value of the critical attribute; failing modules marked as critical result in the runtime halting execution of the current pipeline, where as modules marked as not critical result in the runtime continuing execution of the pipeline and marking the failure as a warning. The default value for the critical attribute as defined by ThackTech.Pipelines.PipelineModule is False, but may be overridden in implementing classes. Further, the user may always mark a module as (not) critical. So, two points to consider: Module authors should raise some exception to signal failure of the module, and module consumers should be aware of the value for the critical attribute.

Declare Module Parameters

In all but the very simplest analysis modules, it is useful to declare parameters that affect how the module runs. PipelineModule's contain a collection of ModuleParameter objects that provide type-safe declaration of options for a given module. These parameters may also have values assigned at runtime from installed configuration files.

Module parameters should be declared by overriding the PipelineModule._declare_parameters() method. Within the method declaration, call the PipelineModule.add_parameter() method, passing a ModuleParameter object:

def _declare_parameters(self):
	self.add_parameter(ModuleParameter('samtools_path', str, 'samtools', desc="Path to samtools"))
#end _declare_parameters()

Here we define a string type parameter named samtools_path with a default value of 'samtools'. We can get the value of this parameter at runtime by using the ``PipelineModule.get_parameter_value()` method.

Other important information about ModuleParameters

• Value is type-safe, so on setting value, the passed value is coerced to the type set for the parameter
• Parameters can be nullable (i.e. accept None values)
• Parameters can define valid values (besides type) using the choices attribute

Declare Module Resolvers

Most modules will require some input that can only be determined at runtime, for example, a file that is created by a module earlier in the pipeline. To facilitate a declarative approach that decouples data from the computation, Modules offer resolvers are essentially named callables.

Resolvers should be declared by overriding the PipelineModule._declare_resolvers() method, and place in the body calls to PipelineModule._name_resolver().

def _declare_resolvers(self):
	self._name_resolver('alignments')
#end _declare_resolvers()

Within the run() method, one can retrieve the runtime result of the resolver by calling PipelineModule.resolve_input() with the resolver name and the current module run context.

Other Notes about Modules

• Set the number of processors dedicated to this module using the processors attribute.
• Set the criticalness of the module with the critical boolean attribute. Modules marked as critical will cause the entire pipeline to fail if the module fails, otherwise the module fault will be marked as a warning and pipeline will continue to execute.
• The critical and processors attributes may be passed to the module constructor.
• It may be sometimes useful to override the module name by passing a name kwarg to the constructor to differentiate multiple of the same module within a given pipeline.