Customizing location of flag files in the results_pipeline dir

[nsheff]

Right now, project.results_pipeline is used as the subdir within the project.output_dir. Looper then adds the sample name on to the end of this folder automatically, and then looks there for flags and summarizer files

The refgenie pipeline shows that we need more flexibility. We're setting up the pipeline to place flags and outputs in a much deeper folder (so we can exclude it from archive checksumming, etc). But there's no way to tell looper about this, and hence, flag checks and summarizers will never work with this pipeline.

This could be solved by making the 'results_pipeline' + {sample.name} hard-coding into a template that can be populated with sample properties, which should point directly to the output folder.

Related to pepkit/looper#242

[nsheff]

a thought: for cloud-based job execution, maybe it should even be possible to read flags on s3.

[nsheff]

This may actually be solvable now using the new paths section in pepkit/looper#285.

if the results pipeline is a templatable path, then it can be specified in the pipeline interface and looper will know where to look. I guess it would just be a reserved path key, like "looper_flag_path" or "pipestat_path" or something, and looper would know where to look if that path was provided.

[nsheff]

With the spinoff of the status flag system into pipestat, this is now a pipestat issue, really; the question is, how will pipestat status flags be saved? will the pipeline interface provide a way to specify the location of the flags?

[nsheff]

I think this might make sense to add as an attribute in the pipestat schema, maybe under the status components?

Or wait, would it make more sense to add to the looper pipeline interface (maybe I moved it from looper to pipestat prematurely...)

[donaldcampbelljr]

Currently, the user can pass flag_file_dir to the PipestatManager class during initialization or provide the directory in the config file.

Then, pipestat determines:

  flag_file_dir = self[CONFIG_KEY].priority_get(
      "flag_file_dir", override=flag_file_dir, default=os.path.dirname(self.file)
  )
  self[STATUS_FILE_DIR] = mk_abs_via_cfg(flag_file_dir, self.config_path)