Should the pipeline interface include path to sample yaml outputs?

[nsheff]

In #32 we build a pipeline interface section called summary_results, which records the location of summarizer results.

What about something similar to report the location of sample yaml file from the pipeline?

[stolarczyk]

I'm not sure I understand. Could you give more context here?

is the goal here to set the future location of sample.yaml file in a pipeline interface and then use this path to save the file there instead of in a default spot?

[nsheff]

well, this is a few years old... but yes I believe your interpretation is correct. I believe this is sort of accomplished by the output schema concept...

[stolarczyk]

more by the input schema, I think. {sample_name}.yaml file consists of key-value pairs of all public sample attributes, so input schema is related. Yet it just defines the type of the attrs, not their values.

[stolarczyk]

so what's the key for the sample yaml path in the pipeline interface, if we even want to proceed?

sample_yaml_path, sample_attrs_path, sample_path, sample_file_path, sample_file, sample_yaml?

[nsheff]

Well, there can be an input sample.yaml, which is in some sense an instance of the object specified by the input schema, and an output sample.yaml which is in some sense an instance of the object specified by the output schema.

we could produce both yamls. right now we only produce the first.
there could be:

pipelines:
  pipeline:
    input_sample_yaml_path: {sample.sample_name}.yam
    output_sample_yaml_path: {sample.sample_name}_output.yaml

those are relative to the path specified in looper.output_dir, with the above values as defaults, but can be overridden the pipeline interface?

just brainstorming here...

[stolarczyk]

I think it all makes sense.

So, in practice:
output sample yaml = input sample yaml + populated sample attrs defined in the pipeline output schema ?

[nsheff]

output sample yaml = input sample yaml + populated sample attrs defined in the pipeline output schema ?

makes sense to me... I think it's a superset of the input yaml. the only reason we make the input yaml is because it's used as an input to the pipeline.

In fact... why even make the input yaml? if the output yaml is a superset of it, then it could be used as an input to the pipeline as well...

so now, with this model, there is only 1 sample.yaml, which is exactly what you say: input yaml + populated sample attrs defined in the output schema.

one question: would this yaml include a property given in the table table that is not specified in the input schema?

[stolarczyk]

that's right, input sample.yaml is probably not necessary in such a case.

one question: would this yaml include a property given in the table table that is not specified in the input schema?

I'd say yes, I can imagine writing a small schema (for example with just required attrs), that does not necessarily cover all the sample attributes. Still I'd expect all my columns from sample table to be accessible in the yaml file.

[nsheff]

I'd say yes,

I agree.

[stolarczyk]

for future reference:

sample yaml file path is constructed from a template in pipeline interface file

pipelines:
  pipeline:
    sample_yaml_path: >
     {sample.sample_name}.yaml  # relative to looper.output_dir

if sample_yaml_path section is missing it is saved in submission directory in <sample_name>.yaml