unifying output name generation

[satra]

This is based on souheil's question on the dev list to rename afni output properly.

some background:

in general for command line interfaces there should be two ways to generate an output name:

1. explicit specification:
   a) absolute path + name
   b) just name - should be created wherever the underlying interface wants it to - in case of workflow nodes this location might change because the input file may have been symlinked or copied to the local working directory.
2. automatic generation
   a) specify the input trait from which the output name should be derived.
   b) generate your own name

we used _gen_filename to generate output names for command line specs and _list_outputs for output traits. more recently the name_source parameter 2(a) above was introduced for the afni interfaces and this complicated things.

Furthermore, we need to link up what should be specified on the command line and what should be returned in the outputs. (e.g., os.path.exists cannot magically infer if echo00_bp exists as echo00_bp+orig.HEAD/BRIK)

here is a possible solution:

general idea is to pack as much into the specification and use generic routines - to be overridden when necessary (e.g. afni - removal of +orig).

• unify generation code. remove all _gen_filename and decorate existing traits that require gen_filename to use name_source metadata together with suffix

• decorate outputs with name_source as well to minimize list_outputs specification.

  so in trait inputs: out_file = File(..., name_source='in_file', suffix = '_brain')
  and in outputs:
  out_file = File(..., name_source=['out_file'])

this will require a major overhaul - but i feel we should do this soon. i'll send a starting PR over this weekend.

[inati]

Now that I've wrapped my mind around how you're doing this, the proposal seems reasonable. Looks like we should be overloading _filename_from_source. Should this be in AFNIBaseCommand or AFNICommand? Should we get rid of the _overload_ext definition in AFNICommand? What about _gen_filename in AFNIBaseCommand?

I'm not sure I understand how out_file is handled in the output_spec. For a .HEAD/.BRIK pair, is that supposed to be .HEAD or .BRIK or is it supposed to be a list with both of them in it?

AFNI programs can read NIFTI, NIFTI-PAIR, with or without compression. The can write them also, but I don't see a need support NIFTI-PAIR on the way out.

Some more comments about AFNI and compression:
If you define the environment variable AFNI_COMPRESSOR, typically to GZIP, although other choices are possible (see http://afni.nimh.nih.gov/pub/dist/doc/program_help/README.compression.html) then AFNI programs will automatically compress .BRIK and .nii on output (.BRIK.gz and .nii.gz). NIFTI files are also compressed if you say 3dXXX -prefix foo.nii.gz blah. BTW, why are there comments in the code about AFNI not having environment variables? That's definitely incorrect.

[satra]

Now that I've wrapped my mind around how you're doing this, the proposal seems reasonable. Looks like we should be overloading _filename_from_source. Should this be in AFNIBaseCommand or AFNICommand?

could you look to see if those two base classes can be merged? i suspect so.

Should we get rid of the _overload_ext definition in AFNICommand?

i think you probably need that, unless we make outputtype a property of all commandline interfaces

What about _gen_filename in AFNIBaseCommand?

this should go as per the above proposal.

I'm not sure I understand how out_file is handled in the output_spec. For a .HEAD/.BRIK pair, is that supposed to be .HEAD or .BRIK or is it supposed to be a list with both of them in it?

it's handled here:

https://github.com/nipy/nipype/blob/master/nipype/utils/filemanip.py#L264

[inati]

Oh dear. So it looks like what you have is an implicit list of known file types.

• single file (this captures .nii and .nii.gz and everything else)
• multifile nifti or analyze or spm (.hdr, .img, possibly with a .mat)
• afni (.HEAD and .BRIK)
• does this also capture a directory?

So yes, I think the right thing to do would be to have the spec tell me what I should be expecting. For example, if it's a single nifti file, then it has a name of foo.nii and a type of NIFTI, if it's compressed it would be foo.nii.gz and a type of NIFTI-GZ, if it's afni, then the name would be foo+orig (corresponding to two files) and the type could be AFNI if it's .HEAD and .BRIK or AFNI-GZ if it's .HEAD and .BRIK.GZ.

Alternatively a less intrusive change would be to make the AFNI type look for .HEAD (instead of .BRIK) and then put the two cases in filemanip.

How are the other file types handles? Is the free surfer stuff single files?

It's also worth remembering that nifti is a lowest common denominator format (which is why the extensions are there and why spm has .mat files). If you pass it from one package to another, all the stuff in the extensions is guaranteed to not be preserved.

[chrisgorgo]

I like the proposal and this is a long overdue refactor. One question: why would we have to specify name_source in both inputs and outputs?

[chrisgorgo]

@inati could you shed some light or point to a reference on what different "+orig" "+tlrc" suffixes mean and when are they used for autogenerated filenames?

[chrisgorgo]

Sorry I just saw your e-mail.

[chrisgorgo]

So depending on the space it will either "+orig" (for native) or "+tlrc" (for standard)? Any other options? Ideally we need to figure out how to get those suffixes without running the command itself.

[inati]

There are 3 possibilities +orig, +acpc, and +tlrc. It is up to the command line AFNI program to decide if it is going to change the "view" or not. This is why in AFNI commands the "prefix" does not include the view, because you shouldn't be specifying this yourself. Generally the view will only change when you are doing a registration. However the 3dRefit program can be used to modify the view. I don't think we should be in the business of specifying these from the wrapper.

Note: The view is encoded in the meta data in the .HEAD file, so if you specify the wrong view in the -prefix option your choice will be overridden.

On Jul 3, 2013, at 1:23 PM, Chris Filo Gorgolewski <notifications@github.commailto:notifications@github.com> wrote:

So depending on the space it will either "+orig" (for native) or "+tlrc" (for standard)? Any other options? Ideally we need to figure out how to get those suffixes without running the command itself.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/594#issuecomment-20431313.

[satra]

How are the other file types handles? Is the free surfer stuff single files?

freesurfer is single files

[inati]

Hi guys, I'm seeing some possible action on #617, which is closely related, but I'm sort of lost as to the big picture. Can we bring this closer to the front of the stove top?

[bpinsard]

I am not really experienced about the AFNI file formats, the idea in PR617 is to allow the use of name_source features in all the interfaces to avoid writing many _gen_filename methods with a list of if...elif...
As Satra proposed, a more general refactor than just fixing AFNI interfaces is the point.

Having name_source for interfaces OutputSpec would be great because many output filenames are automatically generated from input file with no choice to change it.
Also it seems that presently we specify the output files both in InputSpec and OutputSpec, and _gen_filename (+_list_outputs) handles this, but there might be interest in specifying it once with name_source?

[bpinsard]

re-reading the original issue, you mention using suffix, whereas currently name_source uses a format string 'prefix_%s_suffix' which allows far more flexible renaming (but there is still a bug with original file including a dot not for extension which can be overriden by specifying extension in name).
Also I wonder if the name_sourced file could include more complex renaming taking parameterization for example:

out_file = File('smoothed_%s_fwhm%.2f', name_source=['subject','fwhm'],output_type='NIFTI')

resulting in file 'smoothed_AMRO0010_fwhm5.00.nii'
to include other parameters with 'subject' from node parameterization (ie. iterables over subjects list) and fwhm being a numerical value as a node parameter.

What do you think?

[inati]

Moving comments over to #617.