Directory timestamps are not hashed

[mwaskom]

When an interface input is a directory (I encountered this with DataSink; it may be specific to that interface), it is treated as a string and only the name is used to generate the hash. This is problematic when the contents of the directory, but not the directory name, change. I can't track down the root code that does the hashing, but I can only assume that there is an

if os.path.isfile(s):
    ... hash file info
else:
    ... hash name

somewhere. It should be easy to add a directory check onto this code, I just don't know where it is. I'm also not totally sure what the right behavior should be when the config option for hashing says to use file contents.

[satra]

so i looked at this in a little more detail and now remembered why this was hard. here is even the code for adding to base traits class.

                elif (hash_files and isinstance(object, str) and
                        os.path.isdir(object)):
                    dirfiles = []
                    for path, _, files in os.walk(object):
                        for f in files:
                            dirfiles.append(os.path.join(path, f))
                    out = self._get_sorteddict(dirfiles, dictwithhash,
                                               hash_method=hash_method,
                                               hash_files=hash_files)

the problem is that we have to go in an make sure that the directory is not an input directory on the system like subjects_dir for the freesurfer interfaces.

the easier solution in this case might be to add a force=None flag to datasink. even if datasink reruns, it will only copy files that are not already on the output path.

[mwaskom]

I'm not sure I understand the workaround. Here's a more concrete example of where the problem arises:

• A node that does some processing at the level of contrasts and returns a directory with each contrast's file in it (say it's plotting zstats).
• After running once, you decide to add more contrasts to the experimental design. This triggers rerunning of the model (because the design spec has changed) and the plot_zstat node (because it's getting a list of filenames).
• This doesn't trigger rerunning of the DataSink node, which is just getting a directory path it's treating as a simple string. So the zstat plots for the new contrasts don't end up in the output directory.

[satra]

so force=True would force datasink to run always independent of it's hash.

def __init__(self, ..., force=None):
   ...
   if force:
       self._always_run = True

[mwaskom]

Did you mean force=True in the original message instead of force=None?

[satra]

i did mean force=None, but we can change the default to True and have people switch it to None, if they are comfortable.

[mwaskom]

Oh, I think I see. You meant add a flag force to DataSink with a default value of None, but where True would make it run and avoid the problem I described.

I thought we already had a general node attribute that enabled that kind of behavior? The name is currently escaping me ... overwrite maybe

Sorry, being dense today.

[satra]

yup node.overwrite will always rerun a node, but setting _always_run will do it at the interface level.

[djarecka]

@satra : didn't your PR #646 (changes in io.py) solve the issue? can we close it?

[djarecka]

@satra ping

[satra]

@djarecka - yup that fixes this issue. fixed by #646