How to use SelectFiles to read all directories inside a folder?

[rattar1989]

Here is my data_organization function. This is used in the first node of my workflow.

def data_organization(in_path):
    do_something...
    return out_path

The function receives in_path as an input (a directory) which is a path of one subject folder. Then it organizes all the sub-folders in a desired way and it returns the path of the organized folder. Folders' name have a pattern. As SelectFiles finds files with specific patterns, is it possible to use it to find specific directories and pass them other nodes (e.g. to the above node)?

[mgxd]

Hi @rattar1989 -
You should be able to find directories as well as files, here's an example:

from nipype import SelectFiles
t = {'dirs': "/code/datasets/*/"}
f = SelectFiles(t)
f.run().outputs.dirs

However, at present there is no way to only select directories - this could be useful so I'm adding a PR for it.

[rattar1989]

Hi @mgxd
Thanks for your reply.
Yes, It would be very useful to have such a functionality.

[rattar1989]

Now how can I pass each element of the list (a directory) to next nodes?
If I do it by a for-loop it will affect the parallelization of the workflow?
The workflow must be executed in parallel.

[rattar1989]

I have done this to pass every single directory to next nodes:

# make a list of all directories: 
t = {'dirs': '/inputData/*/'}
sf = SelectFiles(t)
subject_list = sf.run().outputs.dirs
# subject_list will be like: ['/path/one', '/path/two', '/path/three', '/path/four']


# to iterate over the list of subjects in subject_list
infosource = Node(IdentityInterface(fields=['subject_id'], name="infosource"))
infosource.iterables = [('subject_id', subject_list)]

# SelectFiles to pass all directories to next node
templates = {'func': '{subject_id}/'}
selectfiles = Node(SelectFiles(templates), name="selectfiles")

Finally for nodes connection I have the following:

wf = Workflow(name='pipeline')
wf.connect([(infosource, selectfiles, [('subject_id', 'subject_id')]),
            (selectfiles, organization, [('func', 'in_path')])])
wf.run('MultiProc', plugin_args={'n_procs': 4})

I get the following error:

in <module>
    infosource = Node(IdentityInterface(fields=['subject_id'], name="infosource"))

raise ValueError('Identity Interface input is not in the fields: %s' % in_field)
ValueError: Identity Interface input is not in the fields: name

[mgxd]

@rattar1989 your IdentityInterface should be

infosource = Node(IdentityInterface(fields=['subject_id']), name="infosource")

[rattar1989]

@mgxd thanks for pointing out this issue. it solved the above error.
but the errors like the following appear:

IOError: No files were found matching func template: /tmp/tmpSpyJU3/pipeline/_subject_id_..home..ubuntu..PycharmProjects..data..InputData..11xxxxx..1101325/selectfiles//home/ubuntu/PycharmProjects/data/InputData/11xxxxx/1101325
Interface SelectFiles failed to run. 

I general, am I doing the right way of data input streaming?

[mgxd]

@rattar1989 I believe you can start connecting from the IdentityInterface, you don't need the second SelectFiles

[mgxd]

this was addressed in #2134