Is it possible to copy part of a Treant?

[kain88-de]

On the cluster I may want to add several simulations and all analysis that take a long time to run in a datreant. But I don't necessarily want to have the complete datreant on my workstation all the time (because of space reason or other things).

Is it possible to only copy part of a datreant? Extra points if I only need to call rsync on a subfolder.

[dotsdl]

Yeah, the only thing that really defines a particular Treant is its state file. You can rsync the state file plus anything else within its tree anywhere else and it will behave the same. Note that for Groups, though, they clearly won't find their members if they don't get copied too.

Does that answer your question? I think it would be really cool if there was a convenient mechanism for copying Treants with rsync, but that's a little project of its own. If the dependencies required are deemed heavy it can become its own subpackage (datreant.rsync, datreant.replicate, datreant.duplicate...). What would be ideal is to be able to do something like:

import datreant.core as dtr
from datreant.duplicate import rsync

t = dtr.Treant('sprout')

# copy Treant elsewhere, but take only what we want from it by 
# feeding include/exclude Views with directories/files we want/don't want
rsync(t, dest='me@remote_cluster:~/treants/',
      include=t.glob('stuff*_I_want'),
      exclude=t.glob('stuff_I*dont_want'))

I found pyrsync, which might be a good starting point, but looks abandoned.

[kain88-de]

Yes it answers that it is theoretically possible. Practically is something different. So lets leave it open.

It would be easier if every subfolder in a Treant Folder would be itself a Treant Folder, then I could easily copy subfolders. But as I remember the last time I checked this wasn't the case. I'm not even sure this is something we want to have.

[dotsdl]

You can make a subfolder a Treant if you like. That's not only allowed, but encouraged. Treants can be nested:

import datreant.core as dtr

t = dtr.Treant('sprout')

for subdir in ('one/', 'two/', 'three/'):
    dtr.Treant(t[subdir])

This will make Treants out of the subdirectories sprout/one/, sprout/two/, etc., whether the subdirectories exist already or not.

[kain88-de]

That is pretty darn cool and would make things for me and colleagues easier in organizing our data. If this could be in the docs somehow that would be nice. With some more explanation as well.

If this is encouraged why aren't subfolders Treants by default?

[dotsdl]

That is pretty darn cool and would make things for me and colleagues easier in organizing our data. If this could be in the docs somehow that would be nice. With some more explanation as well.

I think I know a good place for it. Yeah, it should be more explicit that this is completely allowed and perhaps invaluable for certain applications. For reference, it's the basis of my proposed alternative for datreant/MDSynthesis#47.

If you have Treants within the immediate directory of a Treant (as in the example above), you can get them directly with Treant.treants:

>>> t = dtr.Treant('sprout')
>>> t.treants
<Bundle([<Treant: 'one'>, <Treant: 'two'>, <Treant: 'three'>])>

if you have Treants nested throughout 'sprout', you could use Treant.discover to grab them all, with options for how deep to go within found Treants or directories.