copying multiple files at once

[isantana-bio]

I recently became aware of this python sdk. I need to copy 600-3000 files (300MB-100GB per file) to various NFS destinations at the same time on the same cluster about twice or so per week. Is there a method to be able to do this in parallel or/and is it ok to have multiple threads using the same api user ?

Also, I was told by our isilon IT that there is an issue with the clone feature. Can someone confirm?

[fr3man2]

Hi,
What about native synciq policy? Have you checked that?

[isantana-bio]

we have tried synciq, generally for copying entire folders, it requires source/target (?) read only and it's not possible in this case. Another complication is that for the case in question, source files are many and scattered across multiple folders, the same goes for their respective copy targets.

[fr3man2]

Read only lock is only for the final sync phase, while most of the replication makes only the destination read - only.

[isantana-bio]

Unfortunately, my sources/destinations can not be read only at any point. Other processes can be adding more files in those areas at the same time.

[nalinkanwar]

Closing as inactive; Please file this as a feature request/enhancement to OneFS using official support channels. This Issues page is for SDK related issues only.