DM-24352: Add new auto and link transfer modes #252
Conversation
The behavior of auto depends on the datastore.
auto will now fall back to this mode in posix datastore.
| return transfer | ||
| try: | ||
| # Assume first dataset is representative | ||
| self._pathInStore(datasets[0].path) |
There was a problem hiding this comment.
This seems like it could yield pretty surprising behavior if the assumption is incorrect. Better to check them all and disallow "auto" if the answer is inconsistent, even if that could be a lot of overhead. After all, it's always possible to avoid that overhead by being explicit about the transfer mode.
An alternative would be to just make in-place the default so it's an error if the file is outside. I'm slightly concerned that having symlinks be any kind of default (even the "default of last resort") could be bad, because it makes for a fragile repository.
| Raises | ||
| ------ | ||
| RuntimeError | ||
| Raised if the supplied path is outside the datastore. |
There was a problem hiding this comment.
Given that in at least one context this is a totally normal condition (as opposed to an error) it might be better to return None.
@DinoBektesevic in theory with this ticket you should be able to do a 2 to 3 conversion directly to S3.