Inplace Import via SymlinkFileTransfer (See #11573) (rebased onto develop) #2020
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The uploadFile method has a number of responsibilities
that were not easily modifiable. Rather than insert
further `if (symlink) {} then {}` complications, its
been extracted behind the new `FileTransfer` interface
to permit others to inject their own logic.
The number of parameters passed to the `transfer` method was daunting at best. Further, much of the uploadFile method was doing logging so that moving the execution into the state class, the FileTransfer implementors can be ignorant of most of the arguments.
The rest of the reusable logic from uploadFile has now been moved to AbstractFileTransfar and re-used by the new SymlinkFileTransfer class. Several remote-access methods were needed in the ImportLibrary itself, though these could perhaps live in a more general gateway-style utility.
The following options are now all available for choosing which transfer method to use: ``` bin/omero import -t upload bin/omero import --transfer upload bin/omero import -t symlink bin/omero import -- -t symlink bin/omero import -t your.class.name.Here ```
A simple `mv` command is not possible since 1) the file must be present for subsequent readds and 2) it's not safe to remove the file until the import has completed successfully. Now, users of the `FileTransfer` API have the responsibility of calling `afterSuccess` once the `ImportLibrary` has done its work. An exception may be thrown if cleanup is not possible.
```
$ bin/omero import -- --transfer-help
...
Advanced arguments:
--transfer=ARG File transfer method
Examples:
-t upload # Default
-t ln # Use hard-link. Locally only!
-t ln_rm # Caution! Hard-link followed by rm. Locally only!
-t ln_s # Use symlnk. Locally only!
-t some.class.Name # Use a class on the CLASSPATH
ex. importer-cli --tranasfer=ln_s foo.tiff
Report bugs to <ome-users@lists.openmicroscopy.org.uk>
```
With this change, an annotation is added to the `Fileset` object on import describing which `FileTransfer` implementation was used. Use of the default `UploadFileTransfer` is ignored; otherwise, the FQN is stored as the content of the annotation. If we decided to move to subclasses of `Fileset` these values should suffice. Representative output after an import includes: ``` ome2=> \x ome2=> select discriminator, id, ns, textValue from annotation order by id desc limit 2; -[ RECORD 1 ]-+--------------------------------------------------- discriminator | /type/OriginalFile/ id | 3102 ns | openmicroscopy.org/omero/import/logFile textvalue | -[ RECORD 2 ]-+--------------------------------------------------- discriminator | /basic/text/comment/ id | 3101 ns | openmicroscopy.org/omero/import/fileTransfer textvalue | ome.formats.importer.transfers.SymlinkFileTransfer ```
2 tasks
|
--rebased-from #1989 |
|
As this is a rebase, I've just run through a simple example of each import and everything looks fine. Good to merge. |
joshmoore
added a commit
that referenced
this pull request
Jan 27, 2014
Inplace Import via SymlinkFileTransfer (See #11573) (rebased onto develop)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the same as gh-1989 but rebased onto develop.
https://trac.openmicroscopy.org.uk/ome/ticket/11573
This PR implements the same logic as the
InPlaceImporter.pyscript from gh-1929 but in Java. The only thing that is currently not possible is the remote invocation that a script provides. But considering there are security issues when using the script, this is the safer intermediate solution.To test:
As with gh-1929, the goal is that it's possible to import all files that were previously importable, but the files of the fileset under
/OMERO/ManagedRepositoryshould be symlinks not the original files. You will need to shell into the machine running OMERO, and import via the CLI:The key lines to look for in the output:
To do:
dev_5_0once RawFileStore.getFileId #1987 is merged. (No commit comments until then, please)Once this is merged, the GUI importer can be updated to also symlink. /cc @jburel