irsync -r fails on symbolic links that create file system loops #3988
Comments
|
Current status: The current fix can detect the following errors:
In both of these cases, the utilitiy will terminate with an error message and a USER_INPUT_PATH_ERR error (this is a change in behavior). Previously the irsync and/or iput utility would work through these errors, and simply report a message if the error were actually detected (which they mostly weren't). What is currently not detected, are more sophisticated loops, for example - "irsync -r" of a physical directory which has nested directories, under which a symbolic link is pointing to an existing parent physical directory. The linux "find" utility WILL ALWAYS detect loops, and can be used as a work around, for example - before using "irsync -r" on a folder, let's say "some_folder", using the following command before using irsync, will detect symbolic link loops of this nature, allowing the user to abort the process and fix the file system: Not sure yet if it's irods' job to detect such loops - to be discussed. Still to finish: lots of test cases for iput and irsync that cover all of the above. |
|
Example of using find to find loops (taken from the directory where I'm testing). |
|
Status Update Much of the discussion above is no longer up to date. Please see here for the latest. The development work for this bug fix is mostly done, and features: For both "irsync -r" and "iput -r", a pre-scan (or preflight scan) is run on the icommand physical parameter directories that participate in the icommand operation (copying directories and files into the irods vault). The prescan detects the following issues with the directory/file heirarchy:
The scan operates on all the command line directories at one time -- in other words, if a symlink which exists under one of the parameter directories points to an existing directory which is within one of the other command line directories, it will be found as a file system loop. For most errors, the pre-scan does not terminate when it finds an error. Instead, all the directories are scanned if possible, and all the errors found are reported to stderr, and the icommand (irsync or iput) terminates right after the scan, without making any changes to the catalog or transfering files to the irods vault. The only exception is the target collection which is created immediately, but will be empty after the icommand exits as a result of errors found during the scan. The other exception to the behavior described above is the case where one of the directories being scanned does not provide enough authority for the icommand to access it. When this is detected, the scan is aborted with the following error message: and the icommand exits. Error messages: Either/both utilities will display the following error messages for issues found during the scan:
Performance Cost The pre-scan is extremely inexpensive - much less than 1% of the total time it takes the icommand to run in most cases. Please see the next comment below for some performance numbers. A note about issue #3997: As a result of this issue, under certain circumstances, iput and irsync will terminate with an error when there is more than one physical directory participating in an operation into a collection. For example: If the tmp collection does not yet exist in the catalog, the following error will appear: As a workaround, the collection can be created before the icoomand is issued: This will avoid these errors. This will disappear once issue #3997 is fixed. |
|
Performance Impact of the Pre-Scan: For each of the directories listed below, a count of files, directories, disk space taken, and the scan duration is reported, followed by the output of the time utilitiy while running the iput command. This was run on a Ubuntu 16.04 system with an Intel i7-3770 CPU @ 3.40GHz × 8, and with 24Gb of RAM: dir_9.4M: Note: The pre-scan took 0.04854 seconds out of 1m36.153s for the complete iput -r operation dir_19M: Note: The pre-scan took 0.09436 seconds out of 1m40.813s for the complete iput -r operation dir_45M: Note: The pre-scan took 0.2414 seconds out of 2m28.738s for the complete iput -r operation dir_69M: Note: The pre-scan took 0.3872 seconds out of 8m57.829s for the complete iput -r operation |
|
Status update: currently writing test cases for our Continuous Integration testing environment. The bug fixes and enhancements associated with this issue are complete. |
[irods#4009] iput/irsync: Fixed bad logical path created for symbolic links
|
CI build 1156 for 4-2-stable pull-request - passed |
Cherry-picked from bb8415b
Cherry-picked from a8666a9
[#4009] iput/irsync: Fixed bad logical path created for symbolic links
|
CI Build 1165 (4-2-stable) and 1168 (master) passed. |
|
merged - please update checkmarks and close |
Bug Report
irsync -r fails on symbolic links that create file system loops. In some cases the collection path names created are so wide as a result of a loop, that they are difficult to remove.
(see output from D. Hu below)
This issue includes dealing with similar issues with iput -r.
iRODS Version, OS and Version
D. Hu - irods 4.2.2
Reproduced at irods on 4-2-stable/Ubuntu 16.04
Description and process to reproduce.
From D. Hu's email from 6/18/2019:
Related Issues: #3995, #3997, #4006, #4008, #4009, #4013, #4016
The text was updated successfully, but these errors were encountered: