Ensure destination directory exists + handle index mount directories #167
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change un-reverts PR #161 with a couple twists: account for index mount directories and use mpirun to perform the actions. Logic was added to prepare the destination directory by performing a `mkdir -p` on the user supplied destination. This logic ensures that the correct pathing is performed by determining the filetype (dir vs file) on both the source and destination paths. In addition to the supplied destination path, if the source filesystem is determined to have index mount directories (i.e. xfs/gfs2), those directories are created on the destination path to ensure that all data is retained. - All necessary filesystem interaction is ran through mpirun so that the proper mpi worker nodes are used to prepare the destination. This required for lustre2lustre data movement. - These file system interactions (i.e. `stat`, `mkdir`) are performed through `setpriv` to ensure that all interactions are done as the supplied UID/GID (from the workflow). - The hostfile createion logic now standsalone since these new mpirun calls need the hostfile before the final data movement command is built - Added in some local system calls when using unit tests since mpirun is not available in textenv Signed-off-by: Blake Devcich <blake.devcich@hpe.com>
roehrich-hpe
reviewed
Apr 9, 2024
Signed-off-by: Blake Devcich <blake.devcich@hpe.com>
Signed-off-by: Blake Devcich <blake.devcich@hpe.com>
roehrich-hpe
approved these changes
Apr 9, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change un-reverts PR #161 with a couple twists: account for index mount directories and use mpirun to perform the actions.
Logic was added to prepare the destination directory by performing a
mkdir -pon the user supplied destination. This logic ensures that the correct pathing is performed by determining the filetype (dir vs file) on both the source and destination paths. In addition to the supplied destination path, if the source filesystem is determined to have index mount directories (i.e. xfs/gfs2), those directories are created on the destination path to ensure that all data is retained.stat,mkdir) are performed throughsetprivto ensure that all interactions are done as the supplied UID/GID (from the workflow).Tested via https://github.com/bdevcich/dm-system-test