Issue 47144: Handle large files on Panorama Public via Symlinks #344
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…fb_copy_vs_mv_pano_files
…tion if any of the datafileurls could not be fixed. - PanoramaPublicSymlinkManager.moveAndSymLinkDirectory takes a Logger parameter so the log output can go to the job log - Added a PanoramaPublicMetadataImporter. I moved some of the code out of CopyExperimentFinalTask into this class. This creates a row in the panoramapublic.experimentannotations table. It runs before PanoramaPublicFileImporter so that if there is an error, e.g. datafileurls cannot be fixed, the container can be deleted to move files back to the source container. - Updated test - import a document into a subfolder of the container file root.
… renamed / deleted is in the Panorama Public project. We don't expect folders in other projects to contain symlink targets. - When handling folder rename (ContainerListener.propertyChange), pass the full paths of the old and renamed containers instead of just the folder names. Otherwise, it can lead to updating all symlinks that have the old folder name in the path. - When deleting a folder, use ExperimentAnnotationsManager.getExperimentIncludesContainer(c) to lookup the experiment. This method will return the experiment that contains runs from the folder even if it is a subfolder of the folder where the experiment was created. - When an experiment folder in Panorama Public is deleted, move the files back to next highest experiment version if one exists. Otherwise, move the files back to the source folder.
…s should only include the source container in the submitter's project as well as any containers with older versions of the data on Panorama Public.
This was referenced May 20, 2023
…fileUrls. This should not be required anymore due to LabKey/targetedms#724. Set filePathRoot on the copied expRun to be the target container's file root. Log error if the data file path is unexpected, i.e. it does not contain "Run<runid>"
labkey-jeckels
approved these changes
May 30, 2023
Contributor
labkey-jeckels
left a comment
labkey-jeckels
left a comment
There was a problem hiding this comment.
This is too big to do a full validation just via code review. I know you've been working closely with Vagisha so I think there's good coverage on the higher level functionality, but let me know if there's anything in particular that warrants extra review on my end.
panoramapublic/src/org/labkey/panoramapublic/PanoramaPublicFileImporter.java
Outdated
Show resolved
Hide resolved
panoramapublic/src/org/labkey/panoramapublic/PanoramaPublicFileImporter.java
Show resolved
Hide resolved
panoramapublic/src/org/labkey/panoramapublic/PanoramaPublicFileListener.java
Outdated
Show resolved
Hide resolved
panoramapublic/src/org/labkey/panoramapublic/PanoramaPublicFileListener.java
Outdated
Show resolved
Hide resolved
panoramapublic/src/org/labkey/panoramapublic/PanoramaPublicListener.java
Outdated
Show resolved
Hide resolved
panoramapublic/src/org/labkey/panoramapublic/PanoramaPublicMetadataImporter.java
Outdated
Show resolved
Hide resolved
panoramapublic/src/org/labkey/panoramapublic/PanoramaPublicSymlinkManager.java
Outdated
Show resolved
Hide resolved
…tener.java Co-authored-by: Josh Eckels <jeckels@labkey.com>
… and target folder
…fb_copy_vs_mv_pano_files
…bKey/MacCossLabModules into 23.3_fb_copy_vs_mv_pano_files
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale
Move instead of copying files to Panorama Public.
https://www.labkey.org/Panorama%20Partners/Feature%20Requests/issues-details.view?issueId=47144
Related Pull Requests
Changes