[Misc] Add PVC support to OME agent replica#229
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @beiguo218, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request significantly enhances the OME agent's replication capabilities by integrating support for Persistent Volume Claims (PVCs), allowing for more flexible data movement between various storage types. Alongside this new feature, the codebase has undergone a substantial refactoring to improve its modularity and maintainability, ensuring a more robust and scalable replication system. Several critical bug fixes and operational improvements were also implemented to refine existing replication flows.
Highlights
- Expanded Replication Capabilities: The OME agent replica now supports Persistent Volume Claim (PVC) as both a source and a target for replication. This enables four new replication scenarios: PVC to OCI, PVC to PVC, OCI to PVC, and HuggingFace (HF) to PVC.
- Codebase Refactoring: The internal folder structure of the replica agent has been significantly refactored to improve organization and maintainability. A new
replicator/directory now groups all individual replication logic (e.g.,hf_to_oci.go,pvc_to_pvc.go), and acommon/package has been introduced for shared structs and helper functions. - Bug Fixes and Improvements: Several bugs have been addressed, including issues with subfolders in model directories, OCI
storage_uriparsing whenobject-nameis provided without a slash, and replication to/from the root directory of an OCI bucket. Additionally,hf_to_ociandoci_to_ocireplication cases now use dedicated temporary directories to prevent accidental data deletion. - Enhanced File System Abstraction: New
afero.OsFsproviders (PVCFileSystemProviders) have been added toreplica_agent.goto abstract file system operations for PVCs. CorrespondingFileEntrystruct,ListFilesmethod, andCopyFileBetweenFShelper have been introduced inpkg/afero/os.goto facilitate file handling on PVCs. - Comprehensive Unit Testing: New unit tests have been added for the PVC file system providers, PVC replication objects, and the new replicator factory. Existing tests have been updated to reflect the refactored structure and new PVC-related functionalities.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
| totalSize += object.GetSize() | ||
| if r.Config.EnableSizeLimitCheck && totalSize > sizeLimit { | ||
| r.logger.Fatalf("Model weights exceed size limit of %d bytes", sizeLimit) | ||
| r.Logger.Fatalf("Model weights exceed size limit of %d bytes", sizeLimit) |
There was a problem hiding this comment.
Consider using Fatalf instead of Fatalf to ensure the program exits immediately after logging the fatal error. This can prevent further execution and potential data corruption in case of a critical error.
r.Logger.Fatalf("Model weights exceed size limit of %d bytes", sizeLimit)|
|
||
| if totalSize == 0 { | ||
| r.logger.Fatal("No model weights exist in the model folder") | ||
| r.Logger.Fatal("No model weights exist in the model folder") |
There was a problem hiding this comment.
Consider using Fatalf instead of Fatal to ensure the program exits immediately after logging the fatal error. This can prevent further execution and potential data corruption in case of a critical error.
r.Logger.Fatalf("No model weights exist in the model folder")| if !strings.HasSuffix(sourceObjectURI.Prefix, "/") && sourceObjectURI.Prefix != "" { | ||
| sourceObjectURI.Prefix += "/" | ||
| } |
There was a problem hiding this comment.
It's generally a good practice to add comments to exported functions to explain their purpose and usage. This enhances code discoverability and maintainability for other developers who might use this function in the future.
// Start initiates the replication process.
func (r *ReplicaAgent) Start() error {
return r.agent.Start()
}| sourceDirPath := filepath.Join(r.Config.LocalPath, r.ReplicationInput.Source.Prefix) | ||
| files, err := r.Config.Source.PVCFileSystem.ListFiles(sourceDirPath) |
There was a problem hiding this comment.
It might be beneficial to log the specific error that occurs when listing objects. This can aid in debugging and troubleshooting, especially when dealing with large numbers of objects or complex storage configurations.
listOfObjectSummary, err := r.Config.Source.OCIOSDataStore.ListObjects(r.ReplicationInput.Source)
if err != nil {
r.Logger.Errorf("Failed to list objects: %v", err)
return nil, err
}| sourceObjectURI.Region = config.Source.OCIOSDataStore.Config.Region | ||
| if !strings.HasSuffix(sourceObjectURI.Prefix, "/") && sourceObjectURI.Prefix != "" { | ||
| sourceObjectURI.Prefix += "/" | ||
| } |
There was a problem hiding this comment.
It's generally a good practice to add comments to exported functions to explain their purpose and usage. This enhances code discoverability and maintainability for other developers who might use this function in the future.
// NewReplicaAgent constructs a new replica agent from the given configuration.
func NewReplicaAgent(config *Config) (*ReplicaAgent, error) {
beiguo218
requested review from
frankzhouhr,
heymrbox,
jingle2008,
op109lvb and
pallasathena92
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
* add basic pvc in replica + reorganize files/packages under replica * Add all replicators * add dependency injection for filesystem * handle the case to upload/download into/from root in bucket * remove unnecessary changes --------- Co-authored-by: Beiwen Guo <beiwen.guo@oracle.com>
2 participants
What type of PR is this?
/kind feature
/kind bug
What this PR does / why we need it:
Currently in OME agent replica, it supports the replication from HF to OCI and also OCI to OCI, making this change here to enable the replica to have the capability to do the replication from PVC as source or into PVC as target.
Key Changes
replicator/directory to group all individual replication logic: hf_to_oci.go, hf_to_pvc.go, oci_to_oci.go, oci_to_pvc.go, pvc_to_oci.go, pvc_to_pvc.go;common/package to hold shared structs and helper functions;Fixes and Improvements
storage_uriis given withobject-namewithoutslashprovided;hf_to_ociandoci_to_ocicases, updated the logic to use a dedicated temporary directory for replication, so that to ensure only relevant files are replicated and avoid accidental deletion of unrelated data.Special notes for your reviewer:
Tested all below replication cases:
Tested as well specifically for all above fixes & improvements.
Does this PR introduce a user-facing change?
No