[Misc] replica ome-agent: add support to replicate models from HF to OCI

[beiguo218]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Currently the replica OME agent, it only supports replication for model from OCI to OCI, making this change here to enable replica with the capability to do replication from HF directly to OCI.

Key additions

1. Added support to replicate the required model from a HF repo to a OCI Object Storage bucket;
2. Updated the way how ome-agent configuration structures the values so that makes its config clearer when supporting more sources and targets, and also more intuitive when injecting config into the corresponding struct;
3. Implemented a factory pattern in replica ome-agent to simplify the creation of replication strategies and support future extension for additional storage backends;
4. Added/Fixed corresponding unit tests

Special notes for your reviewer:

Tested in my local for the below scenarios:

1. Replication from OCI to OCI
2. Replication from HF repo to OCI
   1). Replicating models requiring auth token;
   2). Replicating from non-main revision to OCI;
   3). Replicating models with sub-folder.

Does this PR introduce a user-facing change?

Customers now using replica OME agent are able to have their models in HF repo to be replicated to their OCI Object Storage data store.

[gemini-code-assist]

Summary of Changes

Hello @beiguo218, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the ome-agent's replication capabilities by introducing support for Hugging Face as a source for model replication to OCI Object Storage. It achieves this through a substantial refactoring of the internal architecture, including a new factory pattern for replication strategies, a more flexible configuration schema using storage URIs, and a unified abstraction for handling objects from diverse sources. The changes also include comprehensive updates to unit tests and minor cleanups across related modules.

Highlights

• New Feature: Hugging Face to OCI Replication: The replica OME agent now supports replicating models directly from Hugging Face repositories to Oracle Cloud Infrastructure (OCI) Object Storage buckets, expanding its capabilities beyond OCI-to-OCI replication.
• Architectural Refactoring: Factory Pattern for Replicators: A new factory pattern (NewReplicator) has been introduced to dynamically select and instantiate the appropriate replication strategy (e.g., HF-to-OCI, OCI-to-OCI) based on the configured source and target storage types. This makes the system more extensible for future storage backends.
• Configuration Enhancement: Unified Storage URIs: The configuration structure for source and target storage has been updated to use a more generic storage_uri (e.g., oci://..., hf://...) and nested, type-specific configuration blocks. This provides a clearer and more intuitive way to define storage locations.
• Code Abstraction: ReplicationObject Interface: A new ReplicationObject interface has been introduced, along with concrete implementations for OCI Object Storage summaries and Hugging Face repository files. This provides a unified way to handle and process objects regardless of their source, simplifying replication logic.
• Improved Configuration Validation: New validation logic has been added to ensure that required dependencies (like OCI data store clients or Hugging Face clients) are present based on the configured storage types, preventing runtime errors due to misconfigurations.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

The code changes introduce the ability to replicate models, including from Hugging Face to OCI. I've identified a few areas for improvement, primarily concerning error handling and robustness. The most critical issues relate to the use of panic for configuration errors, which could lead to unexpected application crashes. I've also provided suggestions to make the replication process more robust.

[gemini-code-assist]

Using context.Background() is not ideal for an operation like downloading a model, which can be long-running. This makes the operation non-cancellable and prevents timeouts from being enforced by the caller. It's recommended to propagate a context.Context from a higher-level caller (e.g., from ReplicaAgent.Start()) to allow for better control over the execution.

downloadPath, err := config.Source.HubClient.SnapshotDownload(
		context.TODO(),
		input.source.BucketName,
		config.LocalPath,
		downloadOptions...,
	)

[beiguo218]

Will resolve later in a separate PR.