BYOM Model support #812

mayoor · 2024-04-29T22:54:06Z

Bring your own model Support

This support entails support for "any" huggingface model on AI quick actions. There are following scenarios that needs to be addressed to facilitate this feature -

User chooses a huggingface model that has been already tested and certified on AI Quick Actions by the service. To support such models, the user need not specify which runtime container is required for inference or finetuning
User chooses a model that has not been certified, but user chooses from the list of readymade runtimes from the service. To support such models, the user will specify the service managed container name as available on the documentation. The documentation will list out the key libraries and their versions across different service managed container so that the user can choose the right image.
Use chooses a model that has no supporting container in service managed container list. The user builds the inference/finetuning container in their tenancy and provides the the container URI while registering the model

The PR also brings in a notion of "regstering" model. What it means is that user is importing the huggingface model into model catalog within user tenancy.

Assumptions

Model registration requires internet connection. It is assumed that the surface from where the user "registers" the model has internet connection
For the gated model model user will have authorized huggingface token.

Huggingface token setup

Run huggingface-cli login command and follow the on screen instruction

Usage

Scenario 1[Verified Model]- ads aqua model register --model meta-llama/Meta-Llama-3-8B --os_path oci://mayoor-dev-versioned@namespace/cached-models --local_dir `pwd`/cache-models
Scenario 2[Unverified Model with SMC] - ads aqua model register --model meta-llama/Meta-Llama-3-8B --os_path oci://mayoor-dev-versioned@namespace/cached-models --local_dir `pwd`/cache-models --odsc-vllm-container --inference_container_type_smc --finetuning_container odsc-finetuning-llm-container --finetuning_container_type_smc
Scenario 3[Unverified Model with Custom Container] - ads aqua model register --model meta-llama/Meta-Llama-3-8B --os_path oci://mayoor-dev-versioned@namespace/cached-models --local_dir `pwd`/cache-models --inference_container iad.ocir.io/my/custom:1.0 --finetuning_container iad.ocir.io/my/custom-ft:1.0

TODO

Test cases.

github-actions · 2024-04-29T22:54:20Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

mrDzurb · 2024-04-30T00:35:00Z

ads/aqua/model.py

+            else:
+                break
+        os.makedirs(local_dir, exist_ok=True)
+        snapshot_download(


could you add some details here, why do we need to download the snapshot again?

The first download saves the model to local hf cache (not local_dir). This is resumable in case there’s something wrong with the internet. The second download is copying from hf cache to local_dir. Downloading to local_dir is not resumable based how it works but copying is unlikely to have errors.

Got it, i missed this part - local_dir=local_dir.

From HF:

If `local_dir` is provided, the file structure from the repo will be replicated in this location. When using this option, the `cache_dir` will not be used and a `.huggingface/` folder will be created at the root of `local_dir` to store some metadata related to the downloaded files. While this mechanism is not as robust as the main cache-system, it's optimized for regularly pulling the latest version of a repository.

Also looks like local_dir_use_symlinks is deprecated argument?

if local_dir_use_symlinks != "auto": warnings.warn( "`local_dir_use_symlinks` parameter is deprecated and will be ignored. " "The process to download files to a local folder has been updated and do " "not rely on symlinks anymore. You only need to pass a destination folder " "as`local_dir`.\n" "For more details, check out https://huggingface.co/docs/huggingface_hub/main/en/guides/download#download-files-to-local-folder." )

Looks like this deprecation came only 2 days ago - huggingface/huggingface_hub#2223
It will depend on what version of hub api will carry this code.

mrDzurb · 2024-04-30T00:45:47Z

ads/aqua/utils.py

+            f"Error uploading the object. Exit code: {e.returncode} with error {e.stdout}"
+        )
+
+    print(os_details)


Probably used for debug purpose?

mrDzurb · 2024-04-30T00:48:13Z

pyproject.toml

@@ -116,6 +116,7 @@ opctl = [
  "rich",
  "fire",
  "cachetools",
+  "huggingface_hub"


Will we need to get approval for this?

We need to.

mrDzurb · 2024-04-30T03:09:12Z

I've been thinking whether we should allow users to specify the VLLM/TGI version as alternative to the container name, and have our system automatically select the appropriate container based on that input. If a user specifies just the VLLM/TGI interface without the specific version, we could default to the latest container that supports VLLM/TGI.

ads aqua model register --model meta-llama/Meta-Llama-3-8B  --os_path oci://bucket@namespace/cached-models --local_dir `pwd`/cache-models --interface vllm

Additionally, I think it's important to offer users the ability to specify different containers for inference and fine-tuning when creating deployments and fine-tuning. When users register models and specify containers, these can be set as default for deployment and fine-tuning, however given that containers may become obsolete really quick, particularly with regular security updates, users still will have an option to override containers.

VipulMascarenhas · 2024-04-30T16:08:53Z

ads/aqua/deployment.py

+                container_type=container_type_key,
+            )
+            if not is_custom_container
+            else container_type_key


clarifying: for SMC container, container_type_key is odsc-vllm-serving, whereas for byoc container, it will be something like <region>.ocir.io/<namespace>/user-provided-container:1.0.0.0?

That is right

VipulMascarenhas · 2024-04-30T16:31:29Z

ads/aqua/model.py

+        os_path,
+        model_name: str,
+        inference_container,
+        finetuning_contianer,


nit: replace finetuning_contianer with finetuning_container

VipulMascarenhas · 2024-04-30T16:36:18Z

ads/aqua/model.py

+            str: Display name of th model (This should be model ocid)
+        """
+        api = HfApi()
+        model_info = api.model_info(model_name)


will raise RepositoryNotFoundError if model_name is isn't available.

VipulMascarenhas · 2024-04-30T16:40:54Z

ads/aqua/model.py

+            filter_tag = Tags.AQUA_FINE_TUNED_MODEL_TAG.value
+        elif model_type == MODEL_TYPE.BASE.value:
+            filter_tag = Tags.BASE_MODEL_CUSTOM.value
+        print(filter_tag)


use logger.debug instead?

removed, was unintentional commit

VipulMascarenhas · 2024-04-30T16:41:49Z

ads/aqua/model.py

@@ -59,6 +67,11 @@ class FineTuningMetricCategories(Enum):
    TRAINING = "training"


+class MODEL_TYPE(Enum):


nit: use camel case, i.e. ModelType to stick with convention?

VipulMascarenhas · 2024-04-30T16:45:28Z

ads/aqua/model.py

+            os_path=os_path, local_dir=local_dir, model_name=model
+        )
+        # Create Model catalog entry with pass by reference
+        return self._create_model_catalog_entry(


after registering, can we return an AquaModel object instead of just returning the display name? User can refer to info within that returned result to proceed with next steps (deploy, FT, etc.)

I was thinking of returning model id, maybe AquaModel is better.

VipulMascarenhas · 2024-04-30T16:50:52Z

ads/aqua/model.py

+                break
+        os.makedirs(local_dir, exist_ok=True)
+        snapshot_download(
+            repo_id=model, local_dir=local_dir, local_dir_use_symlinks=False


if model size exceeds local_dir, download can be interrupted. Can we check repo metadata first before downloading?

Will add this to TODO. Can we be done in the a separate PR.

github-actions · 2024-04-30T22:53:41Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2024-04-30T23:13:59Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2024-04-30T23:21:59Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2024-05-01T17:13:05Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

VipulMascarenhas

approving this PR, we can address todos in subsequent updates.

github-actions · 2024-05-01T22:07:14Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2024-05-03T20:44:35Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

Externalize container configuration for deployment

github-actions · 2024-05-03T20:59:57Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

BYOM Model support

f7bd2ff

mayoor requested review from darenr, mrDzurb, VipulMascarenhas and qiuosier as code owners April 29, 2024 22:54

oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Apr 29, 2024

darenr approved these changes Apr 29, 2024

View reviewed changes

mrDzurb reviewed Apr 30, 2024

View reviewed changes

VipulMascarenhas reviewed Apr 30, 2024

View reviewed changes

remove debug prints

be466fe

Add comments

8b30e80

handle connection error

164469b

update license information

5fe16b2

VipulMascarenhas approved these changes May 1, 2024

View reviewed changes

Add support for shadow model

27e12c8

Externalize container configuration for deployment

61563ca

Externalize container configuration for deployment

mayoor force-pushed the feature/byom branch from f5a367e to 61563ca Compare May 3, 2024 20:59

VipulMascarenhas approved these changes May 3, 2024

View reviewed changes

mayoor merged commit 10d9393 into feature/aquav1.0.2 May 3, 2024
3 checks passed

mayoor added a commit that referenced this pull request May 6, 2024

BYOM Model support (#812)

1ef5883

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BYOM Model support #812

BYOM Model support #812

mayoor commented Apr 29, 2024 •

edited

github-actions bot commented Apr 29, 2024

mrDzurb Apr 30, 2024

qiuosier Apr 30, 2024

mrDzurb Apr 30, 2024 •

edited

mayoor Apr 30, 2024

mrDzurb Apr 30, 2024

mayoor Apr 30, 2024

mrDzurb Apr 30, 2024

mayoor Apr 30, 2024

mrDzurb commented Apr 30, 2024

VipulMascarenhas Apr 30, 2024

mayoor Apr 30, 2024

VipulMascarenhas Apr 30, 2024

mayoor Apr 30, 2024

VipulMascarenhas Apr 30, 2024

VipulMascarenhas Apr 30, 2024

mayoor Apr 30, 2024

VipulMascarenhas Apr 30, 2024

VipulMascarenhas Apr 30, 2024

mayoor Apr 30, 2024

VipulMascarenhas Apr 30, 2024

mayoor May 1, 2024

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

github-actions bot commented May 1, 2024

VipulMascarenhas left a comment

github-actions bot commented May 1, 2024

github-actions bot commented May 3, 2024

github-actions bot commented May 3, 2024

		@@ -59,6 +67,11 @@ class FineTuningMetricCategories(Enum):
		TRAINING = "training"


		class MODEL_TYPE(Enum):

BYOM Model support #812

BYOM Model support #812

Conversation

mayoor commented Apr 29, 2024 • edited

Bring your own model Support

Assumptions

Huggingface token setup

Usage

TODO

github-actions bot commented Apr 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrDzurb Apr 30, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrDzurb commented Apr 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

github-actions bot commented May 1, 2024

VipulMascarenhas left a comment

Choose a reason for hiding this comment

github-actions bot commented May 1, 2024

github-actions bot commented May 3, 2024

github-actions bot commented May 3, 2024

mayoor commented Apr 29, 2024 •

edited

mrDzurb Apr 30, 2024 •

edited