[ODSC-60499] Update validation for models registered from object storage

[VipulMascarenhas]

Description

This PR covers the following changes:

• Register api takes in a parameter download_from_hf, set to True by default. For API call via model_handler, this is set as False. For CLI command, this will be set to true by default. The changes for HF flow will be covered in a different PR.
• AquaModelSummary has a field name change, model_format to model_formats.
• Validation changes when registering model via object storage.

Request

POST http://localhost:8888/aqua/model

{
    "model": "google/gemma-2b-it",
    "os_path": "oci://<bucket>@<namespace>/<prefix>",
    "inference_container": "odsc-tgi-serving"
}

Response

{
    "compartment_id": "ocid1.compartment.oc1..<ocid>",
    "icon": "",
    "id": "ocid1.datasciencemodel.oc1.iad.<ocid>",
    "is_fine_tuned_model": false,
    "license": "gemma",
    "name": "google/gemma-2b-it",
    "organization": "Google",
    "project_id": "ocid1.datascienceproject.oc1.iad.<ocid>",
    "tags": {
        "aqua_service_model": "ocid1.datasciencemodel.oc1.iad.<ocid>",
        "license": "gemma",
        "task": "text_generation",
        "model_format": "SAFETENSORS",
        "organization": "Google",
        "aqua_custom_base_model": "true",
        "OCI_AQUA": "active",
        "ready_to_fine_tune": "true"
    },
    "task": "text_generation",
    "time_created": "2024-07-26 02:31:33.316000+00:00",
    "console_link": "https://cloud.oracle.com/data-science/models/ocid1.datasciencemodel.oc1.iad.<ocid>?region=us-ashburn-1",
    "search_text": "ocid1.datasciencemodel.oc1.iad.<ocid>,gemma,text_generation,SAFETENSORS,Google,true,active,true",
    "ready_to_deploy": true,
    "ready_to_finetune": true,
    "ready_to_import": false,
    "nvidia_gpu_supported": true,
    "arm_cpu_supported": false,
    "model_formats": [
        "SAFETENSORS"
    ],
    "model_card": "<content>",
    "inference_container": "odsc-tgi-serving",
    "finetuning_container": "odsc-llm-fine-tuning",
    "evaluation_container": "odsc-llm-evaluate",
    "artifact_location": "oci://<bucket>@<namespace>/<prefix>"
}

Unit Tests

Tests will be covered in a separate PR.

[kumar-shivam-ranjan]

If the container selected is odsc-llama-cpp-serving while registering and the artifact has only gguf files no safetensor, there shouldn't be check for config.json right? bt in the current implementation , it is doing and if there is no config.json found, it is throwing error.
Is this expected?

[VipulMascarenhas]

If the container selected is odsc-llama-cpp-serving while registering and the artifact has only gguf files no safetensor, there shouldn't be check for config.json right? bt in the current implementation , it is doing and if there is no config.json found, it is throwing error. Is this expected?

the exception wasn't handled correctly in that case, it's fixed now. thanks for catching this.

[VipulMascarenhas]

note: we're downloading to cached folder first and then to local folder. Need to test this two step download, most likely we just need to download to local dir.