Feature/multi model artifact handler

[YashPandit4u]

Adding a new class to easily handle multi-model artifact creation in ADS.

[oracle-contributor-agreement]

Thank you for your pull request and welcome to our community! To contribute, please sign the Oracle Contributor Agreement (OCA).
The following contributors of this PR have not signed the OCA:

• PR author: YashPandit4u

To sign the OCA, please create an Oracle account and sign the OCA in Oracle's Contributor Agreement Application.

When signing the OCA, please provide your GitHub username. After signing the OCA and getting an OCA approval from Oracle, this PR will be automatically updated.

If you are an Oracle employee, please make sure that you are a member of the main Oracle GitHub organization, and your membership in this organization is public.

[VipulMascarenhas]

left a few comments, I think we should be reusing most of the implementation from DataScienceModel class instead of recreating it in here. A high level design would look like:

class DataScienceModelCollection(DataScienceModel):

    def __init__():
        super().__init__(**kwargs)
        # no need to add model self.modeldescriptionjson since this attribute is already present in DataScienceModel
    
    def add():
        ...
        
    def remove():
        ...
        
    def show():
        ...     

The usage would look like:

    >>> model = (DataScienceModelCollection()
    ...    .with_compartment_id()
    ...    .with_project_id()
    ...    .with_display_name()
    ...    .with_description()
    ...    .with_freeform_tags()
    ...    .with_artifact("model1_path", "model2_path")

    >>> model.add(...)
    >>> model.remove(...)
    >>> model.create(model_by_reference=True)

On second thought, one other option is to have add(), remove() within DataScienceModel itself, then we don't need the collection class. The add/remove functions are only manipulating the model_file_description property that represents multiple models, so having a separate subclass may not be required.

any thoughts on this?

[VipulMascarenhas]

can we reuse the logger from ads.common?

[YashPandit4u]

When I used logger from ads.common I don't see any info messages and for some methods in this class I want to show info messages.

[VipulMascarenhas]

can we reuse is_path_exists from ads.common.utils instead?

[VipulMascarenhas]

ads.common.object_storage_details.list_object_versions can be used instead?

[VipulMascarenhas]

recommend using type hinting for all the functions, for example:
def add(self, namespace: str, bucket: str, ... ): -> None

[Abh1navKumar]

+1

[YashPandit4u]

Added type hints.

[VipulMascarenhas]

nit: use if display_name: instead?

[YashPandit4u]

The suggested statement will trigger even when we have blank string.

[Abh1navKumar]

+1

[mrDzurb]

I may have misunderstood what we're trying to achieve in this PR, but wouldn't a user experience like the following be more streamlined?

model_description = ModelDescription().add("oci://bucket_name@namespace/prefix1", "oci://bucket_name@namespace/person_prefix2")

Here, ModelDescription acts as a builder for the JSON configuration needed.
It still can contain all the introduced methods in this PR: add, remove, build, print...

I would prefer instead of .build to use .to_json(uri="path/to/the/json") where if user don't provide uri then JSON should be returned as a result of the function. Also adding .to_dict() might be useful as well.
In this case we don't need .show() method.

We could then enhance the DatascienceModel class with:

DatascienceModel().with_model_description(model_description).create()

This approach leverages the existing DatascienceModel class, which already supports CRUD operations for models, so we wouldn't need to replicate these in the ModelDescription class. The purpose of ModelDescription would be to simplify the preparation of the correct JSON configuration for users.

Perhaps we could consider a clearer name for clarity, as "ModelDescription" might be somewhat confusing. How about using "ModelArtifactConfigBuilder" to more accurately reflect its purpose as a tool for constructing model artifact configurations?

Then user experience would be

model_artifact_config = ModelArtifactConfigBuilder().add().add()

DatascienceModel().with_model_artifact(model_artifact_config).create()

cc: @mayoor @VipulMascarenhas

[YashPandit4u]

Hi @VipulMascarenhas ,

I have now extended from the class already present in ads and the usage of the utility looks like this now:

import sys
import ads.common
from ads.model.model_collection import DataScienceModelCollection
import ads
import json

ads.set_auth(auth="security_token")

mdes = DataScienceModelCollection()
.with_ref_model_id(model_ocid="ocid1.datasciencemodel.oc1.iad.amaaaaaav66vvnialtuoq5y4ltdz26bxhlow4uh5w544ullowe3jl4vwuayq")
.with_display_name("New Design 8")
.with_compartment_id("ocid1.tenancy.oc1..aaaaaaaahzy3x4boh7ipxyft2rowu2xeglvanlfewudbnueugsieyuojkldq")
.with_project_id("ocid1.datascienceproject.oc1.iad.amaaaaaav66vvniaqsyu2nufljutkn4rzth2nz4q3zqslirct7eayl5ojpma")

mdes.add(namespace="ociodscdev", bucket="unzip-multi-model", prefix="model-linear-1")
mdes.remove(namespace="ociodscdev", bucket="unzip-multi-model", prefix="model-linear-1")

mdes.create()

sys.exit(0)

Please take a look when free and give your comments.

Thanks and Regards
Yash Pandit

[mrDzurb]

Wouldn't this be better - logger = logging.getLogger(__name__)?

[mrDzurb]

Please double check if the existing utils methods are already implemented such functionality.

[VipulMascarenhas]

@YashPandit4u suggested a few changes, just needs to be re-organized to make it consistent with ads usage.

[VipulMascarenhas]

let's rename this to add_artifact(uri:str, files=Optional[List[str]]) which takes in uri and files list as input to be consistent with how paths are referred to within ads.

[YashPandit4u]

The user won't have uri handy, hence we have parameters like namespace, and bucket to make it easy.
Will it be possible to keep these parameters? The function signature would look like:

def add_artifact( self, namespace: str, bucket: str, prefix: Optional[str] = None, files: Optional[List[str]] = None, )

[VipulMascarenhas]

isn't the uri same as f"oci://{bucketName}@{namespace}/{prefix}", all of which are added as input to this function. My concern is that this input makes it incosistent with all other DataScienceModel or GenericModel operations in ads and we wouldn't be able to change it later on.
@mrDzurb what do you think?

[VipulMascarenhas]

Can we modify _prepare_file_description_artifact with optional params for files list and content json? This replaces the above functions check_if_file_exists and list_obj_versions_unpaginated altogether.

If content is passed, then no need to create the dict within that function.
for example:

if not content:
    content = dict()
    content["version"] = MODEL_BY_REFERENCE_VERSION
    ...

In this case _prepare_file_description_artifact would take in inputs like def _prepare_file_description_artifact(bucket_uri: list, content: Optional[Dict] = None, files: Optional[List[List[str]]] = None).

The inputs for files isn't ideal, it might be good to implement a dataclass for this and pass a List[ModelFileDescriptionArtifact] instead of bucket_uri list.

@dataclass
class ModelFileDescriptionArtifact:
    bucket_uri: str
    files: List[str] = None

This isn't urgent though, we can update this later on.

[YashPandit4u]

Thanks for this suggestion.
Will it be okay if we do this in the next PR due to customer deadlines?

[VipulMascarenhas]

sure, let's mark a todo somewhere to make sure of this.

[VipulMascarenhas]

same as add_artifact - rename this to remove_artifact(uri:str) which takes in uri to be consistent with how paths are referred to within ads.

[YashPandit4u]

Yes done.

[VipulMascarenhas]

this isn't needed since we can just the existing create method.

[YashPandit4u]

Yes removed this.

[VipulMascarenhas]

this is covered by model.model_file_description property.

[YashPandit4u]

Yes removing this.

[VipulMascarenhas]

@YashPandit4u thanks for making all the changes here, looks good overall. One concern I have is around the path inputs, it would be good to have the oci path like I mentioned in the comments above as user can construct the path easily with the params (bucket, namespace and prefix). Also, could you please add a couple unit tests in test_datascience_model.py for this change?

cc: @mrDzurb

[mrDzurb]

For consistency it would be better to accept OCIFS "oci://" path.

[YashPandit4u]

Have changed both functions add_artifact and remove_artifact with this uri type input.

[mrDzurb]

Technically the auth object is already constructed and accessible by

self.dsc_model.auth

[YashPandit4u]

Thanks, using this now.

[VipulMascarenhas]

@YashPandit4u thanks for the update. There are a few improvements to make for the next update:

• check_if_file_exists and list_obj_versions_unpaginated should reuse existing methods.
• _extract_oci_uri_components wasn't required, as mentioned in previous comments we could just use ObjectStorageDetails.from_path(uri) to get bucket name, namespace and prefix. If we want to add validation ObjectStorageDetails has a few static methods like is_valid_uri to do this.

[YashPandit4u]

Hi @VipulMascarenhas , thanks for telling the improvement items.

• I have now removed the "_extract_oci_uri_components" method and used the already present ObjectStorageDetails.from_path(uri) to get the namespace, prefix, and filepath. Thanks for the suggestion.

[VipulMascarenhas]

lgtm 👍

[YashPandit4u]

Hi @VipulMascarenhas ,
Based on the management decision, I have reverted the method signatures to the ones suggested by the customer.

[YashPandit4u]

Hi @VipulMascarenhas ,

I have now added options for both uri and (namespace, bucket, prefix) in add_artifact, remove_artifact.

[mrDzurb]

NIT: Could you please add an examples section for the add and delete artifacts feature?

[mrDzurb]

NIT: if not (namespace and bucket):

[mrDzurb]

NIT: Wouldn't it be better to say something like - Artifacts cannot be added. Both 'namespace' and 'bucket' parameters must be provided.. otherwise it would be hard for user to understand where they need to provide the namespace and bucket.

[mrDzurb]

Maybe we can do something like this?

if uri and (namespace or bucket):
    raise ValueError("Artifacts cannot be added. Please provide either a 'uri' alone, or 'namespace' and 'bucket' together, but not 'uri' with 'namespace' or 'bucket'.")

elif not (uri or (namespace and bucket)):
    raise ValueError("Artifacts cannot be added. You must provide either a 'uri' or both 'namespace' and 'bucket'.")

[mrDzurb]

NIT: I would not recommend to create nested functions, it would be hard to write unit tests for such methods. I would rather recommend to move such methods into utils or outside the class.

[mrDzurb]

NIT:

error_message = (
        f"No files found to add in the bucket '{bucket}' within the namespace '{namespace}' "
        f"and prefix '{prefix}'. Expected file names: {files}"
    )

[mrDzurb]

Please add the examples section.

[mrDzurb]

See the comments for the add_artifact method.

changes requested were implemented, pending items will be covered in subsequent update.