Reject asset upload with invalid `id` #273

JAulet · 2021-12-01T22:31:03Z

Excerpt from issue #209:

The actual id of the dataset asset in MLX was codenet-langclass. Notice the dash - which correctly replaced the underscore _ character. However, the component which generated the DataShim metadata used the value of the id field from the YAML file, with the id containing the underscore character.

Proposed fix:

Any _upload_XXX method in api/server/swagger_server/controllers_impl/XXX_service_controller_impl.py which takes an id (or similar field like model_identifier) from the uploaded YAML file to generate the asset id should call a new validate_id() method similar to the swagger_server.controllers_impl.validate_parameters function in api/server/swagger_server/controllers_impl/__init__.py:

api/server/swagger_server/controllers_impl/:

__init__.py, add validate_id(id: str) method
dataset_service_controller_impl.py, line 372: def _upload_dataset_yaml(...) ... use validate_id(id: str)
model_service_controller_impl.py, line 381: def _upload_model_yaml(...)... use validate_id(id: str)
notebook_service_controller_impl.py, line 392: def _upload_notebook_yaml(...) ... use validate_id(id: str)

Resolves #209

Signed-off-by: JAulet <51500017+JAulet@users.noreply.github.com>

model_service_controller_impl.py. Signed-off-by: JAulet <51500017+JAulet@users.noreply.github.com>

ckadner

Thanks @JAulet -- the original problem was not that the id we generate was invalid, but the id field inside the YAML could still be invalid. And since that YAML itself was used by another component to get the id the DataShim logic failed. So this PR should add some code to verify the ids in the YAML adhere to the k8s identifier format.

ckadner · 2022-02-15T22:45:36Z

api/server/swagger_server/controllers_impl/notebook_service_controller_impl.py

@@ -396,6 +396,11 @@ def _upload_notebook_yaml(yaml_file_content: AnyStr, name=None, access_token=Non
    requirements = yaml_dict["implementation"]["github"].get("requirements")
    filter_categories = yaml_dict.get("filter_categories") or dict()

+    errors, status = validate_id(notebook_id)


let's move this check above line 391 where we generate the notebook_id and validate the yaml_dict.get("id")

errors, status = validate_id(yaml_dict.get("id"))

ckadner · 2022-02-15T22:46:59Z

api/server/swagger_server/controllers_impl/model_service_controller_impl.py

@@ -397,6 +397,11 @@ def _upload_model_yaml(yaml_file_content: AnyStr, name=None, existing_id=None):
    if type(api_model.servable_tested_platforms) == str:
        api_model.servable_tested_platforms = api_model.servable_tested_platforms.replace(", ", ",").split(",")

+    errors, status = validate_id(api_model.id)


let's move this check to line 375 after we load the yaml and before we create the ApiModel object and validate the model_def.get("model_identifier")

errors, status = validate_id(model_def.get("model_identifier"))

ckadner · 2022-02-15T22:48:16Z

api/server/swagger_server/controllers_impl/dataset_service_controller_impl.py

@@ -371,6 +371,11 @@ def _upload_dataset_yaml(yaml_file_content: AnyStr, name=None, existing_id=None)
    # if yaml_dict.get("id") != dataset_id:
    #     raise ValueError(f"Dataset.id contains non k8s character: {yaml_dict.get('id')}")

+    errors, status = validate_id(dataset_id)


let's move this check above line 366 after we load the YAML and before we generate the dataset_id and validate the yaml_dict.get("id")

errors, status = validate_id(yaml_dict.get("id"))

call to check the id of the yaml file itself rather than generated id. Signed-off-by: JAulet <51500017+JAulet@users.noreply.github.com>

JAulet · 2022-02-18T21:38:34Z

@ckadner Please Review.

ckadner · 2022-02-18T23:14:57Z

@ckadner Please Review.

@JAulet -- did you try this out by uploading a model, dataset, and notebook YAML respectively with invalid ID values (and with valid ones)?

JAulet · 2022-02-22T00:58:48Z

@ckadner Please Review.

@JAulet -- did you try this out by uploading a model, dataset, and notebook YAML respectively with invalid ID values (and with valid ones)?

Yes, I did. I got the correct output for each.

ckadner

/lgtm

Thanks @JAulet

mlx-bot · 2022-02-24T18:48:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ckadner, JAulet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ckadner]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* Added validate-id to __init__.py * Added check to notebook_service_controller_impl.py * Added validate-id check to dataset_service_controller_impl.py and model_service_controller_impl.py. * Moved validate_id check to before id is generated and changed function call to check the id of the yaml file itself rather than generated id. Resolves machine-learning-exchange#209 Signed-off-by: JAulet <51500017+JAulet@users.noreply.github.com> Signed-off-by: Krishna Kumar <krishnakumar@ibm.com>

Added validate-id to __init__.py

5dbf8a6

Signed-off-by: JAulet <51500017+JAulet@users.noreply.github.com>

mlx-bot added the do-not-merge/work-in-progress label Dec 1, 2021

mlx-bot requested review from ckadner and yhwang December 1, 2021 22:31

Added check to notebook_service_controller_impl.py

d3c2a59

Signed-off-by: JAulet <51500017+JAulet@users.noreply.github.com>

ckadner removed the request for review from yhwang December 2, 2021 00:04

ckadner assigned JAulet Dec 2, 2021

ckadner changed the title ~~[WIP] Added validate-id to __init__.py~~ [WIP] Reject asset upload with invalid id Dec 2, 2021

ckadner added the RCOS Potential work items for RCOS student interns label Dec 2, 2021

JAulet added 3 commits January 27, 2022 14:02

Merge branch 'machine-learning-exchange:main' into vald-id-only

0da1604

Merge branch 'machine-learning-exchange:main' into vald-id-only

b3b7462

Added validate-id check to dataset_service_controller_impl.py and

bb528c9

model_service_controller_impl.py. Signed-off-by: JAulet <51500017+JAulet@users.noreply.github.com>

ckadner requested changes Feb 15, 2022

View reviewed changes

Moved validate_id check to before id is generated and changed function

994879c

call to check the id of the yaml file itself rather than generated id. Signed-off-by: JAulet <51500017+JAulet@users.noreply.github.com>

ckadner marked this pull request as ready for review February 19, 2022 00:52

ckadner changed the title ~~[WIP] Reject asset upload with invalid id~~ Reject asset upload with invalid id Feb 19, 2022

mlx-bot removed the do-not-merge/work-in-progress label Feb 19, 2022

ckadner approved these changes Feb 24, 2022

View reviewed changes

mlx-bot assigned ckadner Feb 24, 2022

mlx-bot added the lgtm label Feb 24, 2022

mlx-bot added the approved label Feb 24, 2022

ckadner merged commit 4e8c003 into machine-learning-exchange:main Feb 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reject asset upload with invalid `id` #273

Reject asset upload with invalid `id` #273

JAulet commented Dec 1, 2021 •

edited by ckadner

Loading

ckadner left a comment

ckadner Feb 15, 2022 •

edited

Loading

ckadner Feb 15, 2022 •

edited

Loading

ckadner Feb 15, 2022 •

edited

Loading

JAulet commented Feb 18, 2022

ckadner commented Feb 18, 2022 •

edited

Loading

JAulet commented Feb 22, 2022

ckadner left a comment

mlx-bot commented Feb 24, 2022

Reject asset upload with invalid id #273

Reject asset upload with invalid id #273

Conversation

JAulet commented Dec 1, 2021 • edited by ckadner Loading

ckadner left a comment

Choose a reason for hiding this comment

ckadner Feb 15, 2022 • edited Loading

Choose a reason for hiding this comment

ckadner Feb 15, 2022 • edited Loading

Choose a reason for hiding this comment

ckadner Feb 15, 2022 • edited Loading

Choose a reason for hiding this comment

JAulet commented Feb 18, 2022

ckadner commented Feb 18, 2022 • edited Loading

JAulet commented Feb 22, 2022

ckadner left a comment

Choose a reason for hiding this comment

mlx-bot commented Feb 24, 2022

Reject asset upload with invalid `id` #273

Reject asset upload with invalid `id` #273

JAulet commented Dec 1, 2021 •

edited by ckadner

Loading

ckadner Feb 15, 2022 •

edited

Loading

ckadner Feb 15, 2022 •

edited

Loading

ckadner Feb 15, 2022 •

edited

Loading

ckadner commented Feb 18, 2022 •

edited

Loading