Issue/1638 model metadata #1671

RafalSkolasinski · 2020-04-07T11:29:47Z

Deprecates metadata method in Python wrapper and introduce init_metadata method.

The init_metadata will be called once after model initialisation to validate and cache returned metadata.
Output of init_metadata is merged with content of MODEL_METADATA environmental variable with values in latter taking priorities.

For prepackaged model server (right now only sklearn) one can either define MODEL_METADATA environmental variables or add metadata.yaml file to the s3 bucket. Please see examples.

review-notebook-app · 2020-04-07T11:29:53Z

Check out this pull request on

You'll be able to see Jupyter notebook diff and discuss changes. Powered by ReviewNB.

seldondev · 2020-04-07T11:31:06Z

Tue Apr 7 11:31:06 UTC 2020
The logs for [lint] [2] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/2.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=2

seldondev · 2020-04-07T11:31:18Z

Tue Apr 7 11:31:17 UTC 2020
The logs for [pr-build] [1] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/1.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=1

seldondev · 2020-04-07T13:33:47Z

Tue Apr 7 13:33:46 UTC 2020
The logs for [lint] [4] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/4.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=4

seldondev · 2020-04-07T13:33:52Z

Tue Apr 7 13:33:51 UTC 2020
The logs for [pr-build] [3] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/3.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=3

seldondev · 2020-04-08T14:40:38Z

Wed Apr 8 14:40:38 UTC 2020
The logs for [lint] [6] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/6.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=6

seldondev · 2020-04-08T14:40:41Z

Wed Apr 8 14:40:40 UTC 2020
The logs for [pr-build] [5] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/5.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=5

seldondev · 2020-04-08T22:33:21Z

Wed Apr 8 22:33:20 UTC 2020
The logs for [lint] [8] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/8.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=8

seldondev · 2020-04-08T22:33:32Z

Wed Apr 8 22:33:31 UTC 2020
The logs for [pr-build] [7] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/7.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=7

seldondev · 2020-04-08T22:40:33Z

Wed Apr 8 22:40:32 UTC 2020
The logs for [pr-build] [9] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/9.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=9

seldondev · 2020-04-08T22:41:56Z

Wed Apr 8 22:41:55 UTC 2020
The logs for [lint] [10] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/10.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=10

seldondev · 2020-04-08T22:43:46Z

Wed Apr 8 22:43:45 UTC 2020
The logs for [pr-build] [11] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/11.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=11

seldondev · 2020-04-08T22:45:07Z

Wed Apr 8 22:45:06 UTC 2020
The logs for [pr-build] [13] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/13.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=13

seldondev · 2020-04-08T22:45:08Z

Wed Apr 8 22:45:07 UTC 2020
The logs for [lint] [12] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/12.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=12

seldondev · 2020-04-08T22:47:00Z

Wed Apr 8 22:46:58 UTC 2020
The logs for [lint] [14] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/14.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=14

seldondev · 2020-04-08T23:07:26Z

Wed Apr 8 23:07:25 UTC 2020
The logs for [lint] [16] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/16.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=16

seldondev · 2020-04-08T23:07:29Z

Wed Apr 8 23:07:28 UTC 2020
The logs for [pr-build] [15] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/15.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=15

seldondev · 2020-04-14T13:37:31Z

Tue Apr 14 13:37:30 UTC 2020
The logs for [pr-build] [18] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/18.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=18

seldondev · 2020-04-14T13:37:33Z

Tue Apr 14 13:37:32 UTC 2020
The logs for [lint] [17] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/17.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=17

seldondev · 2020-04-22T16:58:20Z

Wed Apr 22 16:58:19 UTC 2020
The logs for [pr-build] [67] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/67.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=67

seldondev · 2020-04-22T16:58:25Z

Wed Apr 22 16:58:24 UTC 2020
The logs for [lint] [68] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/68.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=68

seldondev · 2020-04-22T17:08:26Z

Wed Apr 22 17:08:26 UTC 2020
The logs for [lint] [70] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/70.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=70

seldondev · 2020-04-22T17:08:31Z

Wed Apr 22 17:08:30 UTC 2020
The logs for [pr-build] [69] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/69.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=69

RafalSkolasinski · 2020-04-22T17:10:57Z

/test integration
/test notebooks

seldondev · 2020-04-22T17:12:30Z

Wed Apr 22 17:12:29 UTC 2020
The logs for [integration] [72] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/72.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=72

seldondev · 2020-04-22T17:12:32Z

Wed Apr 22 17:12:31 UTC 2020
The logs for [notebooks] [71] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/71.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=71

axsaucedo

Looks great, only a couple of minor comments

examples/models/metadata/example-01/Model.py

axsaucedo · 2020-04-22T17:40:57Z

examples/models/metadata/example-02/model_rest.yaml

+              {"name": "my-model-name",
+               "versions": ["my-model-version-02"],
+               "platform": "seldon-custom",
+               "inputs": [{"name": "input", "datatype": "BYTES", "shape": [1]}],


why is it bytes?

axsaucedo · 2020-04-22T17:42:08Z

python/seldon_core/metadata.py

+    """
+
+    def __init__(self, name: str, datatype: str, shape: Union[List[int]]):
+        self.name = str(name)


Here it would still be name "None"

Only if someone call if with None, here there is no default value.

RafalSkolasinski · 2020-04-22T20:13:31Z

/retest

seldondev · 2020-04-22T20:14:41Z

Wed Apr 22 20:14:41 UTC 2020
The logs for [notebooks] [73] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/73.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=73

axsaucedo · 2020-04-22T22:04:20Z

Looks good, if we confirm that notebook tests failing are flaky should be good to land!
/test notebooks

seldondev · 2020-04-22T22:05:29Z

Wed Apr 22 22:05:28 UTC 2020
The logs for [notebooks] [74] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/74.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=74

RafalSkolasinski · 2020-04-23T07:59:17Z

Looks good, if we confirm that notebook tests failing are flaky should be good to land!
/test notebooks

Did you manage to see in jx logs which notebook test failed?

I can only see

the build pods for this build have been garbage collected and the log was not found in the long term storage bucket

:(

axsaucedo · 2020-04-23T08:29:59Z

/test notebooks

seldondev · 2020-04-23T08:31:33Z

Thu Apr 23 08:31:32 UTC 2020
The logs for [notebooks] [75] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/75.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=75

seldondev · 2020-04-23T09:46:53Z

Thu Apr 23 09:46:52 UTC 2020
The logs for [pr-build] [76] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/76.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=76

seldondev · 2020-04-23T09:47:02Z

Thu Apr 23 09:47:01 UTC 2020
The logs for [lint] [77] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1671/77.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1671 --build=77

RafalSkolasinski · 2020-04-23T09:48:26Z

All notebook tests passed for me locally.

Only customMetrics one didn't like that I have zsh as my underlying shell (kubectl run had problems) so I changed that part into a short shell script called explicitly with bash.

seldondev · 2020-04-23T10:10:53Z

@RafalSkolasinski: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
notebooks	`a703319`	link	`/test notebooks`

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

RafalSkolasinski · 2020-04-23T10:20:01Z

In CI it failed on TestNotebooks.test_ambassador_custom which passed for me locally.

ukclivecox · 2020-04-23T10:37:10Z

/approve

seldondev · 2020-04-23T10:37:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cliveseldon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [cliveseldon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

seldondev added the do-not-merge/work-in-progress label Apr 7, 2020

seldondev added the size/XL label Apr 7, 2020

seldondev added size/XXL and removed size/XL labels Apr 7, 2020

RafalSkolasinski force-pushed the issue/1638-model-metadata branch from 0c6f1b8 to 93027a3 Compare April 8, 2020 14:39

RafalSkolasinski force-pushed the issue/1638-model-metadata branch from 6b828f0 to d742051 Compare April 8, 2020 22:39

RafalSkolasinski force-pushed the issue/1638-model-metadata branch from d742051 to 518a506 Compare April 8, 2020 22:41

RafalSkolasinski force-pushed the issue/1638-model-metadata branch from 518a506 to 12fdac5 Compare April 8, 2020 22:43

RafalSkolasinski force-pushed the issue/1638-model-metadata branch from 12fdac5 to f878f3b Compare April 8, 2020 23:04

RafalSkolasinski force-pushed the issue/1638-model-metadata branch from f878f3b to bbabcb6 Compare April 14, 2020 13:36

RafalSkolasinski force-pushed the issue/1638-model-metadata branch from bbabcb6 to acc2afe Compare April 15, 2020 15:08

Add initial model metadata support + examples + basic dvc example

a703319

RafalSkolasinski force-pushed the issue/1638-model-metadata branch from b488393 to a703319 Compare April 22, 2020 17:07

RafalSkolasinski changed the title ~~WIP: Issue/1638 model metadata~~ Issue/1638 model metadata Apr 22, 2020

seldondev removed the do-not-merge/work-in-progress label Apr 22, 2020

axsaucedo reviewed Apr 22, 2020

View reviewed changes

ukclivecox mentioned this pull request Apr 23, 2020

Feature request: Provide a metadata endpoint with custom metadata #319

Closed

fix customMetrics notebook to run on zsh-enabled systems

1a267c8

seldondev added the approved label Apr 23, 2020

seldondev merged commit 3d71703 into SeldonIO:master Apr 23, 2020

Issue/1638 model metadata #1671

Issue/1638 model metadata #1671

Conversation

RafalSkolasinski commented Apr 7, 2020 • edited

review-notebook-app bot commented Apr 7, 2020

seldondev commented Apr 7, 2020

seldondev commented Apr 7, 2020

seldondev commented Apr 7, 2020

seldondev commented Apr 7, 2020

seldondev commented Apr 8, 2020

seldondev commented Apr 8, 2020

seldondev commented Apr 8, 2020

seldondev commented Apr 8, 2020

seldondev commented Apr 8, 2020

seldondev commented Apr 8, 2020

seldondev commented Apr 8, 2020

seldondev commented Apr 8, 2020

seldondev commented Apr 8, 2020

seldondev commented Apr 8, 2020

seldondev commented Apr 8, 2020

seldondev commented Apr 8, 2020

seldondev commented Apr 14, 2020

seldondev commented Apr 14, 2020

seldondev commented Apr 22, 2020

seldondev commented Apr 22, 2020

seldondev commented Apr 22, 2020

seldondev commented Apr 22, 2020

RafalSkolasinski commented Apr 22, 2020

seldondev commented Apr 22, 2020

seldondev commented Apr 22, 2020

axsaucedo left a comment

Choose a reason for hiding this comment

axsaucedo Apr 22, 2020

Choose a reason for hiding this comment

axsaucedo Apr 22, 2020

Choose a reason for hiding this comment

RafalSkolasinski Apr 22, 2020 • edited

Choose a reason for hiding this comment

axsaucedo Apr 22, 2020

Choose a reason for hiding this comment

RafalSkolasinski commented Apr 22, 2020

seldondev commented Apr 22, 2020

axsaucedo commented Apr 22, 2020

seldondev commented Apr 22, 2020

RafalSkolasinski commented Apr 23, 2020 • edited

axsaucedo commented Apr 23, 2020

seldondev commented Apr 23, 2020

seldondev commented Apr 23, 2020

seldondev commented Apr 23, 2020

RafalSkolasinski commented Apr 23, 2020

seldondev commented Apr 23, 2020

RafalSkolasinski commented Apr 23, 2020

ukclivecox commented Apr 23, 2020

seldondev commented Apr 23, 2020

RafalSkolasinski commented Apr 7, 2020 •

edited

RafalSkolasinski Apr 22, 2020 •

edited

RafalSkolasinski commented Apr 23, 2020 •

edited