Metrics server with capabilities to fetch response from elasticsearch via request Puid in feedback request #2604

axsaucedo · 2020-11-02T19:15:10Z

What this PR does / why we need it:

Introduces further capability from infrastructure metrics workingroup to enable "RESPONSE" to be fetched directly from the elasticsearch database. PR includes:

Extra parameter for adserver to initialise elasticsearch client
Custom metrics able to fetch RESPONSE if request PUID is provided
Fixes on custom metrics server to work with param ndarrays
Example showing the workflow using metrics server with feedback containing request Puid

Jupyter notebook and README version of example.

Further discussion points:

Currently it's required to pass request Puid as feedback Puid header, however this may be ambiguous if we want to eventually have unique feedback UIDs
Current elasticsearch configuration is quite basic, but there could be some exploration to leverage the logic in the payload request logger to interact with ELK - this could involve merging the payload request logger with the CEServer as it's already quite modular.

review-notebook-app · 2020-11-02T19:15:15Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

seldondev · 2020-11-02T19:16:35Z

Mon Nov 2 19:16:34 UTC 2020
The logs for [lint] [2] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/2.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=2

seldondev · 2020-11-02T19:16:48Z

Mon Nov 2 19:16:47 UTC 2020
The logs for [pr-build] [1] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/1.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=1

axsaucedo · 2020-11-02T19:19:29Z

/test notebooks

seldondev · 2020-11-02T19:21:10Z

Mon Nov 2 19:21:09 UTC 2020
The logs for [notebooks] [3] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/3.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=3

seldondev · 2020-11-03T14:01:53Z

Tue Nov 3 14:01:52 UTC 2020
The logs for [pr-build] [4] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/4.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=4

seldondev · 2020-11-03T14:02:26Z

Tue Nov 3 14:02:25 UTC 2020
The logs for [lint] [5] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/5.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=5

axsaucedo · 2020-11-06T06:56:03Z

/test thigs

axsaucedo · 2020-11-06T06:56:07Z

/test this

seldondev · 2020-11-06T06:57:34Z

Fri Nov 6 06:57:31 UTC 2020
The logs for [pr-build] [6] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/6.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=6

components/alibi-detect-server/Dockerfile

ukclivecox · 2020-11-06T13:50:17Z

examples/feedback/feedback-metrics-server/config/multiclass-deployment.yaml

+          runAsUser: 8888
+      containers:
+      - name: user-container
+        image: seldonio/alibi-detect-server:1.4.0-dev


same - 1.5.0-dev
Might want to work out a way to have these take of the currebt version.txt?

seldondev · 2020-11-06T13:55:46Z

Fri Nov 6 13:55:45 UTC 2020
The logs for [lint] [8] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/8.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=8

examples/feedback/feedback-metrics-server/config/multiclass-service.yaml

seldondev · 2020-11-06T14:02:44Z

Fri Nov 6 14:02:43 UTC 2020
The logs for [pr-build] [9] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/9.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=9

seldondev · 2020-11-06T14:03:41Z

Fri Nov 6 14:03:36 UTC 2020
The logs for [lint] [10] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/10.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=10

seldondev · 2020-11-06T14:07:22Z

Fri Nov 6 14:07:21 UTC 2020
The logs for [pr-build] [11] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/11.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=11

seldondev · 2020-11-06T14:07:46Z

Fri Nov 6 14:07:45 UTC 2020
The logs for [lint] [12] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/12.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=12

seldondev · 2020-11-06T15:08:33Z

Fri Nov 6 15:08:28 UTC 2020
The logs for [lint] [14] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/14.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=14

seldondev · 2020-11-06T15:08:42Z

Fri Nov 6 15:08:41 UTC 2020
The logs for [pr-build] [13] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/13.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=13

seldondev · 2020-11-06T15:09:22Z

Fri Nov 6 15:09:21 UTC 2020
The logs for [pr-build] [15] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/15.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=15

seldondev · 2020-11-06T15:09:23Z

Fri Nov 6 15:09:22 UTC 2020
The logs for [lint] [16] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/16.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=16

seldondev · 2020-11-06T15:10:05Z

Fri Nov 6 15:10:04 UTC 2020
The logs for [pr-build] [17] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/17.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=17

seldondev · 2020-11-06T15:11:17Z

Fri Nov 6 15:11:16 UTC 2020
The logs for [lint] [18] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/18.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=18

seldondev · 2020-11-06T15:28:09Z

Fri Nov 6 15:28:08 UTC 2020
The logs for [lint] [20] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/20.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=20

seldondev · 2020-11-06T15:29:12Z

Fri Nov 6 15:29:11 UTC 2020
The logs for [pr-build] [19] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/19.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=19

axsaucedo · 2020-11-12T10:14:57Z

Merging as discussed in sc meeting
/approve

seldondev · 2020-11-12T10:15:17Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: axsaucedo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [axsaucedo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

seldondev · 2020-11-12T10:17:30Z

Thu Nov 12 10:17:28 UTC 2020
The logs for [lint] [22] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/22.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=22

seldondev · 2020-11-12T10:17:30Z

Thu Nov 12 10:17:28 UTC 2020
The logs for [pr-build] [21] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/21.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=21

seldondev · 2020-11-12T10:35:37Z

Thu Nov 12 10:35:35 UTC 2020
The logs for [lint] [24] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/24.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=24

seldondev · 2020-11-12T10:38:24Z

Thu Nov 12 10:38:23 UTC 2020
The logs for [pr-build] [23] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2604/23.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2604 --build=23

axsaucedo added the Metrics label Nov 2, 2020

axsaucedo added this to the 1.5 milestone Nov 2, 2020

axsaucedo requested review from ukclivecox, jklaise and ryandawsonuk November 2, 2020 19:15

axsaucedo added this to In progress in 1.5 via automation Nov 2, 2020

seldondev added the size/XXL label Nov 2, 2020

ukclivecox assigned axsaucedo Nov 5, 2020

ukclivecox moved this from In progress to In Review in 1.5 Nov 5, 2020

ukclivecox suggested changes Nov 6, 2020

View reviewed changes

components/alibi-detect-server/Dockerfile Outdated Show resolved Hide resolved

seldondev assigned ukclivecox Nov 6, 2020

ukclivecox reviewed Nov 6, 2020

View reviewed changes

axsaucedo added 8 commits November 6, 2020 13:51

Updated base version to latest 1.4.0-dev

3cdafc7

Updated makefile to align with headers

c6f3b7a

Updated cmmodels to support numpy

d426482

Updated adserver to support metrics async check for metrics server

d19a771

Added example for offline metrics from elasticsearch

bf23130

ADded true_negative metric to metrics servers

ad2a657

Added testing to example

05b2d65

Updated to 1.5.0

fba0a50

ukclivecox reviewed Nov 6, 2020

View reviewed changes

examples/feedback/feedback-metrics-server/config/multiclass-service.yaml Outdated Show resolved Hide resolved

Updated to 1.5.0

a3a90bf

Updated to 1.5.0

14bd7ab

axsaucedo requested a review from ukclivecox November 6, 2020 15:05

axsaucedo added 3 commits November 6, 2020 15:06

Updated to 1.5.0

f914fb4

Updated to 1.5.0

1a9bf19

Updated to 1.5.0

3ae97ed

Updated to 1.5.0

63afa91

seldondev added the approved label Nov 12, 2020

seldondev merged commit 824486f into SeldonIO:master Nov 12, 2020

1.5 automation moved this from In Review to Done Nov 12, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Metrics server with capabilities to fetch response from elasticsearch via request Puid in feedback request #2604

Metrics server with capabilities to fetch response from elasticsearch via request Puid in feedback request #2604

axsaucedo commented Nov 2, 2020

review-notebook-app bot commented Nov 2, 2020

seldondev commented Nov 2, 2020

seldondev commented Nov 2, 2020

axsaucedo commented Nov 2, 2020

seldondev commented Nov 2, 2020

seldondev commented Nov 3, 2020

seldondev commented Nov 3, 2020

axsaucedo commented Nov 6, 2020

axsaucedo commented Nov 6, 2020

seldondev commented Nov 6, 2020

ukclivecox Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

axsaucedo commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

Metrics server with capabilities to fetch response from elasticsearch via request Puid in feedback request #2604

Metrics server with capabilities to fetch response from elasticsearch via request Puid in feedback request #2604

Conversation

axsaucedo commented Nov 2, 2020

review-notebook-app bot commented Nov 2, 2020

seldondev commented Nov 2, 2020

seldondev commented Nov 2, 2020

axsaucedo commented Nov 2, 2020

seldondev commented Nov 2, 2020

seldondev commented Nov 3, 2020

seldondev commented Nov 3, 2020

axsaucedo commented Nov 6, 2020

axsaucedo commented Nov 6, 2020

seldondev commented Nov 6, 2020

ukclivecox Nov 6, 2020

Choose a reason for hiding this comment

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

seldondev commented Nov 6, 2020

axsaucedo commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020