Chapter10: seldon_core.wrapper:handle_invalid_usage:60 - ERROR: {'status': {'status': 1, 'info': 'Invalid request data type:

[aquynh1682]

Hello everyone, it's me again :)))

I have resolved the previous errors I encountered. Currently, in this chapter 10, I have successfully executed the pipeline build and push. After a successful deployment, I tried to call the model (I used both curl and Postman), but it encountered an error. Below is the error log I encountered and the command to call the model.

call model with curl:
curl -vvvvk --header "content-type: application/json" -X POST -d @data.json https://yourdomain.com/api/v1.0/predictions; done

and response:

< HTTP/1.1 100 Continue
< HTTP/1.1 400 Bad Request
< Date: Mon, 24 Jul 2023 02:37:56 GMT
< Content-Type: text/plain; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Access-Control-Allow-Headers: Accept, Accept-Encoding, Authorization, Content-Length, Content-Type, X-CSRF-Token
< Access-Control-Allow-Methods: OPTIONS,POST
< Access-Control-Allow-Origin: *
< Seldon-Puid: 36e3f612-eb06-4f9c-98fe-ff74ceebdcf1
< X-Content-Type-Options: nosniff
< Strict-Transport-Security: max-age=15724800; includeSubDomains
* HTTP error before end of send, stop sending
< 
{"status":{"code":-1,"info":"Invalid request data type: {'annotations': {'list': [{'builtIn': 1, 'datasource': '-- Grafana --', 'enable': True, 'hide': True, 'iconColor': 'rgba(0, 211, 255, 1)', 'name': 'Annotations \u0026 Alerts', 'target': {'limit': 100, 'matchAny': False, 'tags': [], 'type': 'dashboard'}, 'type': 'dashboard'}]}, 'editable': True, 'fiscalYearStartMonth': 0, 'graphTooltip': 0, 'id': 2, 'iteration': 1648959757365, 'links': [], 'liveNow': False, 'panels': [{'collapsed': True, 'gridPos': {'h': 1, 'w': 24, 'x': 0, 'y': 0}, 'id': 40, 'panels': [{'gridPos': {'h': 3, 'w': 24, 'x': 0, 'y': 1}, 'id': 27, 'links': [], 'options': {'content': '\u003cdiv class=\"text-center dashboard-header\"\u003e\\n  \u003cspan\u003eSeldon Core API Dashboard\u003c/span\u003e\\n\u003c/div\u003e', 'mode': 'html'}, 'pluginVersion': '8.3.3', 'type': 'text'}], 'title': 'Heading', 'type': 'row'}, {'collapsed': False, 'gridPos': {'h': 1, 'w': 24, 'x': 0, 'y': 1}, 'id': 41, 'panels': [], 'title': 'Global Counts', 'type': 'row'}, {'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'fieldConfig': {'defaults': {'color': {'fixedColor': 'rgb(31, 120, 193)', 'mode': 'fixed'}, 'mappings': [{'options': {'match': 'null', 'result': {'text': 'N/A'}}, 'type': 'special'}], 'thresholds': {'mode': 'absolute', 'steps': [{'color': 'green', 'value': None}, {'color': 'red', 'value': 80}]}, 'unit': 'ops'}, 'overrides': []}, 'gridPos': {'h': 3, 'w': 6, 'x': 0, 'y': 2}, 'id': 16, 'links': [], 'maxDataPoints': 100, 'options': {'colorMode': 'none', 'graphMode': 'area', 'justifyMode': 'auto', 'orientation': 'horizontal', 'reduceOptions': {'calcs': ['lastNotNull'], 'fields': '', 'values': False}, 'textMode': 'auto'}, 'pluginVersion': '8.3.3', 'targets': [{'expr': 'round(sum(irate(seldon_api_executor_server_requests_seconds_count[1m])),0.001)', 'format': 'time_series', 'instant': False, 'intervalFactor': 2, 'refId': 'A', 'step': 20}], 'title': 'Global Request Rate', 'type': 'stat'}, {'fieldConfig': {'defaults': {'color': {'fixedColor': 'rgb(31, 120, 193)', 'mode': 'fixed'}, 'mappings': [{'options': {'match': 'null', 'result': {'text': 'N/A'}}, 'type': 'special'}], 'thresholds': {'mode': 'absolute', 'steps': [{'color': 'green', 'value': None}, {'color': 'red', 'value': 80}]}, 'unit': 'percentunit'}, 'overrides': []}, 'gridPos': {'h': 3, 'w': 6, 'x': 6, 'y': 2}, 'id': 17, 'links': [], 'maxDataPoints': 100, 'options': {'colorMode': 'none', 'graphMode': 'area', 'justifyMode': 'auto', 'orientation': 'horizontal', 'reduceOptions': {'calcs': ['mean'], 'fields': '', 'values': False}, 'textMode': 'auto'}, 'pluginVersion': '8.3.3', 'targets': [{'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'exemplar': True, 'expr': 'sum(rate(seldon_api_executor_server_requests_seconds_count{code!~\"5.*\"}[1m])) / sum(rate(seldon_api_executor_server_requests_seconds_count[1m]))', 'format': 'time_series', 'interval': '', 'intervalFactor': 2, 'legendFormat': '', 'refId': 'A', 'step': 20}], 'title': 'Success', 'type': 'stat'}, {'fieldConfig': {'defaults': {'color': {'fixedColor': 'rgb(31, 120, 193)', 'mode': 'fixed'}, 'mappings': [{'options': {'match': 'null', 'result': {'text': 'N/A'}}, 'type': 'special'}], 'thresholds': {'mode': 'absolute', 'steps': [{'color': 'green', 'value': None}, {'color': 'red', 'value': 80}]}, 'unit': 'ops'}, 'overrides': []}, 'gridPos': {'h': 3, 'w': 6, 'x': 12, 'y': 2}, 'id': 18, 'links': [], 'maxDataPoints': 100, 'options': {'colorMode': 'none', 'graphMode': 'area', 'justifyMode': 'auto', 'orientation': 'horizontal', 'reduceOptions': {'calcs': ['mean'], 'fields': '', 'values': False}, 'textMode': 'auto'}, 'pluginVersion': '8.3.3', 'targets': [{'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'exemplar': True, 'expr': 'sum(irate(seldon_api_executor_server_requests_seconds_count{code=~\"4.*\"}[1m])) ', 'format': 'time_series', 'interval': '', 'intervalFactor': 2, 'legendFormat': '', 'refId': 'A', 'step': 20}], 'title': '4xxs', 'type': 'stat'}, {'fieldConfig': {'defaults': {'color': {'fixedColor': 'rgb(31, 120, 193)', 'mode': 'fixed'}, 'mappings': [{'options': {'match': 'null', 'result': {'text': 'N/A'}}, 'type': 'special'}], 'thresholds': {'mode': 'absolute', 'steps': [{'color': 'green', 'value': None}, {'color': 'red', 'value': 80}]}, 'unit': 'ops'}, 'overrides': []}, 'gridPos': {'h': 3, 'w': 6, 'x': 18, 'y': 2}, 'id': 19, 'links': [], 'maxDataPoints': 100, 'options': {'colorMode': 'none', 'graphMode': 'area', 'justifyMode': 'auto', 'orientation': 'horizontal', 'reduceOptions': {'calcs': ['mean'], 'fields': '', 'values': False}, 'textMode': 'auto'}, 'pluginVersion': '8.3.3', 'targets': [{'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'exemplar': True, 'expr': 'sum(irate(seldon_api_executor_server_requests_seconds_count{code=~\"5.*\"}[1m])) ', 'format': 'time_series', 'hide': False, 'interval': '', 'intervalFactor': 2, 'legendFormat': '', 'refId': 'A', 'step': 20}], 'title': '5xxs', 'type': 'stat'}, {'collapsed': False, 'gridPos': {'h': 1, 'w': 24, 'x': 0, 'y': 5}, 'id': 42, 'panels': [], 'repeat': 'deployment', 'title': 'Deployment Counts $deployment', 'type': 'row'}, {'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'fieldConfig': {'defaults': {'color': {'fixedColor': 'rgb(31, 120, 193)', 'mode': 'fixed'}, 'mappings': [{'options': {'match': 'null', 'result': {'text': 'N/A'}}, 'type': 'special'}], 'thresholds': {'mode': 'absolute', 'steps': [{'color': 'green', 'value': None}, {'color': 'red', 'value': 80}]}, 'unit': 'ops'}, 'overrides': []}, 'gridPos': {'h': 3, 'w': 6, 'x': 0, 'y': 6}, 'id': 36, 'links': [], 'maxDataPoints': 100, 'options': {'colorMode': 'none', 'graphMode': 'area', 'justifyMode': 'auto', 'orientation': 'horizontal', 'reduceOptions': {'calcs': ['lastNotNull'], 'fields': '', 'values': False}, 'textMode': 'auto'}, 'pluginVersion': '8.3.3', 'targets': [{'expr': \"round(sum(irate(seldon_api_executor_client_requests_seconds_count{deployment_name=~'$deployment'}[1m])), 0.001)\", 'format': 'time_series', 'intervalFactor': 2, 'refId': 'A', 'step': 20}], 'title': 'Request Rate ($deployment)', 'type': 'stat'}, {'fieldConfig': {'defaults': {'color': {'fixedColor': 'rgb(31, 120, 193)', 'mode': 'fixed'}, 'mappings': [{'options': {'match': 'null', 'result': {'text': 'N/A'}}, 'type': 'special'}], 'thresholds': {'mode': 'absolute', 'steps': [{'color': 'green', 'value': None}, {'color': 'red', 'value': 80}]}, 'unit': 'percentunit'}, 'overrides': []}, 'gridPos': {'h': 3, 'w': 6, 'x': 6, 'y': 6}, 'id': 37, 'links': [], 'maxDataPoints': 100, 'options': {'colorMode': 'none', 'graphMode': 'area', 'justifyMode': 'auto', 'orientation': 'horizontal', 'reduceOptions': {'calcs': ['mean'], 'fields': '', 'values': False}, 'textMode': 'auto'}, 'pluginVersion': '8.3.3', 'targets': [{'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'exemplar': True, 'expr': 'sum(rate(seldon_api_executor_client_requests_seconds_count{deployment_name=~\"$deployment\",code!~\"5.*\"}[1m])) / sum(rate(seldon_api_executor_client_requests_seconds_count{deployment_name=~\"$deployment\"}[1m]))', 'format': 'time_series', 'interval': '', 'intervalFactor': 2, 'legendFormat': '', 'refId': 'A', 'step': 20}], 'title': 'Success ($deployment)', 'type': 'stat'}, {'fieldConfig': {'defaults': {'color': {'fixedColor': 'rgb(31, 120, 193)', 'mode': 'fixed'}, 'mappings': [{'options': {'match': 'null', 'result': {'text': 'N/A'}}, 'type': 'special'}], 'thresholds': {'mode': 'absolute', 'steps': [{'color': 'green', 'value': None}, {'color': 'red', 'value': 80}]}, 'unit': 'ops'}, 'overrides': []}, 'gridPos': {'h': 3, 'w': 6, 'x': 12, 'y': 6}, 'id': 38, 'links': [], 'maxDataPoints': 100, 'options': {'colorMode': 'none', 'graphMode': 'area', 'justifyMode': 'auto', 'orientation': 'horizontal', 'reduceOptions': {'calcs': ['mean'], 'fields': '', 'values': False}, 'textMode': 'auto'}, 'pluginVersion': '8.3.3', 'targets': [{'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'exemplar': True, 'expr': 'sum(irate(seldon_api_executor_client_requests_seconds_count{deployment_name=~\"$deployment\",code=~\"4.*\"}[1m]))', 'format': 'time_series', 'interval': '', 'intervalFactor': 2, 'legendFormat': '', 'refId': 'A', 'step': 20}], 'title': '4xxs ($deployment)', 'type': 'stat'}, {'fieldConfig': {'defaults': {'color': {'fixedColor': 'rgb(31, 120, 193)', 'mode': 'fixed'}, 'mappings': [{'options': {'match': 'null', 'result': {'text': 'N/A'}}, 'type': 'special'}], 'thresholds': {'mode': 'absolute', 'steps': [{'color': 'green', 'value': None}, {'color': 'red', 'value': 80}]}, 'unit': 'ops'}, 'overrides': []}, 'gridPos': {'h': 3, 'w': 6, 'x': 18, 'y': 6}, 'id': 39, 'links': [], 'maxDataPoints': 100, 'options': {'colorMode': 'none', 'graphMode': 'area', 'justifyMode': 'auto', 'orientation': 'horizontal', 'reduceOptions': {'calcs': ['mean'], 'fields': '', 'values': False}, 'textMode': 'auto'}, 'pluginVersion': '8.3.3', 'targets': [{'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'exemplar': True, 'expr': 'sum(irate(seldon_api_executor_client_requests_seconds_count{deployment_name=~\"$deployment\",code=~\"5.*\"}[1m]))', 'format': 'time_series', 'hide': False, 'interval': '', 'intervalFactor': 2, 'legendFormat': '', 'refId': 'A', 'step': 20}], 'title': '5xxs ($deployment)', 'type': 'stat'}, {'collapsed': True, 'gridPos': {'h': 1, 'w': 24, 'x': 0, 'y': 9}, 'id': 43, 'panels': [{'gridPos': {'h': 3, 'w': 24, 'x': 0, 'y': 13}, 'id': 8, 'links': [], 'options': {'content': '\u003cdiv class=\"text-center dashboard-header\"\u003e\\n  \u003cspan\u003eModels\u003c/span\u003e\\n\u003c/div\u003e', 'mode': 'html'}, 'pluginVersion': '8.3.3', 'type': 'text'}], 'title': 'Models', 'type': 'row'}, {'aliasColors': {}, 'bars': False, 'dashLength': 10, 'dashes': False, 'fieldConfig': {'defaults': {'links': []}, 'overrides': []}, 'fill': 1, 'fillGradient': 0, 'gridPos': {'h': 9, 'w': 12, 'x': 0, 'y': 10}, 'hiddenSeries': False, 'id': 7, 'legend': {'avg': False, 'current': False, 'max': False, 'min': False, 'show': True, 'total': False, 'values': False}, 'lines': True, 'linewidth': 1, 'links': [], 'nullPointMode': 'null', 'options': {'alertThreshold': True}, 'percentage': False, 'pluginVersion': '8.3.3', 'pointradius': 5, 'points': False, 'renderer': 'flot', 'seriesOverrides': [], 'spaceLength': 10, 'stack': False, 'steppedLine': False, 'targets': [{'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'exemplar': True, 'expr': 'sum(rate(seldon_api_executor_client_requests_seconds_count{model_name=~\"$model_name\",model_version=~\"$model_version\",model_image=~\"$model_image\",predictor_name=~\"$predictor\",predictor_version=~\"$version\"}[5s])) by (model_name,predictor_name,predictor_version,model_image,model_version,service)', 'format': 'time_series', 'hide': True, 'interval': '', 'intervalFactor': 2, 'legendFormat': '{{predictor_name}}:{{predictor_version}} ({{model_name}} {{model_image}} : {{model_version}}) {{service}}', 'metric': 'io_seldon_apife_api_rest_RestClientController_home_snapshot_75thPercentile', 'refId': 'A', 'step': 2}, {'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'exemplar': True, 'expr': 'rate(seldon_api_executor_client_requests_seconds_count{model_name=~\"$model_name\",model_version=~\"$model_version\",model_image=~\"$model_image\",predictor_name=~\"$predictor\",predictor_version=~\"$version\"}[1m])', 'hide': False, 'interval': '', 'legendFormat': '', 'refId': 'B'}], 'thresholds': [], 'timeRegions': [], 'title': 'Reqs/sec to $model_image', 'tooltip': {'shared': True, 'sort': 0, 'value_type': 'individual'}, 'type': 'graph', 'xaxis': {'mode': 'time', 'show': True, 'values': []}, 'yaxes': [{'format': 'short', 'logBase': 1, 'min': '0', 'show': True}, {'format': 'short', 'logBase': 1, 'show': True}], 'yaxis': {'align': False}}, {'aliasColors': {}, 'bars': False, 'dashLength': 10, 'dashes': False, 'fieldConfig': {'defaults': {'links': []}, 'overrides': []}, 'fill': 1, 'fillGradient': 0, 'gridPos': {'h': 9, 'w': 12, 'x': 12, 'y': 10}, 'hiddenSeries': False, 'id': 11, 'legend': {'alignAsTable': True, 'avg': False, 'current': False, 'max': False, 'min': False, 'show': True, 'total': False, 'values': False}, 'lines': True, 'linewidth': 1, 'links': [], 'nullPointMode': 'null', 'options': {'alertThreshold': True}, 'percentage': False, 'pluginVersion': '8.3.3', 'pointradius': 5, 'points': False, 'renderer': 'flot', 'seriesOverrides': [], 'spaceLength': 10, 'stack': False, 'steppedLine': False, 'targets': [{'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'expr': 'histogram_quantile(0.5, sum(rate(seldon_api_executor_client_requests_seconds_bucket{service=~\"/Predict\", model_image=~\"$model_image\",predictor_name=~\"$predictor\",predictor_version=~\"$version\",model_name=~\"$model_name\",model_version=~\"$model_version\"}[20s])) by (predictor_name,predictor_version,model_name,model_image,model_version,le))', 'format': 'time_series', 'hide': True, 'intervalFactor': 2, 'legendFormat': '{{predictor_name}}:{{predictor_version}} {{model_name}} {{model_image}}: {{model_version}} (p50)', 'metric': '', 'refId': 'E', 'step': 2}, {'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'exemplar': True, 'expr': 'histogram_quantile(0.75, sum(rate(seldon_api_executor_client_requests_seconds_bucket{model_image=~\"$model_image\",predictor_name=~\"$predictor\",predictor_version=~\"$version\",model_name=~\"$model_name\",model_version=~\"$model_version\"}[1m])) by (model_image, predictor_name, predictor_name, model_name, model_version, le))', 'format': 'time_series', 'hide': False, 'interval': '', 'intervalFactor': 2, 'legendFormat': '{{predictor_name}}:{{predictor_version}} {{model_name}} {{model_image}}:{{model_version}} {{service}} (p75)', 'metric': '', 'refId': 'B', 'step': 2}, {'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'exemplar': True, 'expr': 'histogram_quantile(0.9, sum(rate(seldon_api_executor_client_requests_seconds_bucket{model_image=~\"$model_image\",predictor_name=~\"$predictor\",predictor_version=~\"$version\",model_name=~\"$model_name\",model_version=~\"$model_version\"}[1m])) by (model_image, predictor_name, predictor_name, model_name, model_version, le))', 'format': 'time_series', 'hide': False, 'interval': '', 'intervalFactor': 2, 'legendFormat': '{{predictor_name}}:{{predictor_version}} {{model_name}} {{model_image}}:{{model_version}}  {{service}} (p90)', 'metric': '', 'refId': 'A', 'step': 2}, {'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'exemplar': True, 'expr': 'histogram_quantile(0.95, sum(rate(seldon_api_executor_client_requests_seconds_bucket{model_image=~\"$model_image\",predictor_name=~\"$predictor\",predictor_version=~\"$version\",model_name=~\"$model_name\",model_version=~\"$model_version\"}[1m])) by (model_image, predictor_name, predictor_name, model_name, model_version, le))', 'format': 'time_series', 'hide': False, 'interval': '', 'intervalFactor': 2, 'legendFormat': '{{predictor_name}}:{{predictor_version}} {{model_name}} {{model_image}}:{{model_version}} {{service}} (p95)', 'metric': '', 'refId': 'C', 'step': 2}, {'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'exemplar': True, 'expr': 'histogram_quantile(0.99, sum(rate(seldon_api_executor_client_requests_seconds_bucket{model_image=~\"$model_image\",predictor_name=~\"$predictor\",predictor_version=~\"$version\",model_name=~\"$model_name\",model_version=~\"$model_version\"}[1m])) by (model_image, predictor_name, predictor_name, model_name, model_version, le))', 'format': 'time_series', 'hide': False, 'interval': '', 'intervalFactor': 2, 'legendFormat': '{{predictor_name}}:{{predictor_version}} {{model_name}} {{model_image}}:{{model_version}} {{service}} (p99)', 'metric': '', 'refId': 'D', 'step': 2}, {'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'exemplar': True, 'expr': 'sum(rate(seldon_api_executor_client_requests_seconds_bucket{model_image=~\"$model_image\",predictor_name=~\"$predictor\",predictor_version=~\"$version\",model_name=~\"$model_name\",model_version=~\"$model_version\"}[1m]))', 'hide': True, 'interval': '', 'legendFormat': '', 'refId': 'F'}, {'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'exemplar': True, 'expr': 'code=\"500\",deployment_name=\"model-d4973568c999443a9ab474efc625f496\",method=\"post\",model_image=\"quay.io/ml-on-k8s/container-model\",model_name=\"transformer\",model_version=\"2.0.0\",predictor_name=\"flights-ontime\",predictor_version=\"\",service=\"/transform-input\",le=\"0.005\"}', 'hide': True, 'interval': '', 'legendFormat': '', 'refId': 'G'}], 'thresholds': [], 'timeRegions': [], 'title': '$model_image Latency', 'tooltip': {'shared': False, 'sort': 0, 'value_type': 'individual'}, 'type': 'graph', 'xaxis': {'mode': 'time', 'show': True, 'values': []}, 'yaxes': [{'format': 'short', 'logBase': 1, 'min': '0', 'show': True}, {'format': 'short', 'logBase': 1, 'show': True}], 'yaxis': {'align': False}}, {'collapsed': False, 'gridPos': {'h': 1, 'w': 24, 'x': 0, 'y': 19}, 'id': 44, 'panels': [], 'repeat': 'model_image', 'title': 'Model Metrics $model_image', 'type': 'row'}], 'refresh': '', 'schemaVersion': 34, 'style': 'dark', 'tags': ['seldon'], 'templating': {'list': [{'allValue': '.*', 'current': {'selected': True, 'text': ['All'], 'value': ['$__all']}, 'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'definition': 'label_values(seldon_api_executor_client_requests_seconds_count,deployment_name)', 'hide': 0, 'includeAll': True, 'multi': True, 'name': 'deployment', 'options': [], 'query': {'query': 'label_values(seldon_api_executor_client_requests_seconds_count,deployment_name)', 'refId': 'prometheus-deployment-Variable-Query'}, 'refresh': 1, 'regex': '', 'skipUrlSync': False, 'sort': 0, 'tagValuesQuery': '', 'tagsQuery': '', 'type': 'query', 'useTags': False}, {'allValue': '.*', 'current': {'selected': True, 'text': ['All'], 'value': ['$__all']}, 'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'definition': 'label_values(seldon_api_executor_client_requests_seconds_count,predictor_name)', 'hide': 0, 'includeAll': True, 'multi': True, 'name': 'predictor', 'options': [], 'query': {'query': 'label_values(seldon_api_executor_client_requests_seconds_count,predictor_name)', 'refId': 'prometheus-predictor-Variable-Query'}, 'refresh': 1, 'regex': '', 'skipUrlSync': False, 'sort': 0, 'tagValuesQuery': '', 'tagsQuery': '', 'type': 'query', 'useTags': False}, {'allValue': '.*', 'current': {'selected': True, 'text': ['All'], 'value': ['$__all']}, 'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'definition': 'label_values(seldon_api_executor_client_requests_seconds_count,predictor_version)', 'hide': 0, 'includeAll': True, 'multi': True, 'name': 'version', 'options': [], 'query': {'query': 'label_values(seldon_api_executor_client_requests_seconds_count,predictor_version)', 'refId': 'prometheus-version-Variable-Query'}, 'refresh': 1, 'regex': '', 'skipUrlSync': False, 'sort': 0, 'tagValuesQuery': '', 'tagsQuery': '', 'type': 'query', 'useTags': False}, {'allValue': '.*', 'current': {'selected': True, 'text': ['All'], 'value': ['$__all']}, 'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'definition': 'label_values(seldon_api_executor_client_requests_seconds_count,model_name)', 'hide': 0, 'includeAll': True, 'multi': True, 'name': 'model_name', 'options': [], 'query': {'query': 'label_values(seldon_api_executor_client_requests_seconds_count,model_name)', 'refId': 'prometheus-model_name-Variable-Query'}, 'refresh': 1, 'regex': '', 'skipUrlSync': False, 'sort': 0, 'tagValuesQuery': '', 'tagsQuery': '', 'type': 'query', 'useTags': False}, {'allValue': '.*', 'current': {'selected': True, 'text': ['All'], 'value': ['$__all']}, 'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'definition': 'label_values(seldon_api_executor_client_requests_seconds_count,model_image)', 'hide': 0, 'includeAll': True, 'multi': True, 'name': 'model_image', 'options': [], 'query': {'query': 'label_values(seldon_api_executor_client_requests_seconds_count,model_image)', 'refId': 'prometheus-model_image-Variable-Query'}, 'refresh': 1, 'regex': '', 'skipUrlSync': False, 'sort': 0, 'tagValuesQuery': '', 'tagsQuery': '', 'type': 'query', 'useTags': False}, {'allValue': '.*', 'current': {'selected': True, 'text': ['All'], 'value': ['$__all']}, 'datasource': {'type': 'prometheus', 'uid': 'HPonbHsnk'}, 'definition': 'label_values(seldon_api_executor_client_requests_seconds_count,model_version)', 'hide': 0, 'includeAll': True, 'multi': True, 'name': 'model_version', 'options': [], 'query': {'query': 'label_values(seldon_api_executor_client_requests_seconds_count,model_version)', 'refId': 'prometheus-model_version-Variable-Query'}, 'refresh': 1, 'regex': '', 'skipUrlSync': False, 'sort': 0, 'tagValuesQuery': '', 'tagsQuery': '', 'type': 'query', 'useTags': False}]}, 'time': {'from': 'now-30m', 'to': 'now'}, 'timepicker': {'refresh_intervals': ['5s', '10s', '30s', '1m', '5m', '15m', '30m', '1h', '2h', '1d'], 'time_options': ['5m', '15m', '1h', '6h', '12h', '24h', '2d', '7d', '30d']}, 'timezone': 'browser', 'title': 'Fights Prediction Analytics', 'uid': 'U1cSDzyZzx', 'version': 7, 'weekStart': ''}","reason":"MICROSERVICE_BAD_DATA","status":1}}
* Closing connection 0

Seldon-core Logs:

[2023-07-24 02:40:21 +0000] [82] [DEBUG] Closing connection. 
[2023-07-24 02:40:21 +0000] [82] [DEBUG] Closing connection. 
[2023-07-24 02:40:22 +0000] [82] [DEBUG] POST /transform-input
2023-07-24 02:40:22,049 - seldon_core.wrapper:TransformInput:106 - DEBUG:  REST Request: <Request 'https://localhost:9000/transform-input' [POST]>
2023-07-24 02:40:22,050 - seldon_core.wrapper:handle_invalid_usage:60 - ERROR:  {'status': {'status': 1, 'info': 'Invalid request data type: {\'annotations\': {\'list\': [{\'builtIn\': 1, \'datasource\': \'-- Grafana --\', \'enable\': True, \'hide\': True, \'iconColor\': \'rgba(0, 211, 255, 1)\', \'name\': \'Annotations & Alerts\', \'target\': {\'limit\': 100, \'matchAny\': False, \'tags\': [], \'type\': \'dashboard\'}, \'type\': \'dashboard\'}]}, \'editable\': True, \'fiscalYearStartMonth\': 0, \'graphTooltip\': 0, \'id\': 2, \'iteration\': 1648959757365, \'links\': [], \'liveNow\': False, \'panels\': [{\'collapsed\': True, \'gridPos\': {\'h\': 1, \'w\': 24, \'x\': 0, \'y\': 0}, \'id\': 40, \'panels\': [{\'gridPos\': {\'h\': 3, \'w\': 24, \'x\': 0, \'y\': 1}, \'id\': 27, \'links\': [], \'options\': {\'content\': \'<div class="text-center dashboard-header">\\n  <span>Seldon Core API Dashboard</span>\\n</div>\', \'mode\': \'html\'}, \'pluginVersion\': \'8.3.3\', \'type\': \'text\'}], \'title\': \'Heading\', \'type\': \'row\'}, {\'collapsed\': False, \'gridPos\': {\'h\': 1, \'w\': 24, \'x\': 0, \'y\': 1}, \'id\': 41, \'panels\': [], \'title\': \'Global Counts\', \'type\': \'row\'}, {\'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'fieldConfig\': {\'defaults\': {\'color\': {\'fixedColor\': \'rgb(31, 120, 193)\', \'mode\': \'fixed\'}, \'mappings\': [{\'options\': {\'match\': \'null\', \'result\': {\'text\': \'N/A\'}}, \'type\': \'special\'}], \'thresholds\': {\'mode\': \'absolute\', \'steps\': [{\'color\': \'green\', \'value\': None}, {\'color\': \'red\', \'value\': 80}]}, \'unit\': \'ops\'}, \'overrides\': []}, \'gridPos\': {\'h\': 3, \'w\': 6, \'x\': 0, \'y\': 2}, \'id\': 16, \'links\': [], \'maxDataPoints\': 100, \'options\': {\'colorMode\': \'none\', \'graphMode\': \'area\', \'justifyMode\': \'auto\', \'orientation\': \'horizontal\', \'reduceOptions\': {\'calcs\': [\'lastNotNull\'], \'fields\': \'\', \'values\': False}, \'textMode\': \'auto\'}, \'pluginVersion\': \'8.3.3\', \'targets\': [{\'expr\': \'round(sum(irate(seldon_api_executor_server_requests_seconds_count[1m])),0.001)\', \'format\': \'time_series\', \'instant\': False, \'intervalFactor\': 2, \'refId\': \'A\', \'step\': 20}], \'title\': \'Global Request Rate\', \'type\': \'stat\'}, {\'fieldConfig\': {\'defaults\': {\'color\': {\'fixedColor\': \'rgb(31, 120, 193)\', \'mode\': \'fixed\'}, \'mappings\': [{\'options\': {\'match\': \'null\', \'result\': {\'text\': \'N/A\'}}, \'type\': \'special\'}], \'thresholds\': {\'mode\': \'absolute\', \'steps\': [{\'color\': \'green\', \'value\': None}, {\'color\': \'red\', \'value\': 80}]}, \'unit\': \'percentunit\'}, \'overrides\': []}, \'gridPos\': {\'h\': 3, \'w\': 6, \'x\': 6, \'y\': 2}, \'id\': 17, \'links\': [], \'maxDataPoints\': 100, \'options\': {\'colorMode\': \'none\', \'graphMode\': \'area\', \'justifyMode\': \'auto\', \'orientation\': \'horizontal\', \'reduceOptions\': {\'calcs\': [\'mean\'], \'fields\': \'\', \'values\': False}, \'textMode\': \'auto\'}, \'pluginVersion\': \'8.3.3\', \'targets\': [{\'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'exemplar\': True, \'expr\': \'sum(rate(seldon_api_executor_server_requests_seconds_count{code!~"5.*"}[1m])) / sum(rate(seldon_api_executor_server_requests_seconds_count[1m]))\', \'format\': \'time_series\', \'interval\': \'\', \'intervalFactor\': 2, \'legendFormat\': \'\', \'refId\': \'A\', \'step\': 20}], \'title\': \'Success\', \'type\': \'stat\'}, {\'fieldConfig\': {\'defaults\': {\'color\': {\'fixedColor\': \'rgb(31, 120, 193)\', \'mode\': \'fixed\'}, \'mappings\': [{\'options\': {\'match\': \'null\', \'result\': {\'text\': \'N/A\'}}, \'type\': \'special\'}], \'thresholds\': {\'mode\': \'absolute\', \'steps\': [{\'color\': \'green\', \'value\': None}, {\'color\': \'red\', \'value\': 80}]}, \'unit\': \'ops\'}, \'overrides\': []}, \'gridPos\': {\'h\': 3, \'w\': 6, \'x\': 12, \'y\': 2}, \'id\': 18, \'links\': [], \'maxDataPoints\': 100, \'options\': {\'colorMode\': \'none\', \'graphMode\': \'area\', \'justifyMode\': \'auto\', \'orientation\': \'horizontal\', \'reduceOptions\': {\'calcs\': [\'mean\'], \'fields\': \'\', \'values\': False}, \'textMode\': \'auto\'}, \'pluginVersion\': \'8.3.3\', \'targets\': [{\'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'exemplar\': True, \'expr\': \'sum(irate(seldon_api_executor_server_requests_seconds_count{code=~"4.*"}[1m])) \', \'format\': \'time_series\', \'interval\': \'\', \'intervalFactor\': 2, \'legendFormat\': \'\', \'refId\': \'A\', \'step\': 20}], \'title\': \'4xxs\', \'type\': \'stat\'}, {\'fieldConfig\': {\'defaults\': {\'color\': {\'fixedColor\': \'rgb(31, 120, 193)\', \'mode\': \'fixed\'}, \'mappings\': [{\'options\': {\'match\': \'null\', \'result\': {\'text\': \'N/A\'}}, \'type\': \'special\'}], \'thresholds\': {\'mode\': \'absolute\', \'steps\': [{\'color\': \'green\', \'value\': None}, {\'color\': \'red\', \'value\': 80}]}, \'unit\': \'ops\'}, \'overrides\': []}, \'gridPos\': {\'h\': 3, \'w\': 6, \'x\': 18, \'y\': 2}, \'id\': 19, \'links\': [], \'maxDataPoints\': 100, \'options\': {\'colorMode\': \'none\', \'graphMode\': \'area\', \'justifyMode\': \'auto\', \'orientation\': \'horizontal\', \'reduceOptions\': {\'calcs\': [\'mean\'], \'fields\': \'\', \'values\': False}, \'textMode\': \'auto\'}, \'pluginVersion\': \'8.3.3\', \'targets\': [{\'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'exemplar\': True, \'expr\': \'sum(irate(seldon_api_executor_server_requests_seconds_count{code=~"5.*"}[1m])) \', \'format\': \'time_series\', \'hide\': False, \'interval\': \'\', \'intervalFactor\': 2, \'legendFormat\': \'\', \'refId\': \'A\', \'step\': 20}], \'title\': \'5xxs\', \'type\': \'stat\'}, {\'collapsed\': False, \'gridPos\': {\'h\': 1, \'w\': 24, \'x\': 0, \'y\': 5}, \'id\': 42, \'panels\': [], \'repeat\': \'deployment\', \'title\': \'Deployment Counts $deployment\', \'type\': \'row\'}, {\'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'fieldConfig\': {\'defaults\': {\'color\': {\'fixedColor\': \'rgb(31, 120, 193)\', \'mode\': \'fixed\'}, \'mappings\': [{\'options\': {\'match\': \'null\', \'result\': {\'text\': \'N/A\'}}, \'type\': \'special\'}], \'thresholds\': {\'mode\': \'absolute\', \'steps\': [{\'color\': \'green\', \'value\': None}, {\'color\': \'red\', \'value\': 80}]}, \'unit\': \'ops\'}, \'overrides\': []}, \'gridPos\': {\'h\': 3, \'w\': 6, \'x\': 0, \'y\': 6}, \'id\': 36, \'links\': [], \'maxDataPoints\': 100, \'options\': {\'colorMode\': \'none\', \'graphMode\': \'area\', \'justifyMode\': \'auto\', \'orientation\': \'horizontal\', \'reduceOptions\': {\'calcs\': [\'lastNotNull\'], \'fields\': \'\', \'values\': False}, \'textMode\': \'auto\'}, \'pluginVersion\': \'8.3.3\', \'targets\': [{\'expr\': "round(sum(irate(seldon_api_executor_client_requests_seconds_count{deployment_name=~\'$deployment\'}[1m])), 0.001)", \'format\': \'time_series\', \'intervalFactor\': 2, \'refId\': \'A\', \'step\': 20}], \'title\': \'Request Rate ($deployment)\', \'type\': \'stat\'}, {\'fieldConfig\': {\'defaults\': {\'color\': {\'fixedColor\': \'rgb(31, 120, 193)\', \'mode\': \'fixed\'}, \'mappings\': [{\'options\': {\'match\': \'null\', \'result\': {\'text\': \'N/A\'}}, \'type\': \'special\'}], \'thresholds\': {\'mode\': \'absolute\', \'steps\': [{\'color\': \'green\', \'value\': None}, {\'color\': \'red\', \'value\': 80}]}, \'unit\': \'percentunit\'}, \'overrides\': []}, \'gridPos\': {\'h\': 3, \'w\': 6, \'x\': 6, \'y\': 6}, \'id\': 37, \'links\': [], \'maxDataPoints\': 100, \'options\': {\'colorMode\': \'none\', \'graphMode\': \'area\', \'justifyMode\': \'auto\', \'orientation\': \'horizontal\', \'reduceOptions\': {\'calcs\': [\'mean\'], \'fields\': \'\', \'values\': False}, \'textMode\': \'auto\'}, \'pluginVersion\': \'8.3.3\', \'targets\': [{\'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'exemplar\': True, \'expr\': \'sum(rate(seldon_api_executor_client_requests_seconds_count{deployment_name=~"$deployment",code!~"5.*"}[1m])) / sum(rate(seldon_api_executor_client_requests_seconds_count{deployment_name=~"$deployment"}[1m]))\', \'format\': \'time_series\', \'interval\': \'\', \'intervalFactor\': 2, \'legendFormat\': \'\', \'refId\': \'A\', \'step\': 20}], \'title\': \'Success ($deployment)\', \'type\': \'stat\'}, {\'fieldConfig\': {\'defaults\': {\'color\': {\'fixedColor\': \'rgb(31, 120, 193)\', \'mode\': \'fixed\'}, \'mappings\': [{\'options\': {\'match\': \'null\', \'result\': {\'text\': \'N/A\'}}, \'type\': \'special\'}], \'thresholds\': {\'mode\': \'absolute\', \'steps\': [{\'color\': \'green\', \'value\': None}, {\'color\': \'red\', \'value\': 80}]}, \'unit\': \'ops\'}, \'overrides\': []}, \'gridPos\': {\'h\': 3, \'w\': 6, \'x\': 12, \'y\': 6}, \'id\': 38, \'links\': [], \'maxDataPoints\': 100, \'options\': {\'colorMode\': \'none\', \'graphMode\': \'area\', \'justifyMode\': \'auto\', \'orientation\': \'horizontal\', \'reduceOptions\': {\'calcs\': [\'mean\'], \'fields\': \'\', \'values\': False}, \'textMode\': \'auto\'}, \'pluginVersion\': \'8.3.3\', \'targets\': [{\'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'exemplar\': True, \'expr\': \'sum(irate(seldon_api_executor_client_requests_seconds_count{deployment_name=~"$deployment",code=~"4.*"}[1m]))\', \'format\': \'time_series\', \'interval\': \'\', \'intervalFactor\': 2, \'legendFormat\': \'\', \'refId\': \'A\', \'step\': 20}], \'title\': \'4xxs ($deployment)\', \'type\': \'stat\'}, {\'fieldConfig\': {\'defaults\': {\'color\': {\'fixedColor\': \'rgb(31, 120, 193)\', \'mode\': \'fixed\'}, \'mappings\': [{\'options\': {\'match\': \'null\', \'result\': {\'text\': \'N/A\'}}, \'type\': \'special\'}], \'thresholds\': {\'mode\': \'absolute\', \'steps\': [{\'color\': \'green\', \'value\': None}, {\'color\': \'red\', \'value\': 80}]}, \'unit\': \'ops\'}, \'overrides\': []}, \'gridPos\': {\'h\': 3, \'w\': 6, \'x\': 18, \'y\': 6}, \'id\': 39, \'links\': [], \'maxDataPoints\': 100, \'options\': {\'colorMode\': \'none\', \'graphMode\': \'area\', \'justifyMode\': \'auto\', \'orientation\': \'horizontal\', \'reduceOptions\': {\'calcs\': [\'mean\'], \'fields\': \'\', \'values\': False}, \'textMode\': \'auto\'}, \'pluginVersion\': \'8.3.3\', \'targets\': [{\'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'exemplar\': True, \'expr\': \'sum(irate(seldon_api_executor_client_requests_seconds_count{deployment_name=~"$deployment",code=~"5.*"}[1m]))\', \'format\': \'time_series\', \'hide\': False, \'interval\': \'\', \'intervalFactor\': 2, \'legendFormat\': \'\', \'refId\': \'A\', \'step\': 20}], \'title\': \'5xxs ($deployment)\', \'type\': \'stat\'}, {\'collapsed\': True, \'gridPos\': {\'h\': 1, \'w\': 24, \'x\': 0, \'y\': 9}, \'id\': 43, \'panels\': [{\'gridPos\': {\'h\': 3, \'w\': 24, \'x\': 0, \'y\': 13}, \'id\': 8, \'links\': [], \'options\': {\'content\': \'<div class="text-center dashboard-header">\\n  <span>Models</span>\\n</div>\', \'mode\': \'html\'}, \'pluginVersion\': \'8.3.3\', \'type\': \'text\'}], \'title\': \'Models\', \'type\': \'row\'}, {\'aliasColors\': {}, \'bars\': False, \'dashLength\': 10, \'dashes\': False, \'fieldConfig\': {\'defaults\': {\'links\': []}, \'overrides\': []}, \'fill\': 1, \'fillGradient\': 0, \'gridPos\': {\'h\': 9, \'w\': 12, \'x\': 0, \'y\': 10}, \'hiddenSeries\': False, \'id\': 7, \'legend\': {\'avg\': False, \'current\': False, \'max\': False, \'min\': False, \'show\': True, \'total\': False, \'values\': False}, \'lines\': True, \'linewidth\': 1, \'links\': [], \'nullPointMode\': \'null\', \'options\': {\'alertThreshold\': True}, \'percentage\': False, \'pluginVersion\': \'8.3.3\', \'pointradius\': 5, \'points\': False, \'renderer\': \'flot\', \'seriesOverrides\': [], \'spaceLength\': 10, \'stack\': False, \'steppedLine\': False, \'targets\': [{\'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'exemplar\': True, \'expr\': \'sum(rate(seldon_api_executor_client_requests_seconds_count{model_name=~"$model_name",model_version=~"$model_version",model_image=~"$model_image",predictor_name=~"$predictor",predictor_version=~"$version"}[5s])) by (model_name,predictor_name,predictor_version,model_image,model_version,service)\', \'format\': \'time_series\', \'hide\': True, \'interval\': \'\', \'intervalFactor\': 2, \'legendFormat\': \'{{predictor_name}}:{{predictor_version}} ({{model_name}} {{model_image}} : {{model_version}}) {{service}}\', \'metric\': \'io_seldon_apife_api_rest_RestClientController_home_snapshot_75thPercentile\', \'refId\': \'A\', \'step\': 2}, {\'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'exemplar\': True, \'expr\': \'rate(seldon_api_executor_client_requests_seconds_count{model_name=~"$model_name",model_version=~"$model_version",model_image=~"$model_image",predictor_name=~"$predictor",predictor_version=~"$version"}[1m])\', \'hide\': False, \'interval\': \'\', \'legendFormat\': \'\', \'refId\': \'B\'}], \'thresholds\': [], \'timeRegions\': [], \'title\': \'Reqs/sec to $model_image\', \'tooltip\': {\'shared\': True, \'sort\': 0, \'value_type\': \'individual\'}, \'type\': \'graph\', \'xaxis\': {\'mode\': \'time\', \'show\': True, \'values\': []}, \'yaxes\': [{\'format\': \'short\', \'logBase\': 1, \'min\': \'0\', \'show\': True}, {\'format\': \'short\', \'logBase\': 1, \'show\': True}], \'yaxis\': {\'align\': False}}, {\'aliasColors\': {}, \'bars\': False, \'dashLength\': 10, \'dashes\': False, \'fieldConfig\': {\'defaults\': {\'links\': []}, \'overrides\': []}, \'fill\': 1, \'fillGradient\': 0, \'gridPos\': {\'h\': 9, \'w\': 12, \'x\': 12, \'y\': 10}, \'hiddenSeries\': False, \'id\': 11, \'legend\': {\'alignAsTable\': True, \'avg\': False, \'current\': False, \'max\': False, \'min\': False, \'show\': True, \'total\': False, \'values\': False}, \'lines\': True, \'linewidth\': 1, \'links\': [], \'nullPointMode\': \'null\', \'options\': {\'alertThreshold\': True}, \'percentage\': False, \'pluginVersion\': \'8.3.3\', \'pointradius\': 5, \'points\': False, \'renderer\': \'flot\', \'seriesOverrides\': [], \'spaceLength\': 10, \'stack\': False, \'steppedLine\': False, \'targets\': [{\'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'expr\': \'histogram_quantile(0.5, sum(rate(seldon_api_executor_client_requests_seconds_bucket{service=~"/Predict", model_image=~"$model_image",predictor_name=~"$predictor",predictor_version=~"$version",model_name=~"$model_name",model_version=~"$model_version"}[20s])) by (predictor_name,predictor_version,model_name,model_image,model_version,le))\', \'format\': \'time_series\', \'hide\': True, \'intervalFactor\': 2, \'legendFormat\': \'{{predictor_name}}:{{predictor_version}} {{model_name}} {{model_image}}: {{model_version}} (p50)\', \'metric\': \'\', \'refId\': \'E\', \'step\': 2}, {\'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'exemplar\': True, \'expr\': \'histogram_quantile(0.75, sum(rate(seldon_api_executor_client_requests_seconds_bucket{model_image=~"$model_image",predictor_name=~"$predictor",predictor_version=~"$version",model_name=~"$model_name",model_version=~"$model_version"}[1m])) by (model_image, predictor_name, predictor_name, model_name, model_version, le))\', \'format\': \'time_series\', \'hide\': False, \'interval\': \'\', \'intervalFactor\': 2, \'legendFormat\': \'{{predictor_name}}:{{predictor_version}} {{model_name}} {{model_image}}:{{model_version}} {{service}} (p75)\', \'metric\': \'\', \'refId\': \'B\', \'step\': 2}, {\'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'exemplar\': True, \'expr\': \'histogram_quantile(0.9, sum(rate(seldon_api_executor_client_requests_seconds_bucket{model_image=~"$model_image",predictor_name=~"$predictor",predictor_version=~"$version",model_name=~"$model_name",model_version=~"$model_version"}[1m])) by (model_image, predictor_name, predictor_name, model_name, model_version, le))\', \'format\': \'time_series\', \'hide\': False, \'interval\': \'\', \'intervalFactor\': 2, \'legendFormat\': \'{{predictor_name}}:{{predictor_version}} {{model_name}} {{model_image}}:{{model_version}}  {{service}} (p90)\', \'metric\': \'\', \'refId\': \'A\', \'step\': 2}, {\'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'exemplar\': True, \'expr\': \'histogram_quantile(0.95, sum(rate(seldon_api_executor_client_requests_seconds_bucket{model_image=~"$model_image",predictor_name=~"$predictor",predictor_version=~"$version",model_name=~"$model_name",model_version=~"$model_version"}[1m])) by (model_image, predictor_name, predictor_name, model_name, model_version, le))\', \'format\': \'time_series\', \'hide\': False, \'interval\': \'\', \'intervalFactor\': 2, \'legendFormat\': \'{{predictor_name}}:{{predictor_version}} {{model_name}} {{model_image}}:{{model_version}} {{service}} (p95)\', \'metric\': \'\', \'refId\': \'C\', \'step\': 2}, {\'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'exemplar\': True, \'expr\': \'histogram_quantile(0.99, sum(rate(seldon_api_executor_client_requests_seconds_bucket{model_image=~"$model_image",predictor_name=~"$predictor",predictor_version=~"$version",model_name=~"$model_name",model_version=~"$model_version"}[1m])) by (model_image, predictor_name, predictor_name, model_name, model_version, le))\', \'format\': \'time_series\', \'hide\': False, \'interval\': \'\', \'intervalFactor\': 2, \'legendFormat\': \'{{predictor_name}}:{{predictor_version}} {{model_name}} {{model_image}}:{{model_version}} {{service}} (p99)\', \'metric\': \'\', \'refId\': \'D\', \'step\': 2}, {\'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'exemplar\': True, \'expr\': \'sum(rate(seldon_api_executor_client_requests_seconds_bucket{model_image=~"$model_image",predictor_name=~"$predictor",predictor_version=~"$version",model_name=~"$model_name",model_version=~"$model_version"}[1m]))\', \'hide\': True, \'interval\': \'\', \'legendFormat\': \'\', \'refId\': \'F\'}, {\'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'exemplar\': True, \'expr\': \'code="500",deployment_name="model-d4973568c999443a9ab474efc625f496",method="post",model_image="quay.io/ml-on-k8s/container-model",model_name="transformer",model_version="2.0.0",predictor_name="flights-ontime",predictor_version="",service="/transform-input",le="0.005"}\', \'hide\': True, \'interval\': \'\', \'legendFormat\': \'\', \'refId\': \'G\'}], \'thresholds\': [], \'timeRegions\': [], \'title\': \'$model_image Latency\', \'tooltip\': {\'shared\': False, \'sort\': 0, \'value_type\': \'individual\'}, \'type\': \'graph\', \'xaxis\': {\'mode\': \'time\', \'show\': True, \'values\': []}, \'yaxes\': [{\'format\': \'short\', \'logBase\': 1, \'min\': \'0\', \'show\': True}, {\'format\': \'short\', \'logBase\': 1, \'show\': True}], \'yaxis\': {\'align\': False}}, {\'collapsed\': False, \'gridPos\': {\'h\': 1, \'w\': 24, \'x\': 0, \'y\': 19}, \'id\': 44, \'panels\': [], \'repeat\': \'model_image\', \'title\': \'Model Metrics $model_image\', \'type\': \'row\'}], \'refresh\': \'\', \'schemaVersion\': 34, \'style\': \'dark\', \'tags\': [\'seldon\'], \'templating\': {\'list\': [{\'allValue\': \'.*\', \'current\': {\'selected\': True, \'text\': [\'All\'], \'value\': [\'$__all\']}, \'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'definition\': \'label_values(seldon_api_executor_client_requests_seconds_count,deployment_name)\', \'hide\': 0, \'includeAll\': True, \'multi\': True, \'name\': \'deployment\', \'options\': [], \'query\': {\'query\': \'label_values(seldon_api_executor_client_requests_seconds_count,deployment_name)\', \'refId\': \'prometheus-deployment-Variable-Query\'}, \'refresh\': 1, \'regex\': \'\', \'skipUrlSync\': False, \'sort\': 0, \'tagValuesQuery\': \'\', \'tagsQuery\': \'\', \'type\': \'query\', \'useTags\': False}, {\'allValue\': \'.*\', \'current\': {\'selected\': True, \'text\': [\'All\'], \'value\': [\'$__all\']}, \'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'definition\': \'label_values(seldon_api_executor_client_requests_seconds_count,predictor_name)\', \'hide\': 0, \'includeAll\': True, \'multi\': True, \'name\': \'predictor\', \'options\': [], \'query\': {\'query\': \'label_values(seldon_api_executor_client_requests_seconds_count,predictor_name)\', \'refId\': \'prometheus-predictor-Variable-Query\'}, \'refresh\': 1, \'regex\': \'\', \'skipUrlSync\': False, \'sort\': 0, \'tagValuesQuery\': \'\', \'tagsQuery\': \'\', \'type\': \'query\', \'useTags\': False}, {\'allValue\': \'.*\', \'current\': {\'selected\': True, \'text\': [\'All\'], \'value\': [\'$__all\']}, \'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'definition\': \'label_values(seldon_api_executor_client_requests_seconds_count,predictor_version)\', \'hide\': 0, \'includeAll\': True, \'multi\': True, \'name\': \'version\', \'options\': [], \'query\': {\'query\': \'label_values(seldon_api_executor_client_requests_seconds_count,predictor_version)\', \'refId\': \'prometheus-version-Variable-Query\'}, \'refresh\': 1, \'regex\': \'\', \'skipUrlSync\': False, \'sort\': 0, \'tagValuesQuery\': \'\', \'tagsQuery\': \'\', \'type\': \'query\', \'useTags\': False}, {\'allValue\': \'.*\', \'current\': {\'selected\': True, \'text\': [\'All\'], \'value\': [\'$__all\']}, \'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'definition\': \'label_values(seldon_api_executor_client_requests_seconds_count,model_name)\', \'hide\': 0, \'includeAll\': True, \'multi\': True, \'name\': \'model_name\', \'options\': [], \'query\': {\'query\': \'label_values(seldon_api_executor_client_requests_seconds_count,model_name)\', \'refId\': \'prometheus-model_name-Variable-Query\'}, \'refresh\': 1, \'regex\': \'\', \'skipUrlSync\': False, \'sort\': 0, \'tagValuesQuery\': \'\', \'tagsQuery\': \'\', \'type\': \'query\', \'useTags\': False}, {\'allValue\': \'.*\', \'current\': {\'selected\': True, \'text\': [\'All\'], \'value\': [\'$__all\']}, \'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'definition\': \'label_values(seldon_api_executor_client_requests_seconds_count,model_image)\', \'hide\': 0, \'includeAll\': True, \'multi\': True, \'name\': \'model_image\', \'options\': [], \'query\': {\'query\': \'label_values(seldon_api_executor_client_requests_seconds_count,model_image)\', \'refId\': \'prometheus-model_image-Variable-Query\'}, \'refresh\': 1, \'regex\': \'\', \'skipUrlSync\': False, \'sort\': 0, \'tagValuesQuery\': \'\', \'tagsQuery\': \'\', \'type\': \'query\', \'useTags\': False}, {\'allValue\': \'.*\', \'current\': {\'selected\': True, \'text\': [\'All\'], \'value\': [\'$__all\']}, \'datasource\': {\'type\': \'prometheus\', \'uid\': \'HPonbHsnk\'}, \'definition\': \'label_values(seldon_api_executor_client_requests_seconds_count,model_version)\', \'hide\': 0, \'includeAll\': True, \'multi\': True, \'name\': \'model_version\', \'options\': [], \'query\': {\'query\': \'label_values(seldon_api_executor_client_requests_seconds_count,model_version)\', \'refId\': \'prometheus-model_version-Variable-Query\'}, \'refresh\': 1, \'regex\': \'\', \'skipUrlSync\': False, \'sort\': 0, \'tagValuesQuery\': \'\', \'tagsQuery\': \'\', \'type\': \'query\', \'useTags\': False}]}, \'time\': {\'from\': \'now-30m\', \'to\': \'now\'}, \'timepicker\': {\'refresh_intervals\': [\'5s\', \'10s\', \'30s\', \'1m\', \'5m\', \'15m\', \'30m\', \'1h\', \'2h\', \'1d\'], \'time_options\': [\'5m\', \'15m\', \'1h\', \'6h\', \'12h\', \'24h\', \'2d\', \'7d\', \'30d\']}, \'timezone\': \'browser\', \'title\': \'Fights Prediction Analytics\', \'uid\': \'U1cSDzyZzx\', \'version\': 7, \'weekStart\': \'\'}', 'code': -1, 'reason': 'MICROSERVICE_BAD_DATA'}}

[webmakaka]

Hi, @aquynh1682!
Wrong content from file. You can take right from my repo.

[aquynh1682]

Hi, @webmakaka!
Perfect, thank you have help me resolve this issue. But i still have a small mistake with grafana and promethues, You has done this part?

[webmakaka]

Something wrong with sending metrics.
I did not find how to fix.

But there also was wrong name for Prometheus Data Source in the book.

Should be (I think)

Configuration -> Data sources -> Prometheus

Name: HPonbHsnk
URL: http://prometheus-operated:9090

[aquynh1682]

I has change name datasource same for you, but it not sending metrics. I think, I should try deploying grafana and prometheus to another namespace and I'll will redefine the value myself.

[webmakaka]

If you find how to fix, please share info.

[aquynh1682]

Hi @webmakaka,

I had resolved setup tool monitor seldon in chapter 10. You should try changing spec.endpoints.port="8000" -> spec.endpoints.port=http in Service Monitor file prometheus.yml. And you want check prometheus received, you should use port-forwarding. Good luck.

Dashboard Grafana

Prometheus metric

[webmakaka]

OK! Thanks!

[webmakaka]

There is another book with errors.

[Book][Trevor Grant, Holden Karau, Boris Lublinsky, Richard Liu, Ilan Filonenko] Kubeflow for Machine Learning From Lab to Production [ENG, 2020]

If you are interested in, go to discuss in their issues page.

(Main error was - I didn't find data from apache org website what needed for successful completion)

[aquynh1682]

Thank you sharing for me this book. Could you please send me the GitHub link or pdf file?

[webmakaka]

Ok. Send me your email address.

[aquynh1682]

You can send it to me via email: nm.quynh1608@gmail.com. Thanks you.

[rajat-packt]

Hey @aquynh1682 and @webmakaka, let me know if this issue is resolved and can be closed.
Also, if there is any beneficial information for other readers that should be added to the README, you can share that with me too.

[webmakaka]

Hi, @rajat-packt
I think yes.
Book author made a mistake and replaced right file
4b62595

[rajat-packt]

Thanks @webmakaka for letting us know about this.
I'll try to reach out to the author regarding this.
In the meantime, is it alright if I mention your file that you have shared in the README for other readers to use?

[webmakaka]

Hi, @rajat-packt
I took this file from book repo
https://github.com/PacktPublishing/Machine-Learning-on-Kubernetes/blob/7f2d7537d753514728cc5abd4f7e19deafbbc82f/chapter10/inference/data.json

And if you want mention file from my repo, is OK!