Seldon dashboard with hundreds of REST requests

[beeva-enriqueotero]

Seldon grafana dashboard shows every API request for a client. With Recommendation REST API each user generates a different entry in charts. So after hundreds of users the chart legend is huge and the web interface hangs up.

Regards, and thank you
Enrique

[ukclivecox]

Hi,

The dashboard for recommendations should aggregate by request type as shown in http://docs.seldon.io/img/grafana.png
Do you have the version of Seldon you are running and perhaps some example or screenshot to help us debug this further?

[beeva-enriqueotero]

Hi,

The dashboard for recommendations aggregates by request type based on url. The problem with REST
API is that /users/1/recommendations and /users/10/recommendations, for instance, generate different request types.

I tested this behaviour with Seldon Server 1.3.11

[ukclivecox]

Yes, this will need a fix to the kafka-streams job that aggregates the stats and sends them to influxdb.
We will need to add to it knowledge of the REST endpoint URL structures so the user can be removed from the request.
An alternative would be to change the influxdb (via grafana) SELECT statements if possible, but I'm not sure of the flexibility of the influxdb query langauge to accomplish this.

[beeva-enriqueotero]

I don't know much about influxdb. But I don't find in its documentation any string function. So it doesn't seem easy to be done only by changing SELECT queries.

[seldondev]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
/lifecycle stale