Clipper prediction doesn't seem to match the model prediction before deployment

[sueann]

We're getting mostly 1's for prediction coming from clipper. The model we deployed was giving 0's on some examples, but that is no longer true querying through clipper.

One hypothesis:
The latency requirement for the application is set at 20ms and the /metrics endpoint shows mean latency of 100ms, so maybe it's not serving real predictions to meet the latency goal? Is there a way to check that?

Are there other things we could check?

Also some ideas for making debugging easier:

• Some way to easily view errors as metrics -- number of errors, of what type. I'm guessing currently we can look at the clipper & model container logs to get this? (not sure what is currently logged)
• Model metadata such as when a model was deployed, what (name, version) a particular model container is using, which model containers are connected to clipper (basically a birds-eye view of the system).

[dcrankshaw]

Your hypothesis is probably correct. If the model doesn't return a prediction in time, you will see a 1 as output. The selection policies are sort of broken right now and we are in the process of removing them in #89 until they can be fixed.

You can try setting the application latency requirement at 150ms and see if you get better results. That shouldn't slow down your predictions because they will be returned before the deadline if they are available.

These are great ideas for debugging. Adding more visibility to the system should be at least partially addressed by CLIPPER-114. Once #89 is completed, we will add metrics to report how many predictions returned in time and how many used the default value. Also, #89 makes the default prediction configurable on a per-application basis which should make it easier to debug as well.

[jegonzal]

Does this mean their container is running slowly? We might want to add a basic dashboard for Clipper to monitor this stuff.

[dcrankshaw]

"Slowly" is somewhat subjective. Their container is running more slowly than the latency objective for their application. The metrics already show this. I agree a monitoring dashboard would be good. I created CLIPPER-142 to track this for the 0.2 release.

[dcrankshaw]

#116 now explicitly marks predictions as being the result of a default prediction or a prediction from the model container as part of the response object. The metrics at http://<clipper-url>:1338/metrics also report the ratio of default predictions.