[CLIPPER-107][CLIPPER-109] Return JSON-formatted responses to prediction queries #116

Corey-Zumar · 2017-04-09T23:45:29Z

Modifies query frontend and dependent infrastructure to return JSON-formatted responses to prediction queries.

From CLIPPER-107 and the functional spec:

The format of the response body is a JSON object with the following fields:

Output (JSON key name “output”): The prediction JSON object.

Default used (JSON key name “default”): A boolean set to true if the model did not return in time and the default output was used. False otherwise.

If there is a non-recoverable error during query execution, an error JSON object will be returned instead and the HTTP response code will be set to the most appropriate error code (a 400 or 500 error code). The format of the error is a JSON object with the following fields:

Error (JSON key name “error”): The name of the error.

Cause (JSON key name “cause”): A string describing the source of the error.

Note: This PR is currently based on the branch corresponding to #89. As a result, many changes from #89 are redundantly included here.

AmplabJenkins · 2017-04-09T23:50:24Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/129/
Test FAILed.

AmplabJenkins · 2017-04-10T00:04:19Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/130/
Test PASSed.

AmplabJenkins · 2017-04-11T22:06:39Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/133/
Test PASSed.

dcrankshaw

This looks great. We should wait for #89 to be merged first.

Does this require any changes to clipper_manager.py or the tutorial (beyond those needed for #89)?

dcrankshaw · 2017-04-11T21:39:55Z

src/frontends/src/query_frontend.hpp

+const char* PREDICTION_RESPONSE_KEY_OUTPUT = "output";
+const char* PREDICTION_RESPONSE_KEY_USED_DEFAULT = "default";
+const char* PREDICTION_ERROR_RESPONSE_KEY_ERROR = "error";
+const char* PREDICTION_ERROR_RESPONSE_KEY_CAUSE = "cause";


Why are these char* instead of std::string? (I'm guessing there's a good reason)

Our utility methods for adding key-value pairs to JSON objects expect char* keys. There is no implicit cast from std::string to char*, so I've defined the constants this way. Alternatively, I could define these with the std::string type and call .data() to get char* pointers, but this seems less readable.

dcrankshaw · 2017-04-11T22:03:02Z

src/libclipper/include/clipper/query_processor.hpp

@@ -6,6 +6,7 @@
 #include <tuple>
 #include <utility>


Add #include <stdexcept> (it's better to explicitly include all the headers needed)

Included stdexept in exceptions.hpp

dcrankshaw · 2017-04-11T22:03:41Z

src/libclipper/include/clipper/query_processor.hpp

+  const long query_id_;
+  const std::string msg_;
+};
+


Let's define this in a separate exceptions.hpp file.

dcrankshaw · 2017-04-11T22:06:31Z

src/libclipper/src/query_processor.cpp

+  ss << "Explanation: " << msg_ << std::endl;
+  return ss.str().data();
+}
+


Move to exceptions.cpp

dcrankshaw · 2017-04-11T22:10:38Z

src/management/src/management_frontend.hpp

@@ -52,7 +52,7 @@ const std::string APPLICATION_JSON_SCHEMA = R"(
     {"model_name" := string, "model_version" := int}
   ],
   "input_type" := "integers" | "bytes" | "floats" | "doubles" | "strings",
-   "selection_policy" := string,
+   "default_output" := number,


s/number/float

AmplabJenkins · 2017-04-12T10:15:19Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/138/
Test FAILed.

Corey-Zumar

@dcrankshaw Addressed your comments - moved PredictError to separate exceptions.hpp and exceptions.cpp files.

Corey-Zumar · 2017-04-12T10:04:28Z

src/management/src/management_frontend.hpp

@@ -52,7 +52,7 @@ const std::string APPLICATION_JSON_SCHEMA = R"(
     {"model_name" := string, "model_version" := int}
   ],
   "input_type" := "integers" | "bytes" | "floats" | "doubles" | "strings",
-   "selection_policy" := string,
+   "default_output" := number,


AmplabJenkins · 2017-04-12T10:29:18Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/139/
Test PASSed.

dcrankshaw · 2017-04-12T15:30:24Z

LGTM. We need to wait on #115 and then #89 before merging this though

AmplabJenkins · 2017-04-13T02:39:00Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/142/
Test PASSed.

dcrankshaw · 2017-04-14T00:09:04Z

src/libclipper/src/query_processor.cpp

-      UNREACHABLE();
-    }
+    std::pair<Output, bool> final_output =
+        current_policy->combine_predictions(selection_state, query, outputs);
    std::chrono::time_point<std::chrono::high_resolution_clock> end =


@Corey-Zumar Can you add a ratio counter here that tracks the fraction of predictions that use the default output?

…_utils

AmplabJenkins · 2017-04-14T06:57:24Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/147/
Test PASSed.

AmplabJenkins · 2017-04-14T07:06:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/148/
Test PASSed.

Corey-Zumar · 2017-04-14T07:24:32Z

@dcrankshaw Rebased on develop. Please take another look over the code and confirm that the tutorial runs properly before merging - it's working well for me.

AmplabJenkins · 2017-04-14T07:25:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/149/
Test FAILed.

AmplabJenkins · 2017-04-14T07:32:22Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/150/
Test FAILed.

AmplabJenkins · 2017-04-14T08:41:17Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/151/
Test PASSed.

AmplabJenkins · 2017-04-14T09:15:42Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/152/
Test PASSed.

dcrankshaw · 2017-04-17T16:32:22Z

LGTM. Confirming that the tutorial works for me as well.

Corey-Zumar added the status: needs review label Apr 9, 2017

Corey-Zumar requested a review from dcrankshaw April 9, 2017 23:45

dcrankshaw changed the title ~~[CLIPPER-107][CLIPPER-109]~~ [CLIPPER-107][CLIPPER-109] Return JSON-formatted responses to prediction queries Apr 10, 2017

dcrankshaw requested changes Apr 11, 2017

View reviewed changes

dcrankshaw removed the status: needs review label Apr 11, 2017

dcrankshaw assigned Corey-Zumar Apr 11, 2017

Corey-Zumar commented Apr 12, 2017

View reviewed changes

Corey-Zumar added the status: needs review label Apr 12, 2017

dcrankshaw approved these changes Apr 12, 2017

View reviewed changes

Corey-Zumar removed the status: needs review label Apr 12, 2017

dcrankshaw reviewed Apr 14, 2017

View reviewed changes

dcrankshaw and others added 12 commits April 13, 2017 21:08

draft of the selection policy rewrite

c69e93b

adapted query processor to use new selection policy interface

0b78f33

minor cleanup

83b425c

added default output to application table

9951cb7

things are compiling but there's a linker error from header-only json…

18395ff

…_utils

unit tests passing

08769e4

addressed review comments and fixed end to end bench

956db6b

fixed unittests

85590c5

tweaks

d47e315

start json output

6532b37

json bool support, default output, return json

b20ab63

Set object

19c3637

Corey-Zumar added 11 commits April 13, 2017 21:08

Complete error handling, add tests

6d4a422

Format code

8eeafa1

Remove unused and incorrect method

4317219

Address review comments

8df9ad0

Split exceptions into header and implementation

c8776f8

Format code

796b342

Fix tutorial, issue with PredictError

639c948

format code

a46e8c3

Fix inconsistencies

3c3de2c

Fix rebase issues

6c29846

Fix unit tests, format code

4546912

Corey-Zumar force-pushed the output_json branch from 605f063 to 4546912 Compare April 14, 2017 06:41

Remove unused imports, add newline

3f3d421

Corey-Zumar added 2 commits April 14, 2017 00:22

Remove duplicate field

b1781f9

Revert container name change to clipper mgr

47f6ac8

Format code

5c78d4b

Fix frontend tests

a27d2ec

Corey-Zumar added 2 commits April 14, 2017 02:01

Add default output ratio

f85f503

rename ratio

e080f52

dcrankshaw merged commit 9dc4a80 into ucbrise:develop Apr 17, 2017

This was referenced Apr 17, 2017

JSON response from prediction #93

Closed

Clipper prediction doesn't seem to match the model prediction before deployment #118

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CLIPPER-107][CLIPPER-109] Return JSON-formatted responses to prediction queries #116

[CLIPPER-107][CLIPPER-109] Return JSON-formatted responses to prediction queries #116

Corey-Zumar commented Apr 9, 2017 •

edited

Loading

AmplabJenkins commented Apr 9, 2017

AmplabJenkins commented Apr 10, 2017

AmplabJenkins commented Apr 11, 2017

dcrankshaw left a comment

dcrankshaw Apr 11, 2017

Corey-Zumar Apr 11, 2017

dcrankshaw Apr 11, 2017

Corey-Zumar Apr 12, 2017

dcrankshaw Apr 11, 2017

Corey-Zumar Apr 12, 2017

dcrankshaw Apr 11, 2017

Corey-Zumar Apr 12, 2017

dcrankshaw Apr 11, 2017

Corey-Zumar Apr 12, 2017

AmplabJenkins commented Apr 12, 2017

Corey-Zumar left a comment

Corey-Zumar Apr 12, 2017

AmplabJenkins commented Apr 12, 2017

dcrankshaw commented Apr 12, 2017

AmplabJenkins commented Apr 13, 2017

dcrankshaw Apr 14, 2017

Corey-Zumar Apr 14, 2017

AmplabJenkins commented Apr 14, 2017

AmplabJenkins commented Apr 14, 2017

Corey-Zumar commented Apr 14, 2017

AmplabJenkins commented Apr 14, 2017

AmplabJenkins commented Apr 14, 2017

AmplabJenkins commented Apr 14, 2017

AmplabJenkins commented Apr 14, 2017

dcrankshaw commented Apr 17, 2017

[CLIPPER-107][CLIPPER-109] Return JSON-formatted responses to prediction queries #116

[CLIPPER-107][CLIPPER-109] Return JSON-formatted responses to prediction queries #116

Conversation

Corey-Zumar commented Apr 9, 2017 • edited Loading

AmplabJenkins commented Apr 9, 2017

AmplabJenkins commented Apr 10, 2017

AmplabJenkins commented Apr 11, 2017

dcrankshaw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Apr 12, 2017

Corey-Zumar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Apr 12, 2017

dcrankshaw commented Apr 12, 2017

AmplabJenkins commented Apr 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Apr 14, 2017

AmplabJenkins commented Apr 14, 2017

Corey-Zumar commented Apr 14, 2017

AmplabJenkins commented Apr 14, 2017

AmplabJenkins commented Apr 14, 2017

AmplabJenkins commented Apr 14, 2017

AmplabJenkins commented Apr 14, 2017

dcrankshaw commented Apr 17, 2017

Corey-Zumar commented Apr 9, 2017 •

edited

Loading