Wait for confident result (Blocking submit) #18
Conversation
robotrapta
force-pushed
the
blocking-submit
branch
from
6d17886 to
f589822
Compare
robotrapta
force-pushed
the
blocking-submit
branch
from
e3e4da0 to
36b01cf
Compare
robotrapta
changed the title
| ```Python | ||
| from groundlight import Groundlight | ||
| gl = Groundlight() | ||
| d = gl.create_detector("door", query="Is the door open?") # define with natural language |
There was a problem hiding this comment.
technically its just one line of code to build the model :) the other line is to use the model you already built.
UserGuide.md
Outdated
| ## Managing confidence levels and latency | ||
|
|
||
| How to build a computer vision system in 5 lines of python code: | ||
| Groundlight gives you a simple way to control the trade-off of latency against accuracy. The longer you can wait for an answer to your image query, the better accuracy you will get. In particular, if the ML models are unsure of the best response, they will escalate the image query to a real-time human monitor to review them. Your code can easily wait for this delayed response. |
There was a problem hiding this comment.
talk about escalating to more advanced and larger models too and eventually to real-time human monitor or to a task-specific expert ?
UserGuide.md
Outdated
| How to build a computer vision system in 5 lines of python code: | ||
| Groundlight gives you a simple way to control the trade-off of latency against accuracy. The longer you can wait for an answer to your image query, the better accuracy you will get. In particular, if the ML models are unsure of the best response, they will escalate the image query to a real-time human monitor to review them. Your code can easily wait for this delayed response. | ||
|
|
||
| The desired confidence level is set as the escalation threshold on your detector. This determines what is the minimum confidence score for the ML system to provide before the image query is escalated to a human monitor. |
There was a problem hiding this comment.
query is escalated. (no additional detail required)
| while time.time() - start_time < wait: | ||
| current_confidence = img_query.result.confidence | ||
| if current_confidence is None: | ||
| logging.debug(f"Image query with None confidence implies human label (for now)") |
There was a problem hiding this comment.
this is really confusing and needs to be documented further up as well.
There was a problem hiding this comment.
Completely agree. In fact, we should write a wrapper which hides this.
| You can see the confidence score returned for the image query: | ||
|
|
||
| ```Python | ||
| print(f"The confidence is {image_query.result.confidence}") |
There was a problem hiding this comment.
the None result here will be confusing...
UserGuide.md
Outdated
| print(f"The answer is {image_query.result}") | ||
| ``` | ||
|
|
||
| Or if you want to run as fast as possible, set `wait=0`. This way you will only get the ML results, without waiting for human review. Image queries which are below the desired confidence level still get escalated to human review, and the results are incorporated as training data to improve your ML model, but your code will not wait for that to happen. |
There was a problem hiding this comment.
escalation isn't always humans...
There was a problem hiding this comment.
scattering of comments but really just two things:
escalation isn't always about going straight to human. it could be to additional models, edge to cloud, etc. even though today is just one level, let's sell the dream here.
we need to either explain "confidence=None" or just start sending 100%. or 99%. or something like that.
updated verbage
2 participants
New
waitargument tosubmit_image_querywhich makes it wait (poll) for a confident result.Uses the detector's default confidence threshold - looks it up if necessary.
Is "
wait" the best parameter name? Maybedelayorwait_secorwait_until?block?Also would this be better as its own method?
submit_image_queryis likely to explode into a complex method.But maybe that's okay?