Clarify how perception task handles timing of submissions

[osrf-migration]

Original report (archived issue) by Michael McCarrin (Bitbucket: m1chaelm).

---

The behavior of the scoring plugin appears to be that it only allows the team to submit an answer while the object in question is still visible (i.e. before the next object appears). Given that we will only score the first answer, it seems like the optimal solution would want to take advantage of the full window before giving an answer, but would need to make sure it doesn't answer too late. However, we don't give any indication of when the object is going to go away, so this is hard to guess. I think there are a few solutions we might consider:

1. Simply state that every object will be visible for some fixed amount of time. For example, it looks like we currently switch every 5 seconds, so we could just document that this is the time.
2. If the time is meant to be variable, we can give some indication that an object is about to go away--either by publishing a message or giving a visual indication.
3. Ignore the time of submission and simply score the submissions in order. In this version, the objects would still appear and disappear at any speeds (or variable speeds), and teams can submit solutions any time before the onFinished state arrives.

My preference is option 3 because it seems relatively simple to implement and allows us to vary the difficulty of the task over different runs. Option 2 seems like the most complicated.

Of course, I doubt the above list is exhaustive. Other ideas?

[osrf-migration]

Original comment by Brian Bingham (Bitbucket: brian_bingham).

---

I was thinking #1 as the simplest option.

[osrf-migration]

Original comment by Michael McCarrin (Bitbucket: m1chaelm).

---

OK, sounds good to me. In your update of the task description, you define a “trial time” and we say it “is a fixed value specified ahead of time for the entire task.” I think this makes sense, but I’m not sure how we will communicate the trial time to the teams. Adding a rostopic for it seems like overkill. Is there something simple we can do?

[osrf-migration]

Original comment by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

I’d go for option 1 too.

[osrf-migration]

Original comment by Michael McCarrin (Bitbucket: m1chaelm).

---

OK, 1 works for me but see my response to Brian above. I still don’t understand how the teams will know what the “trial time” is unless we explicitly state a concrete value, like “5 seconds,” in the task description documentation.

[osrf-migration]

Original comment by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

That works for me. We could say 5 seconds in the task guide.

[osrf-migration]

Original comment by Michael McCarrin (Bitbucket: m1chaelm).

---

OK, I added the following to the task description: “In competition, the trial time for all trials will be fixed at five seconds.”

[osrf-migration]

Original comment by Michael McCarrin (Bitbucket: m1chaelm).

---

• changed state from "new" to "resolved"

Resolved using option 1 with a fixed trial time of 5 seconds, recorded in the task description.