Induction into the FAQ #1
Comments
|
For the fourth question above, I propose instead the following answer. Currently the only available evaluation is entirely based on Monte Carlo trials, where scoring is obtained from satisfaction of the task specification and performance metrics. The duration for each trial is provided as part of the problem instance in some cases, and it is randomly generated in others. This statistical approach to evaluation has well-known limitations in deciding correctness of solutions. However, it permits a great variety of possible controllers because essentially no internal structure of solutions is assumed. We are considering other methods of evaluation that require certain forms of solutions and allow correspondingly stronger assertions. @vraman If you approve, then I will use it. |
I approve of the modified answer to the fourth question above. |
For the third question above, I propose instead the following answer. This is a controller challenge, meaning that given an instance of a problem in a specified format, each team will submit a controller that provides a solution using an appropriate interface. Concretely, the interface is defined using ROS topics and services. Currently the only method of evaluation is through Monte Carlo (randomized, finite duration) trials. As such, controllers themselves are largely opaque entities. An example is provided in examples/sci_concrete_examples/scripts/lqr.py. We are happy to help troubleshoot any compatibility issues with submitted controller software. Though not planned for the first challenge, we are considering other messaging-based interfaces, e.g., using LCM, as well as solutions of particular forms, e.g., involving Mealy machines and discrete abstractions. |
Looks good to me. |
Because the first question "Is this a physical or virtual event?" is not an FAQ so much as an advertising suggestion, I am not including it. Having addressed all current candidate questions, I am closing this issue for now. Feel free to re-open or re-initiate discussion. The first FAQ list is now public at http://fmrchallenge.org/faq.html |
The ambition of this issue is to collect questions that are candidates for being listed as frequently asked questions (FAQ) together with answers on the website. As such, it may open and close according to whether there are new questions under consideration.
The text was updated successfully, but these errors were encountered: