Induction into the FAQ

[slivingston]

The ambition of this issue is to collect questions that are candidates for being listed as frequently asked questions (FAQ) together with answers on the website. As such, it may open and close according to whether there are new questions under consideration.

[vraman]

• Is this a physical or virtual event?
  Prior to the physical event at ICRA, we will be running virtual trials in simulation. We have released some software for running these trials for one of the problem domains, and more is forthcoming in the next week or so if all goes well.
• Is there a problem format?
  There are three problem domains, as described on the website. The problem format for the first challenge domain has been released for some time now, and an example of how controller code provided by the teams will be run can be found in the User's Guide. Similar code and documentation for the second domain is forthcoming.
• What should submissions look like?
  This is a controller challenge, meaning that given a problem in a specified format, teams will submit a controller (independent of how it is constructed) which is evaluated in a black-box fashoin. It is the second option. Our motivation is that in the end, what we care about is the production of a controller as efficiently as possible. Thus, teams will be able to submit controllers that are evaluated as black-box entities, and we will record the time it takes to generate a controller. Thus, methods that incorporate hand-constructed controllers may be at a disadvantage.
  To enable us to view submissions as more-or-less black boxes, we plan on using ROS to run trials, so the only requirement is that the controller consume and produce ROS messages. An example is provided here. We are happy to help troubleshoot any compatibility issues with submitted controller software.
• How will submissions be evaluated?
  Since we want to allow a wide range of possible controllers, we felt that requiring them to be, e.g. in the form of a finite state machine, was too restrictive. So the evaluation is going to be trial-based, with scoring based on how many trials executed satisfy the specification. This duration for each trial will be provided as a part of the problem instance in some cases, and randomly generated in others For reachability properties, there will be a reasonable time duration within which the property has to be demonstrated.

[slivingston]

The second question and answer above is introduced after slight editing via 8ab4df1.

[slivingston]

For the fourth question above, I propose instead the following answer.

Currently the only available evaluation is entirely based on Monte Carlo trials, where scoring is obtained from satisfaction of the task specification and performance metrics. The duration for each trial is provided as part of the problem instance in some cases, and it is randomly generated in others. This statistical approach to evaluation has well-known limitations in deciding correctness of solutions. However, it permits a great variety of possible controllers because essentially no internal structure of solutions is assumed. We are considering other methods of evaluation that require certain forms of solutions and allow correspondingly stronger assertions.

@vraman If you approve, then I will use it.

[vraman]

I approve of the modified answer to the fourth question above.

[slivingston]

For the third question above, I propose instead the following answer.

This is a controller challenge, meaning that given an instance of a problem in a specified format, each team will submit a controller that provides a solution using an appropriate interface. Concretely, the interface is defined using ROS topics and services. Currently the only method of evaluation is through Monte Carlo (randomized, finite duration) trials. As such, controllers themselves are largely opaque entities. An example is provided in examples/sci_concrete_examples/scripts/lqr.py. We are happy to help troubleshoot any compatibility issues with submitted controller software. Though not planned for the first challenge, we are considering other messaging-based interfaces, e.g., using LCM, as well as solutions of particular forms, e.g., involving Mealy machines and discrete abstractions.

[vraman]

Looks good to me.

[slivingston]

The answer proposed above and corresponding question are added in 2e6a04d.

[slivingston]

The answer proposed above and corresponding question are added in 57da0d2

[slivingston]

Because the first question "Is this a physical or virtual event?" is not an FAQ so much as an advertising suggestion, I am not including it. Having addressed all current candidate questions, I am closing this issue for now. Feel free to re-open or re-initiate discussion.

The first FAQ list is now public at http://fmrchallenge.org/faq.html