Legged-robot posture and trajectory Optimization Benchmark
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
data
destruction_scenarios @ c02625f
robots
solutions
.gitmodules
LegOptEnv.xml
LegOptEvalFast.py
LegOptMergeJSON.py
LegOptShowResultsHumanoids2017.py
README.md
legopt.png

README.md

LegOpt Benchmark

LegOpt Benchmark

This repository is for the LegOpt Benchmark (Legged-robot posture and trajectory Optimization Benchmark).

LegOpt is a benchmark for evaluating different algorithms at the tasks of posture generation and trajectory optimization on legged robots. There are 50 problems for each task, which are associated with a pre-defined robot model, scenario, constraints and cost functions.

The scenario is from the Destruction Scenarios Dataset.

In this repository we provide:

  1. latest benchmark results
  2. problem definition data
  3. solution evaluation scripts
  4. usage and submission instructions

Paper

If you use this in your research, please cite:

Martim Brandao, Kenji Hashimoto, and Atsuo Takanishi, "SGD for robot motion? The effectiveness of stochastic optimization on a new benchmark for biped locomotion tasks", in 17th IEEE-RAS International Conference on Humanoid Robots, 2017.

Make sure to check the paper for details (PDF here).


Latest benchmark results

Posture generationTrajectory optimization
MethodMulti starts Sample sizeSuccess rateCostTime (s)Success rateCostTime (s)
SQP 0 100 49/50 426.68 1.32 19/50 0.13 0.63
SGD 0 100 44/50 719.83 3.76 4 /50 0.20 1.38
Adam 0 100 42/50 606.38 4.15 18/50 0.14 1.22
Nadam 0 100 39/50 590.26 2.83 16/50 0.14 0.81
SQP 10 100 50/50 426.64 1.44 42/50 0.13 0.62
SGD 10 100 49/50 797.55 6.30 27/50 0.17 2.17
Adam 10 100 49/50 642.92 5.97 31/50 0.14 1.50
Nadam 10 100 49/50 641.97 5.67 35/50 0.15 1.19
SQP 10 80 16/50 297.75 2.74 0 /50 0.00 0.00
SGD 10 80 50/50 788.22 5.82 26/50 0.16 1.99
Adam 10 80 50/50 628.55 4.18 35/50 0.14 2.05
Nadam 10 80 50/50 619.66 2.90 33/50 0.14 1.10
I-SQP 10 80 50/50 460.58 1.00 42/50 0.13 0.53
I-SGD 10 40 49/50 833.01 4.13 30/50 0.15 1.30
I-Adam 10 40 50/50 660.53 3.27 40/50 0.14 1.07
I-Nadam 10 40 50/50 666.94 2.66 41/50 0.15 0.73

Note: costs and times are averaged over successfully solved problems.


Latest benchmark results (from "tough" initialization in collision)

Posture generationTrajectory optimization
MethodMulti starts Sample sizeSuccess rateCostTime (s)Success rateCostTime (s)
SQP 0 100 43/50 415.62 1.43 11/50 0.12 0.67
SGD 0 100 41/50 769.82 3.55 2 /50 0.18 1.26
Adam 0 100 46/50 616.37 3.78 2 /50 0.14 1.27
Nadam 0 100 48/50 645.27 2.33 1 /50 0.11 0.92
SQP 10 100 50/50 429.45 1.74 29/50 0.12 0.84
SGD 10 100 48/50 783.34 5.11 29/50 0.16 3.04
Adam 10 100 50/50 671.93 4.30 32/50 0.15 2.10
Nadam 10 100 50/50 678.50 3.20 30/50 0.15 1.96
SQP 10 80 20/50 278.90 4.00 0 /50 0.00 0.00
SGD 10 80 49/50 774.82 4.44 30/50 0.16 2.49
Adam 10 80 50/50 642.19 3.91 36/50 0.15 2.12
Nadam 10 80 50/50 637.78 2.70 34/50 0.14 1.47
I-SQP 10 80 50/50 468.25 1.34 38/50 0.14 0.71
I-SGD 10 40 49/50 809.90 5.17 30/50 0.15 1.40
I-Adam 10 40 50/50 668.34 3.66 34/50 0.15 1.34
I-Nadam 10 40 50/50 684.47 3.44 28/50 0.14 1.64

Note: costs and times are averaged over successfully solved problems.


Data format

JSON data for problems

Problems for each task are defined on JSON files. Please have a look inside folder data.

Maybe more intuitive than writing the schema here is to show how you would access the information in C++ or python. So here it is:

jsonroot["robot"]                         # name of the robot model used (this is always "atlas" for now)
jsonroot["scenario"]                      # name of the Destruction Scenario used (this is always "garage_easier" for now)
jsonroot["task"]                          # either "posture" or "motion", for posture generation or trajectory optimization respectively
jsonroot["problems"][p]                   # problem number p
jsonroot["problems"][p]["id"]             # the id of problem number p
jsonroot["problems"][p]["definition"]     # the definition of problem number p
jsonroot["problems"][p]["definition"][d]  # a string representing stance number d of problem number p

The format for strings representing stances is the following:

"<name-of-link-in-contact>,<space-separated-quaternion-xyzw> <space-separated-translation-xyz>,<name-of-link-in-contact>,<space-separated-quaternion-xyzw>,"

It assumes a rotation and translation from the world reference frame.

Example 1 (only the left foot is in contact):

"l_foot,-0.4896072729 -0.1111596603 -0.0645970827 -0.8624125841 -12.53751087 0.6949124932 5.57654953 ,"

Example 2 (both feet in contact):

"l_foot,-0.4896072729 -0.1111596603 -0.0645970827 -0.8624125841 -12.53751087 0.6949124932 5.57654953 ,r_foot,-0.4893193662 -0.1129687271 -0.06397458697 -0.8623873125 -12.60409355 1.338053823 5.595780373 ,"

JSON data for solutions

In order to evaluate your solutions with the scripts we provide, you should add them to a JSON file in the following way.

You should start from the respective problems JSON and add a "solution" object to each problem that you solve.

jsonroot["problems"][p]["solution"]                 # holds your solutions for problem number p

You can solve each problem with several methods, so "solution" is a list where each item contains the solution vector, the name of the method used and the computation time.

jsonroot["problems"][p]["solution"][s]["method"]    # name of the method used to obtain solution s
jsonroot["problems"][p]["solution"][s]["timeSec"]   # computation time (in seconds) to obtain solution s
jsonroot["problems"][p]["solution"][s]["x"][f]      # a string representing the full-body configuration number f of the solution

The format for strings representing a full-body configuration are "\n"-separated values of the DOFs (joints followed by translation and then quaternion). Basically, what you would obtain in C++ with operator<< on a Eigen::VectorXd.

"<joint1value>\n<joint2value>\n(...)<jointNvalue>\n<x>\n<y>\n<z>\n<qx>\n<qy>\n<qz>\n<qw>"

Example:

"0.0750938\n-0.0115883\n0.0618958\n0.174954\n-1.39626\n3.05126\n0.365756\n-0.442819\n-0.0552547\n-0.610865\n0.13954\n0.169922\n1.39626\n3.07882\n-0.31723\n-0.0876133\n-0.072731\n-0.201933\n0.0678042\n-0.0458344\n0.750349\n-0.783705\n0.208692\n-0.211367\n-0.0466808\n-0.786813\n0.742743\n-0.0395884\n0.324678\n-12.55\n0.949195\n6.42098\n0.448645\n0.00514189\n-0.0392512\n0.997566"

Usage and submission instructions

Clone this repository

Clone this repository using

git clone --recursive https://github.com/martimbrandao/legopt-benchmark.git

Dependencies

The list of dependencies is:

  • Python 2.7
  • numpy
  • scipy
  • JSON
  • OpenRAVE (we tested on commit 0a05f794a5901885a25ed5cbd6e70eb3f37e0edd)

If you don't have them yet do:

sudo apt-get install python python-numpy python-scipy

Clone, build and install OpenRAVE. You can follow these instructions if you are unsure how to do so.

Downloading the additional data

If you just want the problem definition data, then the JSON files are already inside the data folder. However, if you would like to compare the performance of your algorithms using the same problem initializations as I did, please use these files. The files are equivalent to the ones under the data folder, except for additional x0 and x0hard variables on each problem (same format as the solution variable x).

If you would like to try and run the scripts on my solutions, you can download them from here, and place the contents (a folder called solutions12-full-with-easy-and-hard-init) inside solutions/.

Evaluating your benchmark solutions

Run:

python LegOptEvalFast.py <path-to-solution-json-file> --verbose

You can also add a --interactive argument to visualize the solutions as they are checked in OpenRAVE.

If you would like to try and run the script with my solutions, see Downloading the additional data above.

Adding your results to this page

Contact me and I will add your results here.

For normal evaluations you are not required to initialize your algorithms from the same staring point x0 as provided in the additional data. However, if you want to evaluate your algorithm under tough initializations then you should initialize your algorithms using the x0hard values provided in the additional data (see Downloading the additional data above).