Benchmarks for Dircon performance

[yminchen]

While we are modifying Dircon, we benchmark the performance here for future reference.

CPU: Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz
Program: run_dircon_squatting.cc

Commit: ea86f21
Snopt solve time: 125.76 seconds

---

Constraint scaling
Commit: 1645804
Snopt solve time: 56.16 seconds
Cost: 8370.99

---

Constraint scaling + variable scaling
CPU: i7-8750H
Commit: af0fc3b
Snopt solve time: 93.7 seconds
Cost: 8233.8

---

Constraint scaling + variable scaling + sparsity pattern
CPU: i7-8750H
Commit: dc79d8e
Snopt solve time: 138.3 seconds
Cost: 8233.43

---

• Note that the solve time is not the only metric for performance evaluation. We also need to look at the cost (quality of the optimal value).
• Evidence that variable scaling improves solving speed:
  2fb9d4d vs 8e44186
  Run command:
  ./bazel-bin/examples/Cassie/run_dircon_walking --store_data=true --tol=1e-5
  The solver will find a local minimum, and then solve the problem again with a new initial guess: ./bazel-bin/examples/Cassie/run_dircon_walking --store_data=true --tol=1e-5 --init_file="z.csv"
  It solves much slower in 2fb9d4d than in 8e44186. In snopt log, we can see that although snopt is taking unit step size, the cost doesn't get improved much after a step. Also, if we look at the gradient of the dynamic constraint, we can notice that the columns related to the slack variables have values about 1000 times smaller than other columns.

[yminchen]

There are some benchmark tests done in #169. We can see that a single change to the traj opt problem doesn't necessarily improve the solve time.

[yminchen]

I compared ipopt with snopt using the walking example (https://github.com/DAIRLab/dairlib/blob/5f5205cbfc9466531bcefc10ad524222fd0998fb/examples/Cassie/run_dircon_walking.cc) in my local branch.

Ipopt setting:

    // Ipopt settings adapted from CaSaDi and FROST
    auto id = drake::solvers::IpoptSolver::id();
    trajopt.SetSolverOption(id, "tol", tol);
    trajopt.SetSolverOption(id, "dual_inf_tol", tol);
    trajopt.SetSolverOption(id, "constr_viol_tol", tol);
    trajopt.SetSolverOption(id, "compl_inf_tol", tol);
    trajopt.SetSolverOption(id, "max_iter", max_iter);
    trajopt.SetSolverOption(id, "nlp_lower_bound_inf", -1e6);
    trajopt.SetSolverOption(id, "nlp_upper_bound_inf", 1e6);
    trajopt.SetSolverOption(id, "print_timing_statistics", "yes");
    trajopt.SetSolverOption(id, "print_level", 5);

    // Set to ignore overall tolerance/dual infeasibility, but terminate when
    // primal feasible and objective fails to increase over 5 iterations.
    trajopt.SetSolverOption(id, "acceptable_compl_inf_tol", tol);
    trajopt.SetSolverOption(id, "acceptable_constr_viol_tol", tol);
    trajopt.SetSolverOption(id, "acceptable_obj_change_tol", 1e-3);
    trajopt.SetSolverOption(id, "acceptable_tol", 1e2);
    trajopt.SetSolverOption(id, "acceptable_iter", 5);

Snopt setting:

    auto id = drake::solvers::SnoptSolver::id();
    // trajopt.SetSolverOption(id, "Print file", "../snopt.out");
    trajopt.SetSolverOption(id, "Major iterations limit", max_iter);
    trajopt.SetSolverOption(id, "Iterations limit", 100000);
    trajopt.SetSolverOption(id, "Verify level", 0);

    // snopt doc said try 2 if seeing snopta exit 40
    trajopt.SetSolverOption(id, "Scale option", 0);

    // target nonlinear constraint violation
    trajopt.SetSolverOption(id, "Major optimality tolerance", tol);

    // target complementarity gap
    trajopt.SetSolverOption(id, "Major feasibility tolerance", tol);

where max_iter= 100000, and tol = 1e-4

Solve time and the optimal cost:

	no scaling	constraint scaling	variable scaling	constraint and variable scaling
ipopt	951s (cost = 0.26)	441s (cost = 0.4)	N/A*	N/A*
snopt	N/A**	55s (cost = 27)***	169s (cost = 0.8)	14s (cost = 0.48)

* Variable scaling is not implemented for ipopt yet. The benchmark should be exactly the same as the one without variable scaling.
** I stopped snopt after 10 mins of solving, because it doesn't look like it's converging judging from the callback visualization.
*** Bad solution.

Ipopt spent more than 90% of the time evaluating the nonlinear constraints

[mposa]

Very useful results, and discouraging for ipopt. Did ipopt really need all that time to converge to good solution?

[yminchen]

Very useful results, and discouraging for ipopt. Did ipopt really need all that time to converge to good solution?

From the log file, it looks like ipopt could terminate a bit earlier if we changed the setting?

iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
...
 464  3.0930749e-01 6.11e-05 1.63e-01 -10.7 1.76e-02    -  1.00e+00 1.00e+00h  1
...
1012  2.6461627e-01 1.39e-05 7.15e-01 -11.0 1.52e-01    -  1.00e+00 3.12e-02h  6 (final iteration)

I think ipopt's performance is not so bad in this benchmark test. Without any scaling, snopt probably won't find a solution if we give it more time to solve. With only constraint scaling, snopt was stuck in a local bad solution. If we implement variable scaling for ipopt in drake, it's possible that ipopt can perform decently against snopt, maybe