Skip to content

Conversation

@janbruedigam
Copy link
Member

No description provided.

@rejuvyesh
Copy link
Contributor

Any estimates for difference in eval time with ForwardDiff vs FiniteDiff?

@janbruedigam
Copy link
Member Author

Haven't checked yet in detail, feels a bit faster, but I could also be completely wrong

@rejuvyesh
Copy link
Contributor

rejuvyesh commented Mar 24, 2022

Julia v1.8 (beta) shows timing information in tests as well. So possibly of interest.

Current master:

Integrator              |    7      7  9.7s
  FiniteDiff comparison |    7      7  9.4s
Test Summary:                         | Pass  Total     Time
Minimal Coordinates                   |  861    861  3m43.9s
  Get and set position and velocities |  602    602    22.7s
  Minimal to maximal to minimal       |   58     58    41.5s
  Jacobians                           |  201    201  2m38.1s
Test Summary:                 | Pass  Total  Time
Modified Rodrigues Parameters |    9      9  0.8s
  FiniteDiff comparison       |    5      5  0.7s
  axes pair to quaternion     |    4      4  0.1s
Test Summary:                | Pass  Total   Time
Damper                       |  120    120  51.8s
  rotational damper jacobian |  120    120  51.7s
Test Summary:              | Pass  Total     Time
Jacobian (Solution Matrix) |   32     32  5m06.5s
  Flying                   |   16     16  3m11.5s
  In contact               |   16     16  1m54.7s
Test Summary:                              | Pass  Total      Time
Data                                       |  184    184  10m41.6s
  Get and set data                         |    8      8      6.7s
  Data Jacobian without contact and limtis |   88     88   4m13.3s
  Data Jacobian with contact and limtis    |   88     88   6m17.4s
Test Summary:              | Pass  Total   Time
Collisions                 | 1566   1566  24.5s
  Collision: Sphere-sphere | 1566   1566  24.3s
Test Summary: | Pass  Total     Time
Momentum      |   73     73  4m12.8s
  Box         |    2      2     1.1s
  Pendulum    |    1      1     1.3s
  Humanoid    |    2      2    39.4s
  Atlas       |    2      2    53.4s
  Quadruped   |    2      2  1m38.1s
  Snake       |   32     32    29.4s
  Twister     |   32     32    30.0s
Test Summary: | Pass  Total     Time
Energy        |   42     42  3m15.8s
  Dice        |    1      1     0.7s
  Pendulum    |    1      1     5.9s
  Slider 1    |    3      3     1.4s
  Slider 2    |    1      1     0.6s
  Slider 3    |    1      1     2.4s
  Humanoid    |    1      1    10.8s
  Atlas       |    1      1    56.3s
  Quadruped   |    1      1    18.6s
  Snake       |   16     16    49.9s
  Twister     |   16     16    48.8s
Test Summary:      | Pass  Total   Time
Behavior           |   22     22  51.6s
  Quadruped        |    1      1  19.2s
  Box toss         |    8      8  11.0s
  Four-bar linkage |   12     12  18.3s
  Tennis racket    |    1      1   2.9s
Test Summary: | Pass  Total  Time
Joint Limits  |    1      1  3.6s
  Pendulum    |    1      1  3.5s
Test Summary:                            | Pass  Total  Time
Impulse Map                              |   20     20  7.3s
  Displacement Jacobian                  |    4      4  1.4s
  Impulse transform Jacobian: rotational |    8      8  2.9s
  Impulse map                            |    8      8  2.9s
Test Summary:      | Pass  Total  Time
Bodies             |    1      1  1.0s
  Shape convertion |    1      1  0.9s
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8700
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8701
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8702
length(joint_list) = 1
length(joint_list) = 2
length(joint_list) = 3
length(joint_list) = 4
[j.name for j = joint_list] = [:neck_ay, :back_bkx, :back_bky, :back_bkz, :floating_base]
[((get_body(mechanism, j.parent_id)).name, (get_body(mechanism, j.child_id)).name) for j = joint_list] = [(:utorso, :head), (:mtorso, :utorso), (:ltorso, :mtorso), (:pelvis, :ltorso), (:origin, :pelvis)]
Test Summary:             | Pass  Total   Time
Mechanism (Miscellaneous) |   12     12  17.4s
  Pendulum                |   12     12  17.3s
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8703
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8704
Test Summary: | Pass  Total  Time
Simulate      |    6      6  0.5s
  step!       |    2      2  0.0s
  Storage     |    4      4  0.5s
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8705
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8706
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8707
                                                 
n    bvio    rvio     α       μ     |res|∞   |Δ|∞
–––––––––––––––––––––––––––––––––––––––––––––––––
1    2e+0    2e+0    1e+0    0e+0    2e+0    2e+0
2    3e-1    7e-3    1e+0    5e-2    3e-1    3e-1
3    8e-2    9e-4    1e+0    5e-3    7e-2    7e-2
4    7e-3    2e-4    1e+0    1e-4    7e-3    7e-3
5    4e-4    1e-5    1e+0    1e-6    4e-4    4e-4
6    2e-5    6e-7    1e+0    4e-8    2e-5    2e-5
                                                 
n    bvio    rvio     α       μ     |res|∞   |Δ|∞
–––––––––––––––––––––––––––––––––––––––––––––––––
1    2e+0    2e+0    1e+0    0e+0    2e+0    2e+0
2    5e-1    2e-1    9e-1    5e-2    4e-1    4e-1
3    3e-1    1e-1    5e-1    7e-2    2e-1    2e-1
4    2e-1    2e-2    9e-1    3e-2    2e-1    2e-1
5    4e-2    2e-4    1e+0    4e-3    3e-2    3e-2
6    4e-3    1e-4    1e+0    5e-5    4e-3    4e-3
7    2e-4    6e-6    1e+0    6e-7    2e-4    2e-4
8    1e-5    3e-7    1e+0    2e-8    1e-5    1e-5
                                                 
n    bvio    rvio     α       μ     |res|∞   |Δ|∞
–––––––––––––––––––––––––––––––––––––––––––––––––
1    2e+0    3e+0    1e+0    0e+0    3e+0    3e+0
2    5e-1    3e-1    9e-1    4e-2    4e-1    4e-1
3    3e-1    2e-1    4e-1    8e-2    2e-1    2e-1
4    4e-1    3e-2    9e-1    5e-2    3e-1    3e-1
5    6e-2    2e-4    1e+0    8e-3    6e-2    6e-2
6    8e-3    2e-4    1e+0    1e-4    8e-3    8e-3
7    4e-4    1e-5    1e+0    2e-6    4e-4    4e-4
8    2e-5    6e-7    1e+0    5e-8    2e-5    2e-5
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8708
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8709
Test Summary:       | Pass  Total   Time
Visuals             |    3      3  11.9s
  Utilities         |    1      1   2.2s
  MeshCat mechanism |    1      1   7.1s
  URDF mesh         |    1      1   2.5s
Test Summary: | Pass  Total  Time
Utilities     |    5      5  0.0s
Test Summary: | Pass  Total  Time
Mechanisms    |   21     21  8.7s
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8710
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8711
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8712
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8713
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8714
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8715
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8716
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8717
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8718
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8719
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8720
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8721
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8722
Test Summary:   | Pass  Total   Time
Environments    |   26     26  52.7s
  ant           |    2      2   3.9s
  atlas         |    2      2  10.0s
  block         |    2      2   1.3s
  block2d       |    2      2   1.5s
  cartpole      |    2      2   1.2s
  halfcheetah   |    2      2   0.9s
  hopper        |    2      2   2.7s
  panda         |    2      2  12.7s
  pendulum      |    2      2   0.3s
  quadruped     |    2      2   1.4s
  raiberthopper |    2      2   6.5s
  rexhopper     |    2      2   9.5s
  walker        |    2      2   0.5s
     Testing Dojo tests passed 

This PR:

Integrator               |    7      7  9.1s
  ForwardDiff comparison |    7      7  8.8s
Test Summary:                         | Pass  Total     Time
Minimal Coordinates                   |  861    861  4m22.8s
  Get and set position and velocities |  602    602    22.9s
  Minimal to maximal to minimal       |   58     58    41.6s
  Jacobians                           |  201    201  3m16.8s
Test Summary:                 | Pass  Total  Time
Modified Rodrigues Parameters |    9      9  2.6s
  FiniteDiff comparison       |    5      5  2.5s
  axes pair to quaternion     |    4      4  0.1s
Test Summary:                | Pass  Total     Time
Damper                       |  120    120  1m30.0s
  rotational damper jacobian |  120    120  1m29.9s
Test Summary:              | Pass  Total     Time
Jacobian (Solution Matrix) |   32     32  5m15.0s
  Flying                   |   16     16  3m06.1s
  In contact               |   16     16  2m08.7s
Test Summary:                              | Pass  Total      Time
Data                                       |  184    184  11m11.2s
  Get and set data                         |    8      8      6.8s
  Data Jacobian without contact and limtis |   88     88   4m25.7s
  Data Jacobian with contact and limtis    |   88     88   6m34.3s
Test Summary:              | Pass  Total   Time
Collisions                 | 1566   1566  43.4s
  Collision: Sphere-sphere | 1566   1566  43.2s
Test Summary: | Pass  Total     Time
Momentum      |   73     73  4m32.8s
  Box         |    2      2     1.1s
  Pendulum    |    1      1     1.4s
  Humanoid    |    2      2    43.1s
  Atlas       |    2      2    56.4s
  Quadruped   |    2      2  1m46.8s
  Snake       |   32     32    31.5s
  Twister     |   32     32    32.5s
Test Summary: | Pass  Total     Time
Energy        |   42     42  3m27.5s
  Dice        |    1      1     0.8s
  Pendulum    |    1      1     6.1s
  Slider 1    |    3      3     1.5s
  Slider 2    |    1      1     0.5s
  Slider 3    |    1      1     2.6s
  Humanoid    |    1      1    11.3s
  Atlas       |    1      1    59.2s
  Quadruped   |    1      1    20.0s
  Snake       |   16     16    53.0s
  Twister     |   16     16    52.0s
Test Summary:      | Pass  Total   Time
Behavior           |   22     22  54.6s
  Quadruped        |    1      1  20.6s
  Box toss         |    8      8  11.4s
  Four-bar linkage |   12     12  19.6s
  Tennis racket    |    1      1   2.8s
Test Summary: | Pass  Total  Time
Joint Limits  |    1      1  3.6s
  Pendulum    |    1      1  3.6s
Test Summary:                            | Pass  Total     Time
Impulse Map                              |   20     20  1m05.7s
  Displacement Jacobian                  |    4      4     4.0s
  Impulse transform Jacobian: rotational |    8      8    21.1s
  Impulse map                            |    8      8    40.4s
Test Summary:      | Pass  Total  Time
Bodies             |    1      1  1.1s
  Shape convertion |    1      1  1.1s
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8700
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8701
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8702
length(joint_list) = 1
length(joint_list) = 2
length(joint_list) = 3
length(joint_list) = 4
[j.name for j = joint_list] = [:neck_ay, :back_bkx, :back_bky, :back_bkz, :floating_base]
[((get_body(mechanism, j.parent_id)).name, (get_body(mechanism, j.child_id)).name) for j = joint_list] = [(:utorso, :head), (:mtorso, :utorso), (:ltorso, :mtorso), (:pelvis, :ltorso), (:origin, :pelvis)]
Test Summary:             | Pass  Total   Time
Mechanism (Miscellaneous) |   12     12  17.7s
  Pendulum                |   12     12  17.7s
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8703
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8704
Test Summary: | Pass  Total  Time
Simulate      |    6      6  0.2s
  step!       |    2      2  0.0s
  Storage     |    4      4  0.2s
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8705
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8706
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8707
                                                 
n    bvio    rvio     α       μ     |res|∞   |Δ|∞
–––––––––––––––––––––––––––––––––––––––––––––––––
1    2e+0    2e+0    1e+0    0e+0    2e+0    2e+0
2    3e-1    7e-3    1e+0    5e-2    3e-1    3e-1
3    8e-2    9e-4    1e+0    5e-3    7e-2    7e-2
4    7e-3    2e-4    1e+0    1e-4    7e-3    7e-3
5    4e-4    1e-5    1e+0    1e-6    4e-4    4e-4
6    2e-5    6e-7    1e+0    4e-8    2e-5    2e-5
                                                 
n    bvio    rvio     α       μ     |res|∞   |Δ|∞
–––––––––––––––––––––––––––––––––––––––––––––––––
1    2e+0    2e+0    1e+0    0e+0    2e+0    2e+0
2    5e-1    2e-1    9e-1    5e-2    4e-1    4e-1
3    3e-1    1e-1    5e-1    7e-2    2e-1    2e-1
4    2e-1    2e-2    9e-1    3e-2    2e-1    2e-1
5    4e-2    2e-4    1e+0    4e-3    3e-2    3e-2
6    4e-3    1e-4    1e+0    5e-5    4e-3    4e-3
7    2e-4    6e-6    1e+0    6e-7    2e-4    2e-4
8    1e-5    3e-7    1e+0    2e-8    1e-5    1e-5
                                                 
n    bvio    rvio     α       μ     |res|∞   |Δ|∞
–––––––––––––––––––––––––––––––––––––––––––––––––
1    2e+0    3e+0    1e+0    0e+0    3e+0    3e+0
2    5e-1    3e-1    9e-1    4e-2    4e-1    4e-1
3    3e-1    2e-1    4e-1    8e-2    2e-1    2e-1
4    4e-1    3e-2    9e-1    5e-2    3e-1    3e-1
5    6e-2    2e-4    1e+0    8e-3    6e-2    6e-2
6    8e-3    2e-4    1e+0    1e-4    8e-3    8e-3
7    4e-4    1e-5    1e+0    2e-6    4e-4    4e-4
8    2e-5    6e-7    1e+0    5e-8    2e-5    2e-5
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8708
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8709
Test Summary:       | Pass  Total   Time
Visuals             |    3      3  11.7s
  Utilities         |    1      1   2.3s
  MeshCat mechanism |    1      1   7.2s
  URDF mesh         |    1      1   2.2s
Test Summary: | Pass  Total  Time
Utilities     |    5      5  0.0s
Test Summary: | Pass  Total  Time
Mechanisms    |   21     21  9.7s
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8710
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8711
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8712
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8713
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8714
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8715
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8716
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8717
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8718
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8719
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8720
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8721
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8722
Test Summary:   | Pass  Total   Time
Environments    |   26     26  54.4s
  ant           |    2      2   4.0s
  atlas         |    2      2  10.7s
  block         |    2      2   1.5s
  block2d       |    2      2   1.4s
  cartpole      |    2      2   1.2s
  halfcheetah   |    2      2   0.9s
  hopper        |    2      2   2.9s
  panda         |    2      2  12.9s
  pendulum      |    2      2   0.3s
  quadruped     |    2      2   1.4s
  raiberthopper |    2      2   6.5s
  rexhopper     |    2      2   9.8s
  walker        |    2      2   0.6s
     Testing Dojo tests passed 

@thowell
Copy link
Member

thowell commented Mar 25, 2022

Thanks for the PR Jan!

@rejuvyesh what is the correct way to include dependencies for the tests? I was under the impression that now there should be a test/Project.toml file: https://pkgdocs.julialang.org/v1/creating-packages/#Test-specific-dependencies-in-Julia-1.2-and-above

Also, what are your thoughts on FiniteDiff vs ForwardDiff for the tests? Ideally we would just support one.

@rejuvyesh
Copy link
Contributor

Test dependencies

Yeah, test/Project.toml is the way to go!

FiniteDiff vs ForwardDiff for tests

Main thing I am not fully sure about is why do you have to define custom gradient/jacobian functions if AD like ForwardDiff already works? I would assume for performance/efficiency? In that case best to compare against proper AD like ForwardDiff. However, if you see that you are defining gradients for functions where AD like ForwardDiff is not going to work, then supporting FiniteDiff makes sense.

@codecov-commenter
Copy link

codecov-commenter commented Mar 29, 2022

Codecov Report

Merging #35 (104bfa7) into main (1c87f5c) will decrease coverage by 0.11%.
The diff coverage is 100.00%.

❗ Current head 104bfa7 differs from pull request most recent head 303c208. Consider uploading reports for the commit 303c208 to get more accurate results

@@            Coverage Diff             @@
##             main      #35      +/-   ##
==========================================
- Coverage   91.73%   91.61%   -0.12%     
==========================================
  Files          87       87              
  Lines        4477     4295     -182     
==========================================
- Hits         4107     3935     -172     
+ Misses        370      360      -10     
Impacted Files Coverage Δ
src/integrators/integrator.jl 97.43% <100.00%> (ø)
src/joints/rotational/minimal.jl 100.00% <100.00%> (ø)
src/mechanism/state.jl 100.00% <100.00%> (ø)
src/graph/entry.jl 75.00% <0.00%> (-12.50%) ⬇️
src/contacts/utilities.jl 42.85% <0.00%> (-2.60%) ⬇️
src/solver/line_search.jl 95.55% <0.00%> (-1.39%) ⬇️
src/gradients/state.jl 88.98% <0.00%> (-0.79%) ⬇️
src/joints/constraints.jl 89.94% <0.00%> (-0.44%) ⬇️
src/mechanism/traversal.jl 95.00% <0.00%> (-0.35%) ⬇️
src/mechanism/urdf.jl 92.05% <0.00%> (-0.32%) ⬇️
... and 31 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1c87f5c...303c208. Read the comment docs.

@janbruedigam
Copy link
Member Author

Thanks for the clarification @rejuvyesh ! I've readded the test Project.toml.
Moving the remaining tests to ForwardDiff will require some additional work with rotations and mrps, so I would revisit that at a later stage.
I updated the test tolerances, so if tests keep failing they can be loosened again.

@janbruedigam janbruedigam merged commit 42d8db9 into main Mar 29, 2022
@janbruedigam janbruedigam deleted the test_forwarddiff branch March 29, 2022 18:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants