Add human joint constraint example #1016

jslee02 · 2018-02-26T04:20:25Z

This PR is a revised version of #919.

Summary of changes from #919:

Make tiny-dnn as a dependency of humanJointLimits example. In absence of tiny-dnn installed, the example target is not built. If you are on macOS, you can install tiny-dnn as brew install dartsim/dart/tiny-dnn. Otherwise, you should build and install it from source.
Remove C++14 requirements from DART libraries. But humanJointLimits example requires it.
Move Human[~]JointLimitConstraint classes to the example directory since they are application specific at this moment.
Fix invalid box dimensions (i.e., negative size in z-coordinate).

Depends on #1015 (diff from #1015)

Before creating a pull request

Document new methods and classes

Before merging a pull request

Set version target by selecting a milestone on the right side
Summarize this change in CHANGELOG.md
Code format for new files
Install tiny-dnn on CI tests to test build of this example (only macOS for now)
Update README.md of the example adding author and paper info

# Conflicts: # CMakeLists.txt

This reverts commit 6c04669.

# Conflicts: # cmake/DARTFindDependencies.cmake

jslee02 · 2018-02-26T04:22:35Z

@jyf588 I've revised your PR to be able to merge it into DART 6.4 seamlessly. Feedback from you would be appreciated!

codecov · 2018-02-26T05:20:43Z

Codecov Report

Merging #1016 into release-6.4 will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff              @@
##           release-6.4    #1016   +/-   ##
============================================
  Coverage        56.58%   56.58%           
============================================
  Files              314      314           
  Lines            24312    24312           
============================================
  Hits             13758    13758           
  Misses           10554    10554

jyf588 · 2018-02-26T05:58:54Z

@jslee02 That sounds awesome! I just changed my code base link in this paper from my own github to this pull request. (2 hours before the Final Revision Deadline. You created this just at the time :) )

I am currently working very hard towards IROS. Will definitely take a look immediately after March 1st. Let me know if you need any information from me (e.g. documentations)!

jslee02 · 2018-02-26T06:08:39Z

No worries. This PR is not urgent. I just didn't want to leave the other PR alone like forever. 😄

Oh, if there is a chance, would you change the citation of DART to this paper (BibTeX)?

Good luck to the IROS paper!

jyf588 · 2018-02-26T06:19:21Z

@jslee02 Have changed the citation previously :D

…constraints_6.4_js

mxgrey · 2018-02-27T17:59:46Z

I should have a chance to review this later today.

Right off the bat, I like the idea of having it as an example. That way, we get the benefit of making it publicly available, easily accessible, and an effective starting point for people who are interested in the topic of human-like joint limits, but at the same time we don't need to worry about rigorously supporting the API + implementation or developing the feature to be comprehensive and generalized.

mxgrey

Overall this looks really good to me. Just a few remarks:

It looks like some forward declarations accidentally sneaked into one of the publicly installed headers. I left an in-line comment for this. This is the only important change to make; the rest of my comments are just passing thoughts.
I wonder if data/neuralnets/net-larm and net-lleg should maybe go into a data/humanJointLimits/ directory instead? I don't mind keeping them where they are, but I suspect putting them in a directory that's named after the example might make it more clear what their purpose is. I don't imagine it's likely that we'll be storing enough general-purpose neural networks to justify a data directory dedicated to neural networks, but I could be wrong.

mxgrey · 2018-02-28T05:31:50Z

dart/constraint/SmartPointer.hpp

@@ -51,6 +51,8 @@ DART_COMMON_MAKE_SHARED_WEAK(LCPSolver)

 DART_COMMON_MAKE_SHARED_WEAK(BallJointConstraint)
 DART_COMMON_MAKE_SHARED_WEAK(WeldJointConstraint)
+DART_COMMON_MAKE_SHARED_WEAK(HumanArmJointLimitConstraint)


Since these types are only defined in example code, I think they should go in a header of that example code instead of a public DART header.

mxgrey · 2018-02-28T05:44:27Z

examples/humanJointLimits/humanJointLimits.cpp

+    joint->setPositionLimitEnforced(true);
+  }
+
+  //    world->getSkeleton("arm")->enableSelfCollisionCheck();


Maybe put a comment on why these lines are commented out, and why someone might want to comment/uncomment these lines?

jslee02 · 2018-02-28T15:36:22Z

@mxgrey In the last commits,

humanJointLimits classes are removed from DART namespace (including the pointer declaration)
the NN files are moved to /dart/humanJointLimits/

@jyf588 I updated the copyright and README.md. Feel free to update README.md.

jslee02 · 2018-03-14T23:23:35Z

@jyf588 This is a gentle reminder. Have you found anything want to change in this PR? I'd be happy to merge this if this looks good to you as well.

Nit: Like @mxgrey said, it would be good to either we remove the commented code (e.g., this) or leave a comment why we need this commented code here.

jyf588 · 2018-03-15T01:09:00Z

@jslee02 Yes, it looks good to me, and we can just delete the commented code.

I just have not get the time to add some documentation to README.md, but I will do that before tonight. My apologies for the delay here.

I seem to not having push access to the branch. Can I just put the new texts for README here, and ask you to copy and paste it to README?

jslee02 · 2018-03-15T03:47:37Z

I seem to not having push access to the branch. Can I just put the new texts for README here, and ask you to copy and paste it to README?

Sure, you could put the changes here, or you could create a branch from example/human-joint-constraints_6.4_js branch of this repo in your fork of DART repo and then create a PR targeting this branch. Either way would be fine.

jyf588 · 2018-03-15T05:02:46Z

@jslee02 I think we are good to merge once you think the new texts added to README are okay:

This example program sets realistic human joint limits to a simulated human agent, so that the agent’s movements under laws of physics will never exceed human range of motion.

Specifically, naively setting upper and lower bounds for each DoF is far from accurate for simulating our four limbs. (For example, our elbow has a smaller range when the arm is behind the torso.) We thus represent the joint limit constraints for the limbs in more general, mutual-dependent forms C_{arm}(q_arm) > 0.5 and C_{leg}(q_leg) > 0.5, where q is the joint configuration for the whole limb. (For the arm excluding the hand, we consider q_arm as 3DoF in shoulder plus 1DoF in elbow; for the leg, q_leg is 3DoF in hip, 1DoF in knee and 2DoF in ankle.) Just like how upper/lower bounds are handled in DART, when C(q) is about to decrease below 0.5 at next time step, a constraint force in dC/dq direction is generated by LCP solver to push q back to valid region.

The C(q) functions are trained neural nets from real human data. In this way, we do not need to use sophisticated biomechanics models, and both C(q) and dC/dq can be quickly evaluated through forward evaluation and back-propagation of the neural nets.

The code for training neural nets is here: https://github.com/jyf588/Human-Joint-Constraints-Training

————————————————————————
Implementation details: (For more details, please refer to the paper.)

The neural nets are trained specific to a certain joint configuration space. In the .skel file, one should not modify the axis orders for shoulder/hip Euler joints, or the rest poses defining q=0, unless one wants to modify the training code and retrain the neural nets as well.

Specifically, the axis order for all Euler joints are set to zxy. The rest pose is almost defined the same as the usual human rest pose, except for the shoulder rotation - for humans, if you start to bend your elbow from rest pose, your lower arm will start to move to the other side of torso besides lifting up. In our model, however, rest position for shoulder rotation in defined such that when bending elbow, the agent’s lower up will only lift up, while keeping perpendicular to the torso plane.

There are only two trained neural nets for the left arm and leg respectively. For right arm and leg, we just mirror the q values before feeding them into neural nets. For users, one just need to turn on an additional isMirror flag when specifying a right limb joint-limit constraint.
The “left” and “right” are defined from the perspective of the agent itself.
We do not directly input the q values into neural nets, instead, we input sin(q) and cos(q). As such, the trained functions still work when q exceeds 2pi range.
We still define some naive upper and lower bounds in the .skel file to exclude clearly infeasible poses. Be sure to make them in effect in your code by applying joint->setPositionLimitEnforced(true); to each joint.

jslee02 · 2018-03-15T05:32:57Z

@jyf588 Thanks! I added the text to README.md.

jyf588 · 2018-03-15T14:28:16Z

@jslee02 Great! Thanks!

jslee02 and others added 19 commits February 25, 2018 19:16

Add in-source build script to examples

fa9dfc9

Add in-source build script to tutorials

b9f7e71

Add human meshes and skel files

be77d75

update skel files

69ed7db

upload neural net weight files

3858617

add human arm & leg joint position constraints

b048fb3

add external headers tiny-dnn

c47ac0f

add an example on human joint limits

9941bf0

# Conflicts: # CMakeLists.txt

revise typos

9f938b3

add external library cereal: used by tiny-dnn

b2f302e

Revert "add external library cereal: used by tiny-dnn"

952ce67

This reverts commit 6c04669.

Remove tiny-dnn from DART codebase

d612a5a

Fix compile errors of Human[~]JointLimitConstraint

ade6c8d

Add Threads as required dependency

b9787f2

# Conflicts: # cmake/DARTFindDependencies.cmake

Fix negative box sizes in kima_human_edited.skel

f1fe204

Move Human[~]JointLimitConstraint to example directory

9e7e865

Revert C++14 requirement from DART

1378ab3

Use utils instead of io

3d5199f

Add in-source build CMake script for humanJointLimits

f694cf3

jslee02 added this to the DART 6.4.0 milestone Feb 26, 2018

jslee02 requested review from mxgrey and karenliu February 26, 2018 04:20

jslee02 mentioned this pull request Feb 26, 2018

Human joint constraints #919

Closed

jslee02 added comp: examples labels Feb 26, 2018

jslee02 added 4 commits February 25, 2018 20:50

Add missing rigidCubes example in in-source build list

94c37de

Set minimum requirement DART version to 6.4 for examples/tutorials

ab5bc4c

Move Threads dependencies to humanJointLimits example

388f0cf

Code format per clang-format

0e1339e

jslee02 added 4 commits February 25, 2018 23:59

Fix setting CMake variable

a079111

Fix adding rigidCubes example

a69ce11

Merge branch 'examples/multiple_cmakelists' into example/human-joint-…

661f66b

…constraints_6.4_js

Merge branch 'release-6.4' into example/human-joint-constraints_6.4_js

ad0e4d1

mxgrey requested changes Feb 28, 2018

View reviewed changes

jslee02 added 7 commits February 28, 2018 06:50

Remove pointer deceleration of examples classes

72f84c7

Move NN files to data/humanJointLimits

c428538

Enable serialization of TinyDNN to load NN files

b9e5642

Code format

68ea963

Move examples classes from DART namespace to global

05718ba

Update copyright

7c743ce

Add simple note to humanJointLimits/README.md

c2cae3c

Update changelog

3f2c2cd

mxgrey approved these changes Feb 28, 2018

View reviewed changes

Merge branch 'release-6.4' into example/human-joint-constraints_6.4_js

552b878

Update README.md

2ef2eba

jslee02 merged commit 2f40699 into release-6.4 Mar 15, 2018

jslee02 deleted the example/human-joint-constraints_6.4_js branch March 15, 2018 14:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add human joint constraint example #1016

Add human joint constraint example #1016

jslee02 commented Feb 26, 2018 •

edited

Loading

jslee02 commented Feb 26, 2018 •

edited

Loading

codecov bot commented Feb 26, 2018 •

edited

Loading

jyf588 commented Feb 26, 2018

jslee02 commented Feb 26, 2018

jyf588 commented Feb 26, 2018

mxgrey commented Feb 27, 2018

mxgrey left a comment

mxgrey Feb 28, 2018

mxgrey Feb 28, 2018

jslee02 commented Feb 28, 2018

jslee02 commented Mar 14, 2018

jyf588 commented Mar 15, 2018 •

edited

Loading

jslee02 commented Mar 15, 2018

jyf588 commented Mar 15, 2018 •

edited

Loading

jslee02 commented Mar 15, 2018

jyf588 commented Mar 15, 2018

Add human joint constraint example #1016

Add human joint constraint example #1016

Conversation

jslee02 commented Feb 26, 2018 • edited Loading

jslee02 commented Feb 26, 2018 • edited Loading

codecov bot commented Feb 26, 2018 • edited Loading

Codecov Report

jyf588 commented Feb 26, 2018

jslee02 commented Feb 26, 2018

jyf588 commented Feb 26, 2018

mxgrey commented Feb 27, 2018

mxgrey left a comment

Choose a reason for hiding this comment

mxgrey Feb 28, 2018

Choose a reason for hiding this comment

mxgrey Feb 28, 2018

Choose a reason for hiding this comment

jslee02 commented Feb 28, 2018

jslee02 commented Mar 14, 2018

jyf588 commented Mar 15, 2018 • edited Loading

jslee02 commented Mar 15, 2018

jyf588 commented Mar 15, 2018 • edited Loading

jslee02 commented Mar 15, 2018

jyf588 commented Mar 15, 2018

jslee02 commented Feb 26, 2018 •

edited

Loading

jslee02 commented Feb 26, 2018 •

edited

Loading

codecov bot commented Feb 26, 2018 •

edited

Loading

jyf588 commented Mar 15, 2018 •

edited

Loading

jyf588 commented Mar 15, 2018 •

edited

Loading