New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add human joint constraint example #1016

Merged
merged 39 commits into from Mar 15, 2018

Conversation

Projects
3 participants
@jslee02
Member

jslee02 commented Feb 26, 2018

This PR is a revised version of #919.

Summary of changes from #919:

  • Make tiny-dnn as a dependency of humanJointLimits example. In absence of tiny-dnn installed, the example target is not built. If you are on macOS, you can install tiny-dnn as brew install dartsim/dart/tiny-dnn. Otherwise, you should build and install it from source.
  • Remove C++14 requirements from DART libraries. But humanJointLimits example requires it.
  • Move Human[~]JointLimitConstraint classes to the example directory since they are application specific at this moment.
  • Fix invalid box dimensions (i.e., negative size in z-coordinate).

Depends on #1015 (diff from #1015)


Before creating a pull request

  • Document new methods and classes

Before merging a pull request

  • Set version target by selecting a milestone on the right side
  • Summarize this change in CHANGELOG.md
  • Code format for new files
  • Install tiny-dnn on CI tests to test build of this example (only macOS for now)
  • Update README.md of the example adding author and paper info

@jslee02 jslee02 added this to the DART 6.4.0 milestone Feb 26, 2018

@jslee02 jslee02 requested a review from mxgrey Feb 26, 2018

@jslee02 jslee02 added this to To do in DART 6.4 via automation Feb 26, 2018

@jslee02 jslee02 requested a review from karenliu Feb 26, 2018

@jslee02

This comment has been minimized.

Member

jslee02 commented Feb 26, 2018

@jyf588 I've revised your PR to be able to merge it into DART 6.4 seamlessly. Feedback from you would be appreciated!

@jslee02 jslee02 referenced this pull request Feb 26, 2018

Closed

Human joint constraints #919

@jslee02 jslee02 moved this from To do to In progress in DART 6.4 Feb 26, 2018

@jslee02

This comment has been minimized.

Member

jslee02 commented Feb 26, 2018

No worries. This PR is not urgent. I just didn't want to leave the other PR alone like forever. 😄

Oh, if there is a chance, would you change the citation of DART to this paper (BibTeX)?

Good luck to the IROS paper!

@jyf588

This comment has been minimized.

jyf588 commented Feb 26, 2018

@jslee02 Have changed the citation previously :D

jslee02 added some commits Feb 26, 2018

@mxgrey

This comment has been minimized.

Member

mxgrey commented Feb 27, 2018

I should have a chance to review this later today.

Right off the bat, I like the idea of having it as an example. That way, we get the benefit of making it publicly available, easily accessible, and an effective starting point for people who are interested in the topic of human-like joint limits, but at the same time we don't need to worry about rigorously supporting the API + implementation or developing the feature to be comprehensive and generalized.

@mxgrey

Overall this looks really good to me. Just a few remarks:

  1. It looks like some forward declarations accidentally sneaked into one of the publicly installed headers. I left an in-line comment for this. This is the only important change to make; the rest of my comments are just passing thoughts.

  2. I wonder if data/neuralnets/net-larm and net-lleg should maybe go into a data/humanJointLimits/ directory instead? I don't mind keeping them where they are, but I suspect putting them in a directory that's named after the example might make it more clear what their purpose is. I don't imagine it's likely that we'll be storing enough general-purpose neural networks to justify a data directory dedicated to neural networks, but I could be wrong.

@@ -51,6 +51,8 @@ DART_COMMON_MAKE_SHARED_WEAK(LCPSolver)
DART_COMMON_MAKE_SHARED_WEAK(BallJointConstraint)
DART_COMMON_MAKE_SHARED_WEAK(WeldJointConstraint)
DART_COMMON_MAKE_SHARED_WEAK(HumanArmJointLimitConstraint)

This comment has been minimized.

@mxgrey

mxgrey Feb 28, 2018

Member

Since these types are only defined in example code, I think they should go in a header of that example code instead of a public DART header.

joint->setPositionLimitEnforced(true);
}
// world->getSkeleton("arm")->enableSelfCollisionCheck();

This comment has been minimized.

@mxgrey

mxgrey Feb 28, 2018

Member

Maybe put a comment on why these lines are commented out, and why someone might want to comment/uncomment these lines?

@jslee02

This comment has been minimized.

Member

jslee02 commented Feb 28, 2018

@mxgrey In the last commits,

  • humanJointLimits classes are removed from DART namespace (including the pointer declaration)
  • the NN files are moved to /dart/humanJointLimits/

@jyf588 I updated the copyright and README.md. Feel free to update README.md.

@mxgrey

mxgrey approved these changes Feb 28, 2018

@jslee02

This comment has been minimized.

Member

jslee02 commented Mar 14, 2018

@jyf588 This is a gentle reminder. Have you found anything want to change in this PR? I'd be happy to merge this if this looks good to you as well.

Nit: Like @mxgrey said, it would be good to either we remove the commented code (e.g., this) or leave a comment why we need this commented code here.

@jyf588

This comment has been minimized.

jyf588 commented Mar 15, 2018

@jslee02 Yes, it looks good to me, and we can just delete the commented code.

I just have not get the time to add some documentation to README.md, but I will do that before tonight. My apologies for the delay here.

I seem to not having push access to the branch. Can I just put the new texts for README here, and ask you to copy and paste it to README?

@jslee02

This comment has been minimized.

Member

jslee02 commented Mar 15, 2018

I seem to not having push access to the branch. Can I just put the new texts for README here, and ask you to copy and paste it to README?

Sure, you could put the changes here, or you could create a branch from example/human-joint-constraints_6.4_js branch of this repo in your fork of DART repo and then create a PR targeting this branch. Either way would be fine.

@jyf588

This comment has been minimized.

jyf588 commented Mar 15, 2018

@jslee02 I think we are good to merge once you think the new texts added to README are okay:

This example program sets realistic human joint limits to a simulated human agent, so that the agent’s movements under laws of physics will never exceed human range of motion.

Specifically, naively setting upper and lower bounds for each DoF is far from accurate for simulating our four limbs. (For example, our elbow has a smaller range when the arm is behind the torso.) We thus represent the joint limit constraints for the limbs in more general, mutual-dependent forms C_{arm}(q_arm) > 0.5 and C_{leg}(q_leg) > 0.5, where q is the joint configuration for the whole limb. (For the arm excluding the hand, we consider q_arm as 3DoF in shoulder plus 1DoF in elbow; for the leg, q_leg is 3DoF in hip, 1DoF in knee and 2DoF in ankle.) Just like how upper/lower bounds are handled in DART, when C(q) is about to decrease below 0.5 at next time step, a constraint force in dC/dq direction is generated by LCP solver to push q back to valid region.

The C(q) functions are trained neural nets from real human data. In this way, we do not need to use sophisticated biomechanics models, and both C(q) and dC/dq can be quickly evaluated through forward evaluation and back-propagation of the neural nets.

The code for training neural nets is here: https://github.com/jyf588/Human-Joint-Constraints-Training

————————————————————————
Implementation details: (For more details, please refer to the paper.)

  1. The neural nets are trained specific to a certain joint configuration space. In the .skel file, one should not modify the axis orders for shoulder/hip Euler joints, or the rest poses defining q=0, unless one wants to modify the training code and retrain the neural nets as well.

Specifically, the axis order for all Euler joints are set to zxy. The rest pose is almost defined the same as the usual human rest pose, except for the shoulder rotation - for humans, if you start to bend your elbow from rest pose, your lower arm will start to move to the other side of torso besides lifting up. In our model, however, rest position for shoulder rotation in defined such that when bending elbow, the agent’s lower up will only lift up, while keeping perpendicular to the torso plane.

  1. There are only two trained neural nets for the left arm and leg respectively. For right arm and leg, we just mirror the q values before feeding them into neural nets. For users, one just need to turn on an additional isMirror flag when specifying a right limb joint-limit constraint.

  2. The “left” and “right” are defined from the perspective of the agent itself.

  3. We do not directly input the q values into neural nets, instead, we input sin(q) and cos(q). As such, the trained functions still work when q exceeds 2pi range.

  4. We still define some naive upper and lower bounds in the .skel file to exclude clearly infeasible poses. Be sure to make them in effect in your code by applying joint->setPositionLimitEnforced(true); to each joint.

@jslee02

This comment has been minimized.

Member

jslee02 commented Mar 15, 2018

@jyf588 Thanks! I added the text to README.md.

@jyf588

This comment has been minimized.

jyf588 commented Mar 15, 2018

@jslee02 Great! Thanks!

@jslee02 jslee02 merged commit 2f40699 into release-6.4 Mar 15, 2018

3 of 5 checks passed

continuous-integration/appveyor/branch Waiting for AppVeyor build to complete
Details
continuous-integration/appveyor/pr Waiting for AppVeyor build to complete
Details
Codacy/PR Quality Review Good work! A positive pull request.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

DART 6.4 automation moved this from In progress to Done Mar 15, 2018

@jslee02 jslee02 deleted the example/human-joint-constraints_6.4_js branch Mar 15, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment