Pose bones are "lines" instead of "ellipses"

[EmoryQ]

Thank you so much for creating this posable OpenPose model extension, it is a great resource for me and others in the community.

I have noticed that the extension currently uses straight lines to represent the "bones". However, I believe that using ellipses for the bones would be more aligned with the data the ControlNet model was trained on, and in my tests, I get better results using the thin ellipse shapes instead of straight rectangular lines for the bones connecting the joints.

This is a link to the ControlNet Paper showing the data it was trained on, on page 15:
https://arxiv.org/pdf/2302.05543.pdf (Page 15)
https://arxiv.org/abs/2302.05543 (Source)

I'm sorry that I don't have the ability to code this myself, I currently use photoshop which can be time consuming when moving each layer manually. I understand that this change may require additional work, but I wanted to share my suggestion in the hopes that someone with the proper abilities could create it.

Thank you for your continued efforts in maintaining and improving the project!

[hnmr293]

Sorry, I don't have any ideas to implement it.
Both original ControlNet paper and Mikubill's implementation use ellipse, not straight line.
In contrast with them, I use THREE.MeshLine for bones in current version of posex,. I don't know how to make MeshLine round...

[EmoryQ]

I asked ChatGPT, this is what it says. Is this helpful?:

To create a THREE.MeshLine that has a thicker middle and thinner ends, you can use a THREE.CatmullRomCurve3 with control points that form an elongated ellipse. You can then use the THREE.MeshLine object to create a line that has varying thickness.

Here's an example code snippet that demonstrates how to create a mesh line that has a thicker middle and thinner ends:

javascript
Copy code
// Define control points for the curve
var curvePoints = [
  new THREE.Vector3( -10, 0, 0 ),
  new THREE.Vector3( -5, 10, 0 ),
  new THREE.Vector3( 5, 10, 0 ),
  new THREE.Vector3( 10, 0, 0 ),
  new THREE.Vector3( 5, -10, 0 ),
  new THREE.Vector3( -5, -10, 0 ),
  new THREE.Vector3( -10, 0, 0 )
];

// Create the curve
var curve = new THREE.CatmullRomCurve3( curvePoints );

// Define the material
var material = new THREE.MeshLineMaterial( {
  color: 0xffffff,
  lineWidth: 0.5, // minimum line width
  resolution: new THREE.Vector2( window.innerWidth, window.innerHeight ),
  sizeAttenuation: true, // line thickness attenuates with distance
  lineWidthStyler: function ( distance ) {
    return 0.5 + 0.5 * Math.sin( 5 * distance ); // vary the line width sinusoidally
  }
} );

// Create the mesh line
var meshLine = new THREE.MeshLine();
meshLine.setGeometry( curve.getPoints( 50 ), function( p ) {
  return 1 - p; // set line width based on point position (0 = start, 1 = end)
} );

// Create the mesh
var mesh = new THREE.Mesh( meshLine.geometry, material );

// Add the mesh to the scene
scene.add( mesh );

In this example, we define the control points for a THREE.CatmullRomCurve3 that forms an elongated ellipse. We then create a THREE.MeshLine object and set its geometry using the curve's points. We also set a lineWidthStyler function for the material, which varies the line width sinusoidally based on the distance from the camera. Finally, we create a THREE.Mesh object using the mesh line's geometry and material, and add it to the scene.

You can adjust the control points of the curve to create a different shape, and adjust the lineWidth and lineWidthStyler functions to control the thickness and shape of the line.

[EmoryQ]

Also, I'm not sure if you can use this parameter?

https://github.com/spite/THREE.MeshLine#create-a-meshline-and-assign-the-points
Note: .setPoints accepts a second parameter, which is a function to define the width in each point along the line. By default that value is 1, making the line width 1 * lineWidth in the material.

> // p is a decimal percentage of the number of points
// ie. point 200 of 250 points, p = 0.8
line.setPoints(geometry, p => 2); // makes width 2 * lineWidth
line.setPoints(geometry, p => 1 - p); // makes width taper
line.setPoints(geometry, p => 2 + Math.sin(50 * p)); // makes width sinusoidal
> ```

So, to make this into the ellipse shape using this property, I think it would look like:

line.setPoints(geometry,p => W * 2 * p * (1 - p));

Just replace the 'W' with the number that is the maximum width at the center of the line.

I'm sorry that I don't know more about this so I could be more helpful...

[EmoryQ]

This line of code would work on line 565 in posex.js, except now it seems to only evaluate the thickness at the two points at the end of the lines. The lines would need more points, (maybe just a third center point?) to change the thickness along the line... I think...

limbs[i].geometry.setPoints([from.getWorldPosition(v1), to.getWorldPosition(v2)], p => (LIMB_SIZE * 2 * p * (1 - p)) / camera.zoom);

[hnmr293]

Thanks for your great advice, I've added new option, Limb Width and Elliptic Limbs.
Now we have elliptic bones. Thank you!

note:
I use {1 - (2p-1)^2}^0.5 for limb width according to the formula of ellipse.

[EmoryQ]

This is so much better than what I had in mind! Being able to change the width and Elliptic/Line state opens up so much versatility in the creative possibilities. Since you added this type of control, I've been experimenting with how the ControlNet model responds to these types of changes in the skeleton's form, and I think it would be useful to be able to change the circle/joint size too.

This has also inspired me to start thinking about how I can experiment even further through changing the joint and limb colors in photoshop, hiding/showing different and additional limbs and joints. New discoveries in interesting and unexpected ways.

Thanks for making this cool toy/tool and for adding this ellipse feature in an enhanced and non-constraining way.