Tips and Tricks
Convex objects may exhibit unstable behaviour, such as wiggling/jitter. For example, try dropping a few convex hulls in DemoTrimesh.
The instability in convex collisions assumed to be caused by the way GJK algorithm works. It always reports a single contact and this is generally fine for round-shaped, sphere-like convex hulls, but fails for cube-like shapes with large flat surfaces. It is nicely described in this presentation.
In case of DemoTrimesh we have probably the worst-case scenario as the trimesh consists of large triangles collides with large convex cubes.
For convex - trimesh collisions, applying some contact perturbation as described in the above-mentioned presentation seems to improve the stability significantly. For an example, please see what happens to the 3rd cube in DemoTrimesh with and without perturbation when contact visualization is turned on. Changing the value of CONTACT_PERTURBATION_ANGLE in the CollisionLibccd class to 0 shows the difference. Without perturbations only a single contact is reported which is a different vertex of the cube in each frame leading to the cube slowly sliding down the triangle. With perturbations 3 contact are reported making the cube stay in place.
The problem affects also other collisions that use libCCD and GJK i.e. convex vs convex, convex vs sphere etc. As soon as a convex shape collides with another convex shape, the same stability problems occur due to a single contact being reported at a time. As long as the shapes are round enough a single contact will do, but for cubes it is most likely not enough.