PhysicsScene.raycast / boxCast / sphereCast / capsuleCast don't normalize direction before PhysX, causing incorrect scan distance

[zhuxudong]

Problem

PhysicsScene.raycast, boxCast, sphereCast, and capsuleCast pass the caller-supplied direction vector directly to the PhysX backend without normalization. PhysX's PxScene::raycast/sweep APIs require unitDir to be a unit vector (per NVIDIA PhysX docs); passing a non-unit direction is undefined behavior. In practice, PhysX treats the distance parameter as a t-scale along the supplied vector, so the effective physical scan distance becomes |dir| × distance instead of distance.

This silently causes:

• Raycasts to scan less than the requested distance when |dir| < 1
• Raycasts to scan more than the requested distance when |dir| > 1
• Inconsistency with Ray class documentation, which states direction is expected to be normalized

Evidence

Ray class docs say the direction MUST be normalized

https://github.com/galacean/engine/blob/main/packages/math/src/Ray.ts#L13

/** The normalized direction of the ray. */
readonly direction: Vector3 = new Vector3();

But the constructor does not normalize, and PhysicsScene.raycast does not enforce it either.

Non-unit direction passed straight through to PhysX

PhysicsScene.raycast → _nativePhysicsScene.raycast (core, around line 182):

const result = this._nativePhysicsScene.raycast(
  ray,
  distance,
  preFilter,
  hitResult ? this._createHitCallback(hitResult) : undefined
);

PhysX backend passes ray.direction raw to PhysX WASM (packages/physics-physx/src/PhysXPhysicsScene.ts):

const result = this._pxScene.raycastSingle(
  ray.origin,
  ray.direction,   // <-- not normalized
  distance,
  pxHitResult,
  this._pxFilterData,
  pxRaycastCallback
);

Same pattern in boxCast, sphereCast, capsuleCast — all pass user-supplied direction raw.

Impact

Observed in a Cocos-to-Galacean migrated game: a character-queue script built a ray with direction forward.clone().scale(dt * -20) (length ≈ 0.32 at 60 fps) and called physics.raycast(ray, 1, mask). Expected scan length was 1 unit; actual scan length was 0.32. Queue spacing ended up ~3× tighter than in Cocos.

For comparison, Cocos's Bullet backend normalizes implicitly via Ray.computeHit(out, maxDistance) → Vec3.normalize(out, this.d) before calling bt.CollisionWorld_rayTest(world, from, to, cb), so Cocos-authored non-unit rays "just work". Galacean does not have an equivalent guard.

Expected behavior

PhysicsScene.raycast/boxCast/sphereCast/capsuleCast should normalize the direction (or assert unit) before handing it to the backend, so that distance always reflects physical scan length — matching both the Ray docstring and PhysX's contract.

Suggested fix

Normalize at the PhysicsScene API boundary (not the backend) so all native backends see a unit direction. Rough sketch for raycast:

raycast(ray: Ray, ...): boolean {
  const d = ray.direction;
  const lenSq = d.x * d.x + d.y * d.y + d.z * d.z;
  if (lenSq > 0 && Math.abs(lenSq - 1) > 1e-6) {
    const len = Math.sqrt(lenSq);
    d.x /= len; d.y /= len; d.z /= len;
  }
  // ... existing logic
}

Same for the direction parameter of boxCast/sphereCast/capsuleCast.

Reproduction

const origin = new Vector3(0, 0, 0);
const nonUnitDir = new Vector3(0.1, 0, 0); // length 0.1
const ray = new Ray(origin, nonUnitDir);
// A collider placed at x=1 and distance=1 should be hit; currently it won't,
// because the effective physical scan is 0.1 × 1 = 0.1 units.
const hit = scene.physics.raycast(ray, 1);

Workaround (for current users)

Monkey-patch PhysicsScene.prototype.raycast at runtime to normalize ray.direction before calling the original method.