Skip to content

api: unify det() error behavior across all dimensions #82

@acgetchell

Description

@acgetchell

Dependencies

Summary

det() has inconsistent error semantics depending on dimension: D≤4 returns Ok(~0.0) for singular matrices while D≥5 returns Err(LaError::Singular). A user switching from D=4 to D=5 gets a different error model for the same logical operation.

Current State

  • D=0–4: det() dispatches to det_direct(), which is a closed-form cofactor expansion. A singular matrix produces Ok(0.0) (or a near-zero value). No Singular error is possible.
  • D≥5: det() falls through to self.lu(tol).map(|lu| lu.det()). If no pivot exceeds tol, it returns Err(LaError::Singular).

This means:

// D=4: returns Ok(0.0)
Matrix::<4>::zero().det(DEFAULT_PIVOT_TOL) // Ok(0.0)

// D=5: returns Err(Singular)
Matrix::<5>::zero().det(DEFAULT_PIVOT_TOL) // Err(Singular { pivot_col: 0 })

Proposed Changes

Options (need to pick one):

Option A: Always succeed (return Ok(f64))

For D≥5, catch Singular from LU and return Ok(0.0). This makes det() infallible for finite inputs. Pro: simple, consistent. Con: loses the singularity signal for large D.

Option B: Always error on singular

For D≤4, check if the det_direct() result is below some tolerance and return Err(Singular). Pro: consistent error semantics. Con: introduces a tolerance dependency for the closed-form path, and near-zero determinants aren't necessarily singular (could be a well-conditioned matrix with a small determinant).

Option C: Split the API

Rename det_direct() to emphasize it never fails, and document that det() may return Singular only for D≥5. Pro: no behavior change. Con: the inconsistency remains.

Option D: Return Result<f64, LaError> with separate det_direct for D≤4

Keep det() as-is but strengthen documentation to explicitly call out the dimensional behavior. Add a prominent note in the det() rustdoc.

Benefits

  • Consistent, predictable API regardless of dimension
  • Fewer surprises for generic code parameterized over D

Implementation Notes

  • This is a breaking change depending on the chosen option (milestone: v0.5.0)
  • Related: det_direct() is const fn and cannot return Result ergonomically, so it should remain separate regardless
  • Also gated on generic_const_exprs stabilization for the full v0.5.0 API revision

Metadata

Metadata

Assignees

No one assigned

    Labels

    apienhancementNew feature or requestrustPull requests that update rust code

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions