[tmva][sofie] Improve numerical stability and overflow protection for Softplus operator

[AdityaDRathore]

Explain what you would like to see improved and how.

The current Softplus implementation (recently merged via #21050) relies on the ROperator_BasicUnary trait with the standard formula std::log(std::exp(x) + 1). While functionally correct for small inputs, this implementation has two significant numerical limitations in edge cases for scientific computing:

• Overflow: For inputs $x > 88.72$ (in float32), std::exp(x) overflows to infinity, causing the operator to return inf. The expected behavior for large inputs is linear ($f(x) \approx x$).
• Precision Loss: For large negative inputs, exp(x) becomes very small. Computing 1 + exp(x) discards significant digits before the log calculation, leading to precision errors.

Leading inference frameworks (PyTorch, TensorFlow) mitigate this by using a threshold for the linear region and log1p for the curve.

Proposal

I propose updating the Softplus logic to use numerical stability safeguards:

1. Thresholding: For $x \ge 20.0f$, return $x$.
   • At $x=20$, $\ln(1+e^{20}) \approx 20.000000002$, which is indistinguishable from $20.0$ in float32.
   • This prevents the exp() overflow completely for large positive values.
2. Precision: For $x < 20.0f$, use std::log1p(std::exp(x)).
   • std::log1p(z) computes $\ln(1+z)$ accurately even when $z \approx 0$, preserving precision for negative inputs.

Proposed Logic:

// Threshold 20.0f defined as exact hexfloat 0x1.4p+4f
return (x >= 20.0f) ? x : std::log1p(std::exp(x));

Action Plan

I have a complete implementation ready (from PR #21023) that includes:

• Bit-exact Hexfloat constants for the threshold.
• Comprehensive GTest suite (TestSofieSoftplus.cxx) verifying:
  • Overflow protection (inputs > 89 return $x$, not Inf).
  • Stability in the linear region.
  • Correct usage of std::log1p vs std::log.

I can submit a Pull Request to patch the ROperator_BasicUnary implementation to include this logic and integrate the validation suite.

ROOT version

master

Installation method

build from source

Operating system

Linux (Ubuntu 24.04)

Additional context

References & Prior Art:

• PyTorch Standard: PyTorch's Softplus implementation utilizes a threshold parameter (default 20) to revert to a linear function for large inputs to avoid overflow.
  • Source: https://pytorch.org/docs/stable/generated/torch.nn.Softplus.html
• TensorFlow: Similarly uses softplus(x) = x for x > threshold.

Related PRs:

• Current Implementation: [tmva][sofie] Add Softplus operator #21050 (Merged) - Established the operator.
• Proposed Code Source: [tmva][sofie] Add Softplus operator for ONNX Inference #21023 (Closed) - I have a fully tested branch with the log1p and threshold logic ready to port.

cc: @lmoneta @bellenot @sanjibansg

[sanjibansg]

Hi @AdityaDRathore ,
Thanks for creating this issue. I agree to the improvements you proposed. It would be great if you make changes to your PR #21023 where you add such checks within the ROperator_BasicUnary file.

Thanks,
Sanjiban

[AdityaDRathore]

Hi @AdityaDRathore , Thanks for creating this issue. I agree to the improvements you proposed. It would be great if you make changes to your PR #21023 where you add such checks within the ROperator_BasicUnary file.

Thanks, Sanjiban

Understood @sanjibansg. I will reopen PR #21023, rebase it on the latest master and implement the stability logic directly inside ROperator_BasicUnary.hxx via the traits specialization. I'll also include the comprehensive GTest suite. Updating the PR now