[InstCombine] Add constant folding for hypot library calls

[wentywenty]

This patch adds constant folding optimization for hypot(x, y) when both arguments are constant values.

Implementation details:

• Only supports float and double precision, as std::hypot lacks support for extended precision types (fp128, x86_fp80, ppc_fp128, fp16)
• Respects errno semantics: folds when the function is marked as not accessing memory (intrinsic), fast-math is enabled, or when the result is finite / inputs are non-finite (hypot only sets errno on overflow with finite inputs)
• Follows the same pattern as other LibCall optimizations like remquo

The optimization handles:

• Basic cases: hypot(3.0, 4.0) -> 5.0
• Special values: Inf and NaN propagation
• Overflow: preserves calls in strict mode, folds in fast-math mode
• Strict FP mode: preserves calls to maintain FP exception semantics

Tests cover all major scenarios including precision handling, special values, overflow behavior, and strictfp mode.

[github-actions]

Thank you for submitting a Pull Request (PR) to the LLVM Project!

This PR will be automatically labeled and the relevant teams will be notified.

If you wish to, you can add reviewers by using the "Reviewers" section on this page.

If this is not working for you, it is probably because you do not have write permissions for the repository. In which case you can instead tag reviewers by name in a comment by using @ followed by their GitHub username.

If you have received no comments on your PR for a week, you can request a review by "ping"ing the PR by adding a comment “Ping”. The common courtesy "ping" rate is once a week. Please remember that you are asking for valuable time from other developers.

If you have further questions, they may be answered by the LLVM GitHub User Guide.

You can also ask questions in a comment on this PR, on the LLVM Discord or on the forums.

[llvmbot]

@llvm/pr-subscribers-llvm-transforms

Author: wentywenty (wentywenty)

Changes

This patch adds constant folding optimization for hypot(x, y) when both arguments are constant values.

Implementation details:

• Only supports float and double precision, as std::hypot lacks support for extended precision types (fp128, x86_fp80, ppc_fp128, fp16)
• Respects errno semantics: folds when the function is marked as not accessing memory (intrinsic), fast-math is enabled, or when the result is finite / inputs are non-finite (hypot only sets errno on overflow with finite inputs)
• Follows the same pattern as other LibCall optimizations like remquo

The optimization handles:

• Basic cases: hypot(3.0, 4.0) -> 5.0
• Special values: Inf and NaN propagation
• Overflow: preserves calls in strict mode, folds in fast-math mode
• Strict FP mode: preserves calls to maintain FP exception semantics

Tests cover all major scenarios including precision handling, special values, overflow behavior, and strictfp mode.

---

Full diff: https://github.com/llvm/llvm-project/pull/171863.diff

3 Files Affected:

• (modified) llvm/include/llvm/Transforms/Utils/SimplifyLibCalls.h (+1)
• (modified) llvm/lib/Transforms/Utils/SimplifyLibCalls.cpp (+38)
• (added) llvm/test/Transforms/InstCombine/hypot.ll (+89)

diff --git a/llvm/include/llvm/Transforms/Utils/SimplifyLibCalls.h b/llvm/include/llvm/Transforms/Utils/SimplifyLibCalls.h
index 64d2512308935..7de7ff52c6ac2 100644
--- a/llvm/include/llvm/Transforms/Utils/SimplifyLibCalls.h
+++ b/llvm/include/llvm/Transforms/Utils/SimplifyLibCalls.h
@@ -215,6 +215,7 @@ class LibCallSimplifier {
   Value *optimizeSymmetric(CallInst *CI, LibFunc Func, IRBuilderBase &B);
   Value *optimizeRemquo(CallInst *CI, IRBuilderBase &B);
   Value *optimizeFdim(CallInst *CI, IRBuilderBase &B);
+  Value *optimizeHypot(CallInst *CI, IRBuilderBase &B);
   // Wrapper for all floating point library call optimizations
   Value *optimizeFloatingPointLibCall(CallInst *CI, LibFunc Func,
                                       IRBuilderBase &B);
diff --git a/llvm/lib/Transforms/Utils/SimplifyLibCalls.cpp b/llvm/lib/Transforms/Utils/SimplifyLibCalls.cpp
index c3537f544c432..ee55296863a95 100644
--- a/llvm/lib/Transforms/Utils/SimplifyLibCalls.cpp
+++ b/llvm/lib/Transforms/Utils/SimplifyLibCalls.cpp
@@ -3215,6 +3215,40 @@ Value *LibCallSimplifier::optimizeFdim(CallInst *CI, IRBuilderBase &B) {
   return ConstantFP::get(CI->getType(), MaxVal);
 }
 
+/// Constant folds hypot
+Value *LibCallSimplifier::optimizeHypot(CallInst *CI, IRBuilderBase &B) {
+
+  const APFloat *X, *Y;
+  if (!match(CI->getArgOperand(0), m_APFloat(X)) ||
+      !match(CI->getArgOperand(1), m_APFloat(Y))) {
+    return nullptr;
+  }
+  Type *Ty = CI->getType();
+  APFloat Res(X->getSemantics());
+
+  // Only support float and double, as std::hypot doesn't support extended precision types.
+  if (Ty->isFloatTy())
+    Res = APFloat(std::hypot(X->convertToFloat(), Y->convertToFloat()));
+  else if(Ty->isDoubleTy())
+    Res = APFloat(std::hypot(X->convertToDouble(), Y->convertToDouble()));
+  else {
+    return nullptr;
+  }
+
+  // We can fold in the following cases:
+  // 1. Function doesn't access memory (intrinsic, no errno).
+  // 2. Fast-math is enabled (ignore errno).
+  if (CI->doesNotAccessMemory() || CI->isFast())
+    return ConstantFP::get(Ty, Res);
+
+  // 3. Result is finite OR inputs are non-finite (won't set errno).
+  //    hypot only sets errno on overflow with finite inputs.
+  if (Res.isFinite() || !X->isFinite() || !Y->isFinite())
+    return ConstantFP::get(Ty, Res);
+
+  return nullptr;
+}
+
 //===----------------------------------------------------------------------===//
 // Integer Library Call Optimizations
 //===----------------------------------------------------------------------===//
@@ -4140,6 +4174,10 @@ Value *LibCallSimplifier::optimizeFloatingPointLibCall(CallInst *CI,
   case LibFunc_fdimf:
   case LibFunc_fdiml:
     return optimizeFdim(CI, Builder);
+  case LibFunc_hypot:
+  case LibFunc_hypotf:
+  case LibFunc_hypotl:
+    return optimizeHypot(CI, Builder);
   case LibFunc_fminf:
   case LibFunc_fmin:
   case LibFunc_fminl:
diff --git a/llvm/test/Transforms/InstCombine/hypot.ll b/llvm/test/Transforms/InstCombine/hypot.ll
new file mode 100644
index 0000000000000..4c0c5c82d89c0
--- /dev/null
+++ b/llvm/test/Transforms/InstCombine/hypot.ll
@@ -0,0 +1,89 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes=instcombine -S | FileCheck %s
+
+declare double @hypot(double, double)
+declare float @hypotf(float, float)
+
+; Basic folding: Double precision
+define double @test_hypot_double_basic() {
+; CHECK-LABEL: @test_hypot_double_basic(
+; CHECK-NEXT:    ret double 5.000000e+00
+;
+  %call = call double @hypot(double 3.000000e+00, double 4.000000e+00)
+  ret double %call
+}
+
+; Basic folding: Float precision
+define float @test_hypot_float_basic() {
+; CHECK-LABEL: @test_hypot_float_basic(
+; CHECK-NEXT:    ret float 5.000000e+00
+;
+  %call = call float @hypotf(float 3.000000e+00, float 4.000000e+00)
+  ret float %call
+}
+
+; Special value: Infinity (doesn't set errno, should fold)
+define double @test_hypot_inf() {
+; CHECK-LABEL: @test_hypot_inf(
+; CHECK-NEXT:    ret double 0x7FF0000000000000
+;
+  %call = call double @hypot(double 0x7FF0000000000000, double 3.000000e+00)
+  ret double %call
+}
+
+; Special value: NaN (should propagate NaN)
+define double @test_hypot_nan() {
+; CHECK-LABEL: @test_hypot_nan(
+; CHECK-NEXT:    ret double 0x7FF8000000000000
+;
+  %call = call double @hypot(double 0x7FF8000000000000, double 3.000000e+00)
+  ret double %call
+}
+
+; Overflow: Strict mode (errno-sensitive, should not fold)
+define double @test_hypot_overflow_strict() {
+; CHECK-LABEL: @test_hypot_overflow_strict(
+; CHECK-NEXT:    [[CALL:%.*]] = call double @hypot(double 0x7FEFFFFFFFFFFFFF, double 0x7FEFFFFFFFFFFFFF)
+; CHECK-NEXT:    ret double [[CALL]]
+;
+  %call = call double @hypot(double 0x7FEFFFFFFFFFFFFF, double 0x7FEFFFFFFFFFFFFF)
+  ret double %call
+}
+
+; Overflow: Fast-math mode (ignore errno, force folding)
+define double @test_hypot_overflow_fast() {
+; CHECK-LABEL: @test_hypot_overflow_fast(
+; CHECK-NEXT:    ret double 0x7FF0000000000000
+;
+  %call = call fast double @hypot(double 0x7FEFFFFFFFFFFFFF, double 0x7FEFFFFFFFFFFFFF)
+  ret double %call
+}
+
+; Non-constant argument (should not fold)
+define double @test_hypot_non_constant(double %x) {
+; CHECK-LABEL: @test_hypot_non_constant(
+; CHECK-NEXT:    [[CALL:%.*]] = call double @hypot(double [[X:%.*]], double 4.000000e+00)
+; CHECK-NEXT:    ret double [[CALL]]
+;
+  %call = call double @hypot(double %x, double 4.000000e+00)
+  ret double %call
+}
+
+; Strict FP mode (should not fold even for valid constants)
+define double @test_hypot_strictfp() strictfp {
+; CHECK-LABEL: @test_hypot_strictfp(
+; CHECK-NEXT:    [[CALL:%.*]] = call double @hypot(double 3.000000e+00, double 4.000000e+00) #[[ATTR0:[0-9]+]]
+; CHECK-NEXT:    ret double [[CALL]]
+;
+  %call = call double @hypot(double 3.000000e+00, double 4.000000e+00) strictfp
+  ret double %call
+}
+
+; Zero values (should fold: hypot(0,0) = 0)
+define double @test_hypot_zero() {
+; CHECK-LABEL: @test_hypot_zero(
+; CHECK-NEXT:    ret double 0.000000e+00
+;
+  %call = call double @hypot(double 0.000000e+00, double 0.000000e+00)
+  ret double %call
+}
\ No newline at end of file

[nikic]

This needs to be handled in ConstantFolding rather than SimplifyLibCalls.

[wentywenty]

This needs to be handled in ConstantFolding rather than SimplifyLibCalls.

thanks ，i will try it

[jcranmer-intel]

hypot is dangerous to constant-fold with NaNs, because it's part of the family of functions where qNaN and sNaN have different outputs. (hypot(qNaN, inf) is inf, hypot(sNaN, inf) is qNaN).