InstCombine: incorrect select fast-math folds

[nunoplopes]

Bugzilla Link	50281
Version	trunk
OS	All
CC	@LebedevRI,@RKSimon,@rotateright

Extended Description

Alive2 reports a few incorrect optimizations around select w/ fast-math in Transforms/InstCombine/minmax-fp.ll:

---

nsz allows the output to be +0/-0 non-deterministically. In this example, the source always returns +0 and the target may return +0 or -0. The select in src would need nsz as well:

define <2 x float> @fsub_fmax(<2 x float> %x, <2 x float> %y) {
  %n1 = fsub <2 x float> <float -0.000000, float -0.000000 >, %x
  %n2 = fsub <2 x float> <float -0.000000, float -0.000000 >, %y
  %cond = fcmp nnan nsz uge <2 x float> %n1, %n2
  %max = select <2 x i1> %cond, <2 x float> %n1, <2 x float> %n2
  ret <2 x float> %max
}
=>
define <2 x float> @fsub_fmax(<2 x float> %x, <2 x float> %y) {
  %cond.inv = fcmp nnan nsz ogt <2 x float> %x, %y
  %1 = select nnan nsz <2 x i1> %cond.inv, <2 x float> %y, <2 x float> %x
  %max = fneg <2 x float> %1
  ret <2 x float> %max
}

Transformation doesn't verify!

ERROR: Value mismatch

Example:
<2 x float> %x = < poison, #x80000000 (-0.0) >
<2 x float> %y = < poison, #x00800001 (0.000000000000?) >

Source:
<2 x float> %n1 = < poison, #x00000000 (+0.0) >
<2 x float> %n2 = < poison, #x80800001 (-0.000000000000?) >
<2 x i1> %cond = < poison, #x1 (1) >
<2 x float> %max = < poison, #x00000000 (+0.0) >

Target:
<2 x i1> %cond.inv = < poison, #x0 (0) >
<2 x float> %1 = < poison, #x00000000 (+0.0) >
<2 x float> %max = < poison, #x80000000 (-0.0) >
Source value: < poison, #x00000000 (+0.0) >
Target value: < poison, #x80000000 (-0.0) >

---

select nnan only applies to the chosen operand (?). If that's the case then the optimization below isn't correct for a NaN input. If nnan applies to both select operands, then fptrunc_select_true_val_extra_use in fptrunc.ll is incorrect.

define float @​maxnum_ogt_fmf_on_select(float %a, float %b) {
%cond = fcmp ogt float %a, %b
%f = select nnan nsz i1 %cond, float %a, float %b
ret float %f
}
=>
define float @​maxnum_ogt_fmf_on_select(float %a, float %b) {
%1 = fmax nnan nsz float %a, %b
ret float %1
}
Transformation doesn't verify!

ERROR: Target is more poisonous than source

Example:
float %a = NaN
float %b = #x20008b45 (0.000000000000?)

Source:
i1 %cond = #x0 (0)
float %f = #x20008b45 (0.000000000000?)

Target:
float %1 = poison
Source value: #x20008b45 (0.000000000000?)
Target value: poison

[fhahn]

Still reproduces on current main: https://llvm.godbolt.org/z/YrG81Ybbs