Add bfloat16 to all relevant ops

[fdwr]

Bug Report

Is the issue related to model conversion?

No.

Looks like an oversight, as it's inconsistent with both Less and Equal which support bfloat16 but their combination does not, especially given LessOrEqual can be implemented as an aggregate operator ("function") of those two. The same applies to GreaterOrEqual.

Describe the bug

LessOrEqual and GreaterOrEqual should support bfloat16. Pertinent PR #2770.

System information

• ONNX version: master 1.10

[fdwr]

FYI @gramalingam, @souptc.

[fdwr]

Just noticed also that Sigmoid has bfloat16 but not HardSigmoid, even though all the constituent operators (max, min, mul, add) do support bfloat16.

[gramalingam]

Makes sense. Looks like an oversight. We should add the missing bfloat16 type to all such ops. Would be nice to find all such ops in one go! (Eg., we had to do this for Pow and Where previously I think.)

[gramalingam]

Here is a list of other ops that may be missing bfloat16: acos, acosh, asin, asinh, atan, atanh, averagepool, bernoulli, celu, concatfromsequence, constantofshape, conv, convtranspose, cos, cosh, dequantizelinear, det, dynamicquantizelinear, einsum, eyelike, globalaveragepool, globalpool, globalmaxpool, gru, hardsigmoid, hardswish, if, instancenormalization, isinf, leakyrelu, loop, lstm, maxpool, maxroipool, maxunpool, multinomial, negativeloglikelihoodloss, nonmaxsuppression, onehot, optional, optionalgetelement, optionalhaselement, prelu, resize, reversesequence, rnn, roialign, round, scan, scatterlements, selu, sequenceat, sequenceconstruct, sequenceempty, sequenceerase, sequenceinsert, sequencelength, shrink, sin, sinh, softplus, softsign, splittosequence, thresholdedrelu, topk, unique.

[gramalingam]

Adding a note to clarify that bfloat16 was added for a few selected ops, not all of the ops identified above as missing bfloat16. So, this issue is only partially closed.

[askhade]

I did not close this issue... I think it was closed becuase the PR got merged. Reopening the issue to continue to track the rest of the ops. Per our discussion will remove the 1.11 milestone tag

[jcwchen]

#4051 Another fresh bfloat request for Conv op.

[jcwchen]

I guess it has not been fully completed yet. Let me reopen it for tracking.

[justinchuby]

We can add fp8 now too

[thiagocrepaldi]

#6071 is a duplicate - and more specific case for Conv-11 which has float16 but not bfloat16 support