[RFC] Disable Floating-point distributive property

[liwchang]

TVM seems applying floating-point distributive property (HalideIR\src\arithmetic\Simplify.cc, Around Line 845-86 ).
In general, floating point should allow only commutative property, not associative or distributive.

Applying distributive property in floating point could significantly change rounding error, and impact accuracy.
And we did observe that in some models.

One way to fix it is to check the type and bypass some rules for floating point.
Another is a flag to enable/disable it if someone really want to trade tiny speed with accuracy.
(Default should disable floating-point associative and distributive) i

Halide (at least the current master) seems doing both.

-L

[tqchen]

I think it is fine to disable floating point simplification as we mainly only need integer analysis

[tqchen]

Consolidate this issue to #2588

[Ravenwater]

For what it is worth, the posit number system restores associative and distributive properties for floating point. The culprit is IEEE floating point rounding rules. The solution requires special hardware which is available as of Dec 2017, and we are trying to incorporate this hw into VTA. The benefit is significant: an 8-bit posit beats a 32-bit IEEE float in terms of training accuracy. And since most models are memory bound, we get a big boost in performance as well. Of course, an 8-bit posit is still slower than an 8-bit integer.