Please sign in to comment.
Add symmetric with power2 scale quantization schema (#3437)
Summary: **Summary** Add new quantization schema (named "SymmetricWithPower2Scale"). The new quantization schema is a particular quantization schema which: - is symmetric (offset = 0) - the floating point scale if forced to be a power of 2 (**scale = 2 ^ exp** for example 0.5, 0.125, etc) For the previous quantization schemes the main approximation was: - (x * float_scale + offset) -> ((x >> pre_scale * integer_scale) >> post_scale) + offset For the new quantization schema the approximation has a particular, more simpler form: - (x * float_scale + offset) -> x * integer_scale for positive exponent - (x * float_scale + offset) -> x >> post_scale for negative exponent The new quantization schema: - is expected to be less accurate (in terms of arithmetic approximation of floating-point operations) - is expected to be simpler (arithmetic complexity, number of operations) - can be viewed as another point in the spectrum of quantization schemes with different tradeoffs between accuracy and complexity (this is oriented more towards the "simple"/"less accurate" end) - has proved to be good enough in terms of accuracy for some networks (e.g. lenet_mnist, cifar10) where the network was sufficiently robust/redundant (1..2 % accuracy drop relative to floating-point) - the main point is that it allows the specialization of some the LIBJIT kernels with other open-source hand-optimized kernels/libraries for some specific architectures This quantization schema requires the floating-point scales to preserve their bit-exact values during YAML round-trip. Therefore, I added a workaround for retrieving the floating-point scales bit-exactly during YAML round-trip. **Documentation** None. **Test Plan** Tests for new quantization schema. Tests for verifying that floating-point numbers are preserved bit-exactly during YAML round-trip. Please see a detailed explanation of how to fill out the fields in the relevant sections in PULL_REQUEST.md. Pull Request resolved: #3437 Differential Revision: D17115657 Pulled By: rdzhabarov fbshipit-source-id: 4668e07232d10b159b7e91faa06cdaa286e8e3a4
- Loading branch information...
Showing with 216 additions and 29 deletions.