Skip to content
Permalink
Browse files

Add symmetric with power2 scale quantization schema (#3437)

Summary:
**Summary**
Add new quantization schema (named "SymmetricWithPower2Scale").
The new quantization schema is a particular quantization schema which:
- is symmetric (offset = 0)
- the floating point scale if forced to be a power of 2 (**scale = 2 ^ exp** for example 0.5, 0.125, etc)

For the previous quantization schemes the main approximation was:
- (x * float_scale + offset) -> ((x >> pre_scale  * integer_scale) >> post_scale) + offset

For the new quantization schema the approximation has a particular, more simpler form:
- (x * float_scale + offset) -> x * integer_scale for positive exponent
- (x * float_scale + offset) -> x >> post_scale for negative exponent

The new quantization schema:
- is expected to be less accurate (in terms of arithmetic approximation of floating-point operations)
- is expected to be simpler (arithmetic complexity, number of operations)
- can be viewed as another point in the spectrum of quantization schemes with different tradeoffs between accuracy and complexity (this is oriented more towards the "simple"/"less accurate" end)
- has proved to be good enough in terms of accuracy for some networks (e.g. lenet_mnist, cifar10)
where the network was sufficiently robust/redundant (1..2 % accuracy drop relative to floating-point)
- the main point is that it allows the specialization of some the LIBJIT kernels with other open-source hand-optimized kernels/libraries for some specific architectures

This quantization schema requires the floating-point scales to preserve their bit-exact values during YAML round-trip. Therefore, I added a workaround for retrieving the floating-point scales bit-exactly during YAML round-trip.

**Documentation**
None.

**Test Plan**
Tests for new quantization schema.
Tests for verifying that floating-point numbers are preserved bit-exactly during YAML round-trip.

Please see a detailed explanation of how to fill out the fields in the relevant sections in PULL_REQUEST.md.
Pull Request resolved: #3437

Differential Revision: D17115657

Pulled By: rdzhabarov

fbshipit-source-id: 4668e07232d10b159b7e91faa06cdaa286e8e3a4
  • Loading branch information...
mciprian13 authored and facebook-github-bot committed Sep 4, 2019
1 parent 11657b5 commit 4554c5d7cbd79f078a9046eb98cb3ddb79f42223
@@ -103,24 +103,35 @@ can be listed via comma separation). For example:
```./bin/image-classifier tests/images/imagenet/*.png -image-mode=0to1 -m=shufflenet -model-input-name=gpu_0/data -dump-profile="shufflenet.yaml" -do-not-lower-nodes-for-profiling=Convolution```
By default, the loader will produce quantized results using asymmetric ranges.
That is ranges not necessarily centered on 0. The loader supports three modes
or schemas of quantization: asymmetric, symmetric, and symmetric with uint8. The symmetric schema
will always map the data on ranges centered on 0. In practice, this means
the symmetric schema may extend the range it needs to capture to make
sure 0.0 is at the center of that range. Therefore, this schema potentially
waste some encoding space to enforce the symmetric property, but it comes
with the property that the offset is always equal to zero.
The symmetric with uint8 schema conceptually produces ranges where the offset
is always equal to zero but allows the quantized ranges to be either
int8 [-128; 127] or uint8 [0; 255]. In practice, this schema represents
uint8 ranges using int8 ranges with an offset of -128. Therefore, when
using this schema, the produced profile will have two kinds of ranges:
one with an offset of 0 and the other with an offset of -128.
Use ```quantization-schema=<schema>``` to specify the schema for
the quantization process, where schema is ```asymmetric```,
```symmetric```, or ```symmetric_with_uint8```.
The loader supports the following modes (or schemas) of quantization:
- ```asymmetric``` - maps the floating data to quantized ranges not necessarily
centered on 0. This is the default quantization schema.
- ```symmetric``` - maps the floating data to ranges centered on 0. In practice,
this means the symmetric schema may extend the range it needs to capture to make
sure 0.0 is at the center of that range. Therefore, this schema potentially wastes
some encoding space to enforce the symmetric property, but it comes with the
property that the offset is always equal to zero.
- ```symmetric with uint8``` - produces ranges where the offset is always equal to
zero but allows the quantized ranges to be either int8 [-128; 127] or uint8 [0; 255].
In practice, this schema represents uint8 ranges using int8 ranges with an offset of
-128. Therefore, when using this schema, the produced profile will have two kinds of
ranges: one with an offset of 0 and the other with an offset of -128.
- ```symmetric with power of 2 scale``` - produces quantized ranges centered on 0
(symmetric) but also restricts the scale parameter to be a power of 2. Restricting
the scale parameter to be a power of 2 might result in a poor exploitation of the
quantized range (poor accuracy) but has the potential to provide a better performance.
Use ```quantization-schema=<schema>``` to specify the schema for the quantization
process, where schema will have one of the values:
- ```asymmetric```
- ```symmetric```
- ```symmetric_with_uint8```
- ```symmetric_with_power2_scale```

```load-profile=profile.yaml``` option is used to quantize graph based on the
captured profile in ```profile.yaml``` file. Important note, graph structure
@@ -51,9 +51,10 @@ struct QuantizationTransform32To8 {

/// \returns the scaled integer.
int32_t transform(int32_t input) {
// The operation x >> y is rounded down to negative infinity. To get to
// round-nearest we add (1 << (shift - 1)) to the value prior to shifting.
int rtn = (1 << (post - 1));
// The operation x >> post is rounded down to negative infinity. To get to
// round-nearest we add (1 << (post - 1)) to the value prior to shifting.
// Rounding is performed only when shifting right (pos > 0).
int rtn = (post > 0) ? (1 << (post - 1)) : 0;
return ((((input >> pre) * scale) + rtn) >> post) + offset;
}
};
@@ -118,6 +119,14 @@ enum Schema {
/// version of the quantized type with an offset of zero:
/// For example, int8 is [-128; 127] - (-128) == uint8 [0; 255] - 0
SymmetricWithUnsigned,
/// Quantization schema with:
/// - range centered on 0 (symmetric): offset == 0.
/// - scale parameter is a power of 2: scale = 2^E where E is a signed
/// exponent. Since the scale parameter is mostly subunitary, the
/// exponent is mostly negative.
/// Since the scale parameter is stored as floating point, the values
/// of E which are exactly representable range from -126 to 127.
SymmetricWithPower2Scale,
};

/// Configuration for Quantization, passed into \ref quantizeFunction().
@@ -163,7 +172,9 @@ template <class SrcTy, class DestTy> DestTy clip(SrcTy in) {
template <class DestTy = int8_t>
inline DestTy quantize(float input, const TensorQuantizationParams &TQP) {
float result = input / TQP.scale + TQP.offset;
return quantization::clip<int32_t, DestTy>((int32_t)nearbyintf(result));
// Note: use int64_t since casts of large values might be wrapped around
// before clipping, for example for result = 2147483648.00 (float).
return quantization::clip<int64_t, DestTy>((int64_t)nearbyintf(result));
}

/// Converts a quantized value (type eTy) to floating point based on the
@@ -422,6 +433,13 @@ void tensorFusedRowwiseQuantization(const Tensor &input, Tensor &output) {
destH.setFusedScaleOffsetInRow<T>(i, scale, offset);
}
}

/// Verify if float is an exact power of 2 (mantissa is exactly 1.0).
bool isFloatPowerOf2(float val);

/// Get float 2's exponent.
int getFloat2Exp(float val);

} // namespace quantization
} // namespace glow

@@ -111,8 +111,9 @@ inline int8_t libjit_clip(int32_t val) {
/// See QuantizationTransform32To8 for more details.
inline int32_t libjit_scale_i32i8(int32_t input, int32_t pre, int32_t post,
int32_t scale, int32_t offset) {
// The operation x >> y is rounded down to negative infinity. To get to
// round-nearest we add (1 << (shift - 1)) to the value prior to shifting.
// The operation x >> post is rounded down to negative infinity. To get to
// round-nearest we add (1 << (post - 1)) to the value prior to shifting.
// Rounding is performed only when shifting right (pos > 0).
int rtn = (post > 0) ? (1 << (post - 1)) : 0;

// NOTICE: If your tests are failing because of signed integer overflow then
@@ -225,6 +225,27 @@ QuantizationTransform32To8 quantizeScaleOffset32To8(float scale,
int preShift = 0;
int postShift = 0;

// We treat first the particular case when scale is a power of 2 (2 ^ exp,
// where exp is a signed integer exponent). The operation is specialized as:
// - for positive 2's exponent:
// x * scale + offset (pre = 0, post = 0, scale = (int)scale).
// - for negative 2's exponent:
// x >> post + offset (pre = 0, post = -exp, scale = 1).
if (isFloatPowerOf2(scale)) {
int exp = getFloat2Exp(scale);
if (exp > 0) {
return QuantizationTransform32To8(0, // pre
0, // post
static_cast<int>(scale), // scale
offset); // offset
} else {
return QuantizationTransform32To8(0, // pre
-exp, // post
1, // scale
offset); // offset
}
}

// Calculate the post-shift value. It's always safe to increase scale as long
// as it's below one, and it's always legal to shift at least 15 bits for
// small scale values.
@@ -301,9 +322,11 @@ TensorQuantizationParams chooseQuantizationParams(float min, float max,
schema = quantization::Schema::Symmetric;
}
}
if (schema == quantization::Schema::Symmetric) {
if (schema == quantization::Schema::Symmetric ||
schema == quantization::Schema::SymmetricWithPower2Scale) {
// Check which end saturates the output dynamic range earlier
// and extend the other end to map the zero-point to quantized 0.
assert(qmin < 0 && "Symmetric schema incompatible with unsigned range");
double rmin = min / (double)qmin;
double rmax = max / (double)qmax;
if (rmin > rmax) {
@@ -362,6 +385,11 @@ TensorQuantizationParams chooseQuantizationParams(float min, float max,
nudgedZeroPoint = static_cast<int32_t>(round(initialZeroPoint));
}

// For SymmetricWithPower2Scale, round scale to nearest higher power of 2.
if (schema == quantization::Schema::SymmetricWithPower2Scale) {
scale = std::exp2(std::ceil(std::log2(scale)));
}

TensorQuantizationParams result{static_cast<float>(scale), nudgedZeroPoint};
// The only valid offset for symmetric quantization is 0.
assert((result.offset == 0 || schema != quantization::Schema::Symmetric) &&
@@ -373,6 +401,17 @@ TensorQuantizationParams chooseQuantizationParams(float min, float max,
schema != quantization::Schema::SymmetricWithUnsigned) &&
"Symmetric quantization with unsigned should be centered on 0 or on "
"-qmin");

// For SymmetricWithPower2Scale schema the offset should be 0.
assert((result.offset == 0 ||
schema != quantization::Schema::SymmetricWithPower2Scale) &&
"Symmetric quantization should be centered on 0");

// For SymmetricWithPower2Scale schema the scale should be a power of 2.
assert((isFloatPowerOf2(result.scale) ||
schema != quantization::Schema::SymmetricWithPower2Scale) &&
"Scale quantization parameter should be a power of 2");

return result;
}

@@ -401,5 +440,13 @@ std::vector<int8_t> createMapping(TypeRef inTy, TypeRef outTy,
return mapping;
}

bool isFloatPowerOf2(float val) {
// frexp returns mantissa normalized in [0.5,1) so compare with 0.5.
int exp;
return (std::abs(std::frexp(val, &exp)) == 0.5);
}

int getFloat2Exp(float val) { return std::ilogb(val); }

} // namespace quantization
} // namespace glow
@@ -28,11 +28,52 @@
namespace llvm {
namespace yaml {

/// The default behavior of YAML is to serialize floating point numbers
/// using the "%g" format specifier which is not guaranteed to print all
/// the decimals. During a round-trip (serialize, deserialize) decimals
/// might be lost and hence precision is lost. Although this might not be
/// critical for some quantization schema, for "SymmetricWithPower2Scale"
/// the round-trip must preserve the exact representation of the floating
/// point scale which is a power of 2. The code below is a workaround to
/// overwrite the behavior of the YAML serializer to print all the digits.
struct FloatWrapper {
float val_;
FloatWrapper(float val) : val_(val) {}
};

template <> struct ScalarTraits<FloatWrapper> {
static void output(const FloatWrapper &value, void *ctxt,
llvm::raw_ostream &out) {
// Print number with all the digits and without trailing 0's
char buffer[200];
snprintf(buffer, sizeof(buffer), "%.126f", value.val_);
int n = strlen(buffer) - 1;
while ((n > 0) && (buffer[n] == '0') && (buffer[n - 1] != '.')) {
buffer[n--] = '\0';
}
out << buffer;
}
static StringRef input(StringRef scalar, void *ctxt, FloatWrapper &value) {
if (to_float(scalar, value.val_))
return StringRef();
return "invalid floating point number";
}
static QuotingType mustQuote(StringRef) { return QuotingType::None; }
};

/// Mapping for NodeQuantizationInfo yaml serializer.
template <> struct MappingTraits<glow::NodeQuantizationInfo> {
struct FloatNormalized {
FloatNormalized(IO &io) : val_(0.0) {}
FloatNormalized(IO &, float &val) : val_(val) {}
float denormalize(IO &) { return val_.val_; }
FloatWrapper val_;
};
static void mapping(IO &io, glow::NodeQuantizationInfo &info) {
MappingNormalization<FloatNormalized, float> scale(
io, info.tensorQuantizationParams_.scale);
io.mapRequired("nodeOutputName", info.nodeOutputName_);
io.mapRequired("scale", info.tensorQuantizationParams_.scale);
io.mapRequired("scale", scale->val_);
io.mapRequired("offset", info.tensorQuantizationParams_.offset);
}
};
@@ -107,9 +107,21 @@ void testSerialization(const std::vector<NodeQuantizationInfo> &expected) {
}

TEST(Quantization, Serialize) {
std::vector<NodeQuantizationInfo> expected{
{"first", {1, 10}}, {"second", {-1, 3}}, {"third", {-10, 30}}};
std::vector<NodeQuantizationInfo> expected{{"first", {1, 10}},
{"second", {-1, 3}},
{"third", {-10, 30}},
{"fourth", {0.1, -10}},
{"fifth", {0.123, -30}}};
testSerialization(expected);
}

TEST(Quantization, SerializePower2Scale) {
std::vector<NodeQuantizationInfo> expected{
{"pwr_neg_0", {1.0000000000f, 0}}, {"pwr_neg_1", {0.5000000000f, 0}},
{"pwr_neg_2", {0.2500000000f, 0}}, {"pwr_neg_3", {0.1250000000f, 0}},
{"pwr_neg_4", {0.0625000000f, 0}}, {"pwr_neg_5", {0.0312500000f, 0}},
{"pwr_neg_6", {0.0156250000f, 0}}, {"pwr_neg_7", {0.0078125000f, 0}},
{"pwr_neg_8", {0.0039062500f, 0}}, {"pwr_neg_9", {0.0019531250f, 0}}};
testSerialization(expected);
}

@@ -155,6 +167,34 @@ TEST(Quantization, quantScaleOffset) {
}
}

TEST(Quantization, quantScaleOffsetPower2Scale) {
// Test different power of 2 scale values (from 2^-10 to 2^1).
float scales[] = {0.0009765625f, 0.0019531250f, 0.0039062500f, 0.0078125000f,
0.0156250000f, 0.0312500000f, 0.0625000000f, 0.1250000000f,
0.2500000000f, 0.5000000000f, 1.0000000000f, 2.0000000000f};

// Try all scale factors:
for (float scale : scales) {
// Try all legal integers within the range:
for (int8_t input = -128; input < 127; input++) {
int32_t sum32num = round(input / scale);
auto TR = quantization::quantizeScaleOffset32To8(scale, 0);
EXPECT_EQ(quantization::isFloatPowerOf2(scale), true);
EXPECT_EQ(TR.pre, 0);
int exp = quantization::getFloat2Exp(scale);
if (exp > 0) {
EXPECT_EQ(TR.scale, (int)scale);
EXPECT_EQ(TR.post, 0);
} else {
EXPECT_EQ(TR.scale, 1);
EXPECT_EQ(TR.post, -exp);
}
int32_t computed = TR.transform(sum32num);
EXPECT_NEAR(input, computed, 1);
}
}
}

template <class qtype>
void quantizeTensorTest(ElemKind qTy, quantization::Schema schema) {
// Map float [0.0; 6.0] to a quantized type using its entire value range.
@@ -238,6 +278,18 @@ TEST(Quantization, quantizeTensorSymmetricUInt32) {
quantizeTensorTest<int32_t>(ElemKind::Int32QTy,
quantization::Schema::SymmetricWithUnsigned);
}
TEST(Quantization, quantizeTensorSymmetricPwr2Int8) {
quantizeTensorTest<int8_t>(ElemKind::Int8QTy,
quantization::Schema::SymmetricWithPower2Scale);
}
TEST(Quantization, quantizeTensorSymmetricPwr2Int16) {
quantizeTensorTest<int16_t>(ElemKind::Int16QTy,
quantization::Schema::SymmetricWithPower2Scale);
}
TEST(Quantization, quantizeTensorSymmetricPwr2Int32) {
quantizeTensorTest<int32_t>(ElemKind::Int32QTy,
quantization::Schema::SymmetricWithPower2Scale);
}

/// Test 4-bit fused rowwise quantization.
TEST(Quantization, fused4BitsRowwiseQuantizeTensor) {
@@ -1344,6 +1396,20 @@ TEST(Quantization, chooseQuantizationSymmetricWithUInt8) {
EXPECT_NEAR(symmetricParams.scale, 16.0 / 255, 0.001);
}

/// Verify the SymmetricWithPower2Scale quantization schema.
static void chooseQuantParamsPower2Scale(float min, float max, ElemKind qTy) {
auto quantParams = quantization::chooseQuantizationParams(
min, max, quantization::Schema::SymmetricWithPower2Scale, qTy);
EXPECT_EQ(quantParams.offset, 0);
EXPECT_TRUE(quantization::isFloatPowerOf2(quantParams.scale));
}

TEST(Quantization, chooseQuantizationSymmetricWithPower2Scale) {
chooseQuantParamsPower2Scale(-3.0, 6.0, ElemKind::Int8QTy);
chooseQuantParamsPower2Scale(3.0, 6.0, ElemKind::Int16QTy);
chooseQuantParamsPower2Scale(-6.0, 0.0, ElemKind::Int32QTy);
}

/// Check that LRN and Softmax are quantized.
TEST(Quantization, quantizeSoftmaxAndLRN) {
ExecutionEngine EE{};
@@ -85,7 +85,10 @@ llvm::cl::opt<quantization::Schema> quantizationSchema(
"Use symmetric ranges"),
clEnumValN(quantization::Schema::SymmetricWithUnsigned,
"symmetric_with_uint8",
"Use symmetric ranges with potentially uint8 ranges")),
"Use symmetric ranges with potentially uint8 ranges"),
clEnumValN(quantization::Schema::SymmetricWithPower2Scale,
"symmetric_with_power2_scale",
"Use symmetric ranges with power of 2 scaling factor")),
llvm::cl::init(quantization::Schema::Asymmetric), llvm::cl::cat(loaderCat));

llvm::cl::opt<ElemKind> quantizationPrecision(

0 comments on commit 4554c5d

Please sign in to comment.
You can’t perform that action at this time.