Skip to content

Commit

Permalink
Adjust rtol value for confirming quantization.
Browse files Browse the repository at this point in the history
Fixed for better readability.
This constraint should be enough to test whether a f32 value dequantized from i8 was changed.

PiperOrigin-RevId: 631559455
  • Loading branch information
chococigar authored and tensorflower-gardener committed May 7, 2024
1 parent f216523 commit 62c4300
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -899,7 +899,7 @@ def data_gen() -> repr_dataset.RepresentativeDataset:
# be exactly the same. Indirectly proves that the `FunctionNameMatcherSpec`
# with regex '.*invalid_function_name.*' did not match the quantizable unit.
self.assertAllClose(new_outputs, expected_outputs, rtol=0.04)
self.assertNotAllClose(new_outputs, expected_outputs, rtol=0.00001)
self.assertNotAllClose(new_outputs, expected_outputs, 1e-7)

# Due to other meta data, the compression is not exactly 1/4.
self.assertLess(
Expand Down

0 comments on commit 62c4300

Please sign in to comment.