-
Notifications
You must be signed in to change notification settings - Fork 26
Closed
Description
When a point has a nonzero scale factor, its cvalue (computed, scaled value) can contain floating point error. This causes problems when an application wants to verify a write by comparing the scaled value it has written to the device with the scaled value it reads back from the device.
For example, take a data point having an sf_value of -1. When the value is 302, the cvalue will read back (on my machine) as 30.200000000000003. The cvalue for this point really should be rounded to 30.2
Suggested change from (device.py#L220):
if self.sf_value:
sfv = self.sf_value
if sfv:
v = v * math.pow(10, sfv)To this:
if self.sf_value:
sfv = self.sf_value
if sfv:
v = round(v * math.pow(10, sfv), -1 * sfv)Metadata
Metadata
Assignees
Labels
No labels