generated from ni/github-repo-template
-
Notifications
You must be signed in to change notification settings - Fork 3
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Bug Report
nitypes.bintime.TimeDelta.__repr__ currently displays a constructor expression using decimal.Decimal, but it doesn't limit the decimal to an appropriate number of digits.
Repro or Code Sample
>>> import nitypes.bintime as bt
>>> from decimal import Decimal
>>> x = bt.TimeDelta(Decimal("1.23456789"))
>>> x
nitypes.bintime.TimeDelta(Decimal('1.234567889999999999977708248222185716258536558598279953002929688'))
>>> x._ticks
22773757908449605611
>>> y = bt.TimeDelta(Decimal('1.234567889999999999977708248222185716258536558598279953002929688'))
>>> y._ticks
22773757908449605611
>>> z = bt.TimeDelta(Decimal('1.2345678899999999999777'))
>>> z._ticks
22773757908449605611Expected Behavior
The smallest time increment that NI-BTF can represent is 54210 yoctoseconds, so there should be no more than 24 decimal places (yoctoseconds), and preferably less.
Current Behavior
64 decimal places are displayed, because that is the precision that bintime uses to do decimal conversions.
Possible Solution
Use format strings to specify the decimal precision:
>>> "{0:.024}".format(Decimal('1.234567889999999999977708248222185716258536558598279953002929688'))
'1.23456788999999999997771'
Context
It makes the output of repr harder to read.
Your Environment
nitypesversion: 0.1.0-dev1- Python version: 3.11.9
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working