Generate fixed point decimals

[jshprentz]

Add an optional places parameter to the decimals() strategy to specify the number of decimal places needed in generated values.

Many applications use Decimal numbers to perform precise calculations to a specific number of decimal places. Dollar amounts often use two decimal places, for example. Mutual fund shares may be tracked to five or six decimal places. Manufacturing tolerances are often specified to thousandths of an inch—three decimal places.

The Hypothesis decimals() strategy currently generates decimal numbers from floats or fractions. These value typically have uncontrollably many decimal places.

>>> from hypothesis.strategies import decimals
>>> decimals().example()
Decimal('0.002472471796257515368506383841')

My proof of concept, attached as test_decimals.py.txt, adds a an optional places parameter to the decimals() strategy. When places is specified, decimals() generates values with the given number of digits after the decimal point. When places is not specified, the current unconstrained value generation is used.

>>> import test_decimals
>>> decimals = test_decimals.decimals
>>> decimals(places=3).example()
Decimal('5269885835422919568130741.438')
>>> decimals(places=2).example()
Decimal('27258100601131699143211881.42')
>>> decimals().example()
Decimal('0.2908554201733450571536239166')

The sample code generates fixed point decimals from the integers() strategy rather than the floats and fractions used currently. The Hypothesis generated integers map directly to decimal values. With three decimal places, for example, 1 maps to .001, 2 maps to .002, and so on. For fixed decimal place values, I think that the underlying integer shrink strategy is more appropriate than the float or fraction strategy used for unconstrained decimal place values.

Python has a global decimal context containing various configuration parameters for the decimal module. Applications may change the decimal context globally or locally. My proof of concept code uses the context's precision value, which specifies the maximum number of digits allowed in a value. Hypothesis caching does not play well with global values changed between tests. My sample code does not cache values from decimals(), but does cache values from fixed_point_decimals(), which has precision as an explicit parameter. For example,

>>> from decimal import localcontext
>>> with localcontext() as context:
...     context.prec = 10
...     value = decimals(places=3).example()
...     print(value)
...
2609227.817
>>> with localcontext() as context:
...     context.prec = 20
...     value = decimals(places=3).example()
...     print(value)
...
17961664460289794.568

[mulkieran]

Note that PR #409 is sorta related.

[jshprentz]

The allow_nan and allow_infinity flags requested PR #409 and related PR #186 could be added to my proof of concept code.

As discussed in those other PRs, the allow_infinity flag default value True is somewhat at conflict with the max_value parameter. My sample code always has some max_value, either explicitly passed as an argument or derived from the Decimal context precision.

Applications (and people) using fixed point decimals typically don't worry about NaN or infinite values. Instead, these applications effectively perform integer math on some small units and report with a decimal inserted at an appropriate position. Users often have names for the smallest unit: 0.01 dollars is a cent, .00000001 bitcoins is a Satoshi, and .001 inches is a thousandth.

I hope that the default values for the allow_nan and allow_infinity flags do not require every fixed decimal point request to include allow_nan=False, allow_infinity=False. Those should be the default values when the places argument is provided.

[mulkieran]

I am somewhat dubious because places actually affects the value returned, which seems wrong and AFAICT unnecessary.

I agree with your examples as being the usual and probably best use for the Decimal type.

I dunno about your final point, I haven't looked at that related code in a while. But I don't think it would be a big deal, because the strategy could be wrapped to send the desired values of those parameters anyway. The real problems occur if there are bad ideas implemented within the strategy that actually have to be worked around.

[jshprentz]

@mulkieran wrote,

I am somewhat dubious because places actually affects the value returned, which seems wrong and AFAICT unnecessary.

Shouldn't parameters such as min_value, allow_nan, and the proposed places affect the values returned by the decimals strategy? These parameters control which of all possible Decimal values the decimals strategy will provide. Values below some threshold are prohibited by min_value. Values with more than some number of decimal places are prohibited by places.

I agree with avoiding bad ideas in strategies and the need to work around them. As a relatively new Hypothesis user, I cannot always distinguish good and bad ideas in strategies. I am much more familiar with ordinary unit testing; I demonstrated PythonUnit at the January 2000 Python Conference.

The wrapper approach seems preferable to special cases for the allow_nan and allow_infinity flags. What would such a wrapper look like for, say, dollars with two decimal places and with no NaN or inifinity values?

[mulkieran]

My worry is specifically about the effect of the places parameter. I am not an expert, but I think that the integers() strategy is somewhat tuned to return values like 0, 1, 3. You get an integer from the strategy and transform it by means of places, favoring 1/10^place, 3 / 10 ^ places, etc. I'm not sure that is really right, and the strategy should maybe be preferred to favor things like 1.0, 3.0, etc. My concern is about the use of places to generate the values also affecting their distribution, possibly unhelpfully.

My remark about wrappers means that it is best to choose the simplest and most mathematically valid internal implementation of the strategy. Then, if the best default for allow_nan by that rubric is True it is still easy for someone to wrap the strategy and overrride the default, passing False if they want to.

I'm idly wondering if you might be better off distributing a competing Decimals strategy independently. I established a registry for external strategies a month or so ago, and right now there's only one on the list:
https://github.com/HypothesisWorks/hypothesis-python/blob/master/docs/strategies.rst.

But, I'm not the boss of Hypothesis, that's @DRMacIver :) So, I'm going to bow out for a bit and see what he has to say.