Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

maintain big.Float precision with multiplication and modulo #75

Merged
merged 2 commits into from Nov 12, 2020
Merged

maintain big.Float precision with multiplication and modulo #75

merged 2 commits into from Nov 12, 2020

Conversation

jbardin
Copy link
Contributor

@jbardin jbardin commented Oct 29, 2020

There are probably other places where arbitrarily large values can
overflow the default precision creating unexpected results, but
these two points are the easiest to encounter. There is also
the issue that many functions drop down to operating on native float64
or int internally (floor, log, pow, etc.). I chose to only handle
the Multiply and Modulo functions at this time, since they were
already using arbitrary precision values throughout.

  • Do not lose precision in module function:

When working with big.Float values, it is imperative to ensure the
desired precision for the operations is maintained throughout. The
default precision for a Float is set initially by the max argument
precision when creating the non-zero value. If this is coming from a
Float which was set via SetInt, and that Int value was created by
SetInt64, then the precision is only 64 bits.

  • Don't overflow precision in Multiply:

The product of the arguments may require more precision than either
argument. The working precision was chosen to be 512, since that is
the value used with parsing large numbers from a string.

When working with big.Float values, it is imperative to ensure the
desired precision for the operations is maintained throughout. The
default precision for a Float is set initially by the max argument
precision when creating the non-zero value. If this is coming from a
Float which was set via SetInt, and that Int value was created by
SetInt64, then the precision is only 64 bits.
The product of the arguments may require more precision than either
argument.
@jbardin jbardin changed the title Float precision with multiplies maintain big.Float precision with multiplication and modulo Oct 29, 2020
@apparentlymart
Copy link
Collaborator

Looks great... thanks! 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants