Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The maximum possible number of significant digits may be too low #2616

Closed
Roberto-Colombo opened this issue Nov 26, 2021 · 1 comment
Closed

Comments

@Roberto-Colombo
Copy link

Ref: https://stackoverflow.com/questions/70122920/maximum-allowed-value-of-the-precision-format-specifier-in-the-fmt-library

The following lines in format-inl.h

// Limit precision to the maximum possible number of significant digits in
// an IEEE754 double because we don't need to generate zeros.
const int max_double_digits = 767;
if (precision > max_double_digits) precision = max_double_digits;

seem to cause an unexpected truncation in the representation of floating-point numbers.

Compare the results of this snippet (a)

std::ostringstream oss;
oss << std::fixed << std::setprecision(1074) << 0x0.7ffffffffffffp-1022;
auto const repr{ oss.str() };
std::cout << "Size: " << repr.size() << "\nValue: " << oss.str() << '\n';

To this one (b)

auto repr{ fmt::format("{:.1074f}\n", 0x0.7ffffffffffffp-1022) };
fmt::print("Size: {}\nValue: {}\n", repr.size(), repr);

It seems to me that the code using max_double_digits should take into account the leading zeroes after the decimal point.

@vitaut
Copy link
Contributor

vitaut commented Nov 27, 2021

Fixed in c472a27. Thanks for reporting!

@vitaut vitaut closed this as completed Nov 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants