math: Add scale argument to "round"

[kseistrup]

The fish math functions seems to get it wrong sometimes, when doing integer math:

> math --scale=0 "22 / 5"         # this is all fine
4
> math --scale=0 "22 / 5 - 5"     # this ought to give -1
-0
> math --scale=0 "-5 + (22 / 5)"  # parentheses or reverting expression doesn't help
-0

This has been a problem for all fish3 releases, as far as I can tell, but I always thought it was an error in my scripts.

> fish --version
fish, version 3.5.1
> uname -a
Linux home 5.18.16-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 03 Aug 2022 11:25:04 +0000 x86_64 GNU/Linux

[faho]

Integer math is truncated, not rounded. The documentation explains this.

[faho]

After a further look, the wording there is fairly confusing:

A scale of zero causes results to be rounded down to the nearest integer. So 3/2 returns 1 rather than 2 which 1.5 would normally round to. This is for compatibility with bc which was the basis for this command prior to fish 3.0.0. Scale values greater than zero causes the result to be rounded using the usual rules to the specified number of decimal places.

So, yes, it does explain that it truncates via the example, but "rounded down" and "to the nearest" is somewhere between misleading and outright wrong. I've now tried to improve that.

Anyway, this was implemented as an awkward compatibility compromise, and so it probably needs to stay an awkward compatibility compromise, because I don't believe it's worth the cost of fixing it (because finding places where people even use integer math is hard, let alone figuring out which rounding they expect).

So this will just be how the cookie crumbles.

[kseistrup]

I respect your decision, but finding a place where people use integer math is not hard:

• https://codeberg.org/kas/qtime/issues/5

And bc does get it right:

$ echo '22 / 5 - 5' | bc --scale=0
-1

As I said, I respect your decision, but the conclusion will be that using integer math in fish to calculate array indices cannot be done reliably and should be avoided.

[kseistrup]

And PS:

Integer math is truncated, not rounded.

That is what I expected in my calculations. math 22 / 5 should give 4, and math 22 / 5 - 5 should give the same as math 4 - 5: -1.

[faho]

I respect your decision, but finding a place where people use integer math is not hard:

Not finding a place, finding all the places.

I'm talking about the issues with a compatibility break.

[kseistrup]

I stand corrected.

[krobelus]

I think we should fix this because the other behavior doesn't make sense.
As you point out, truncating only at the end should results in incorrect evaluation of parenthesized expressions (compared to manual evaluation). We should truncate eagerly.

---

my bc doesn't accept --scale but I can set the scaling

$ echo 'scale=0; (22 / 5) - 5' | bc
-1
$ echo '(22 / 5) - 5' | math --scale=0
-0

[krobelus]

Hmm okay it's not so simple.

The problem you are seeing is that bc (hence fish 2.x) use scale for
the precision of all computations, whereas the new builtin math uses the
scale only when printing the results.

Another test case to illustrate the issue:

$ echo 'scale=3; (1 / 3) * 4' | bc
1.332
$ echo '(1 / 3) * 4' | math --scale=3
1.333

This example suggests that the behavior of math is actually better because it gives a more precise result.
Also it's closer to what Python does, which is a good sign.

In Python you'd just use explicit truncation which we can also do:

$ math 'floor(floor(22 / 5) - 5)'
-1

I don't think we should try to go back bc semantics here.
Since the confusion stems from the fact that --scale only applies to the output, how about deprecating it in favor of something like round(x, scale)?
We'd just need to add an optional second argument to round().