Upgrade arb to 2.6.0

[cheuberg]

Upgrade the optional arb spkg to version 2.6.0.

Tar.gz file at https://www.math.aau.at/user/cheuberg/sage/arb-2.6.0.tar.gz

CC: @mezzarobba @fredrik-johansson @jhpalmieri

Component: packages: optional

Keywords: arb

Author: Clemens Heuberger

Branch/Commit: 922b674

Reviewer: Jeroen Demeyer

Issue created by migration from https://trac.sagemath.org/ticket/18560

[cheuberg]

Branch: u/cheuberg/libs/arb-2.6.0

[cheuberg]

Commit: adad105

[cheuberg]

Changed keywords from none to arb

[cheuberg]

New commits:

adad105

Trac 18560: Upgrade arb to 2.6.0

[sagetrac-git]

Changed commit from adad105 to 8d0b599

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

8d0b599

Trac #18560: fix doctests after arb upgrade

[mezzarobba]

comment:3

Replying to @sagetrac-git:

Branch pushed to git repo; I updated commit sha1. New commits:

8d0b599

Trac #18560: fix doctests after arb upgrade

Perhaps it would be better to add tolerances so that the tests pass with both versions of arb. What do you think?

[cheuberg]

comment:4

Replying to @mezzarobba:

Perhaps it would be better to add tolerances so that the tests pass with both versions of arb. What do you think?

• I do not know whether the doctest framework can handle this --- how would it cope with the special representation of a RealBall?
• I think that at least some kind of plausibility checking is interesting --- if the old and new balls do not intersect, we'd be in trouble.
• On the other hand, changing doctests with each new revision of arb seems to be uncomfortable, too.

[mezzarobba]

comment:5

Replying to @cheuberg:

Replying to @mezzarobba:

Perhaps it would be better to add tolerances so that the tests pass with both versions of arb. What do you think?

• I do not know whether the doctest framework can handle this --- how would it cope with the special representation of a RealBall?

The tolerance simply applies independently to the center and the radius, so that in many cases # abs tol 1e-15 will do something reasonable.

[jhpalmieri]

comment:6

This builds and passes its test suite on OS X 10.10.3.

[jdemeyer]

comment:7

bash-3.2$ ./sage -i arb
Found local metadata for arb-2.6.0
Found local sources at /Users/jdemeyer/sage/upstream/arb-2.6.0.tar.gz
Checksum: e8c367b9f617262b4a7e9956cf49b6acf5c0db24 vs 096b61d0576821d9ef6c1d0880df3d225f8bfdf3
Invalid checksum for /Users/jdemeyer/sage/upstream/arb-2.6.0.tar.gz

[sagetrac-git]

Changed commit from 8d0b599 to 6b54edc

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

48680ef	Revert "Trac #18560: fix doctests after arb upgrade"
6b54edc	Trac #18560: add tolerance for doctests

[cheuberg]

comment:9

Replying to @jdemeyer:

bash-3.2$ ./sage -i arb
Found local metadata for arb-2.6.0
Found local sources at /Users/jdemeyer/sage/upstream/arb-2.6.0.tar.gz
Checksum: e8c367b9f617262b4a7e9956cf49b6acf5c0db24 vs 096b61d0576821d9ef6c1d0880df3d225f8bfdf3
Invalid checksum for /Users/jdemeyer/sage/upstream/arb-2.6.0.tar.gz

I do not understand what happened.
I just tried it again:

$ wget https://github.com/fredrik-johansson/arb/archive/2.6.0.tar.gz
...
$ sha1sum 2.6.0.tar.gz 
e8c367b9f617262b4a7e9956cf49b6acf5c0db24  2.6.0.tar.gz

which is what the checksum should be.

I now put a copy of the tar.gz file to http://www.math.aau.at/user/cheuberg/sage/arb-2.6.0.tar.gz , could you please try that file (which also has a better file name for our purposes)? Thank you.

As I cannot reproduce the problem, I set the ticket back to needs_review.

[jdemeyer]

comment:13

OK, the problem was that curl (which I used to download the package) doesn't follow redirects.

[jdemeyer]

comment:14

Why are you choosing absolute tolerances over relative tolerances in the doctests?

[jdemeyer]

comment:15

Can you make the files spkg-install and spkg-check executable please?

[jdemeyer]

comment:16

About the doctests: I would actually prefer to just change the doctests without tolerance. It's kind of ugly to have to add tolerances everywhere. And most tests don't change, so it's not so bad.

[sagetrac-git]

Changed commit from 6b54edc to 3b534cb

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

3b534cb

Trac #18560: spkg-{check,install} executable

[cheuberg]

comment:19

Replying to @jdemeyer:

Why are you choosing absolute tolerances over relative tolerances in the doctests?

In most cases, the relative changes in the radius is around 0.1, but I would not like to allow a relative tolerance of 0.1 for the center of the ball. Unless there is a way of specifying different relative tolerances for radius and center of the ball?

[cheuberg]

comment:20

Replying to @jdemeyer:

About the doctests: I would actually prefer to just change the doctests without tolerance. It's kind of ugly to have to add tolerances everywhere. And most tests don't change, so it's not so bad.

It might well be the case that the tolerances I introduced above might be too small when the upgrade to the next arb version or that other tests require new tolerances. At the moment, we could easily go back to the modified doctests I already reverted once. And switching back to tolerances when we realize that we keep changing doctests with every arb version would not to be hard, either.

OTOH: I am not sure whether all machines will return the same error in all computations, but that's perhaps something that Fredrik could comment on?

Marc, you suggested using tolerances. Any further comments on this issue?

[mezzarobba]

comment:21

Replying to @cheuberg:

OTOH: I am not sure whether all machines will return the same error in all computations

As far as I understand, yes, they will.

Marc, you suggested using tolerances. Any further comments on this issue?

Not really. I agree with you on the choice of absolute tolerances, and I don't understand Jeroen's argument against tolerances, but honestly I don't care much.

[jdemeyer]

comment:22

Replying to @mezzarobba:

I don't understand Jeroen's argument against tolerances, but honestly I don't care much.

The problem with tolerances is:

1. since a ball is defined by 2 numbers, it will always be awkward to use tolerances since you cannot easily give different tolerances for the center and radius.
2. it makes reviewing harder, since a reviewer must check whether all the given tolerances make sense
3. arb is supposed to be system-independent, so we actually want to know if even the last digit of some result is wrong.

[fredrik-johansson]

comment:23

Results can differ on 32-bit and 64-bit system but this should not happen often. Gratuitous nondeterminism is avoided, but it's safest to think of arb as actually employing randomization, which unfortunately doesn't play well with doctesting. Two reasonable workarounds would be:

1. Print doctest examples with a few digits less than the computed precision, so that most of the radius bound in the printed output comes from the decimal rounding. This will only rarely cause a last digit to flip over unpredictably. The disadvantage is that the doctest does not show the actual computed interval. On the other hand, the output is a neater representation of the mathematical value. For example, if you print pi to 10 digits, you should always get [3.141592654 +/- 4.11e-10] beyond a certain working precision (from 43 bits, in fact) since the actual difference between pi and 3.141592654 is 4.102...e-10.

2. Implement a doctest directive that parses intervals and checks whether the actual and expected outputs overlap (this could also take a tolerance and verify that the new actual output is not too imprecise, e.g. 53-bit pi suddenly coming out as [3.1 +/- 0.1]).

[jdemeyer]

comment:24

The two statements seem to contradict each other:

Replying to @fredrik-johansson:

Results can differ on 32-bit and 64-bit system but this should not happen often. Gratuitous nondeterminism is avoided

it's safest to think of arb as actually employing randomization

[jdemeyer]

comment:25

Replying to @fredrik-johansson:

1. Print doctest examples with a few digits less than the computed precision, so that most of the radius bound in the printed output comes from the decimal rounding. This will only rarely cause a last digit to flip over unpredictably. The disadvantage is that the doctest does not show the actual computed interval. On the other hand, the output is a neater representation of the mathematical value. For example, if you print pi to 10 digits, you should always get [3.141592654 +/- 4.11e-10] beyond a certain working precision (from 43 bits, in fact) since the actual difference between pi and 3.141592654 is 4.102...e-10.

If we go with this, I would not make a difference between normal interactive Sage output and doctests. Just print balls with fewer digits always, like we already do for RR.

2. Implement a doctest directive that parses intervals and checks whether the actual and expected outputs overlap (this could also take a tolerance and verify that the new actual output is not too imprecise, e.g. 53-bit pi suddenly coming out as [3.1 +/- 0.1]).

This would require writing a much more complicated doctest parser (remember you don't just want balls, but also polynomials/lists/matrices with ball coefficients and then you'll also want support for complex numbers, mpfi intervals and who knows what), I don't think this is a realistic solution.

[cheuberg]

comment:26

Replying to @jdemeyer:

2. it makes reviewing harder, since a reviewer must check whether all the given tolerances make sense

I cannot tell whether it is harder to review tolerances or changes to doctests like in commit 8d0b599.

[cheuberg]

comment:27

Replying to @jdemeyer:

Replying to @fredrik-johansson:

1. Print doctest examples with a few digits less than the computed precision, so that most of the radius bound in the printed output comes from the decimal rounding. This will only rarely cause a last digit to flip over unpredictably. The disadvantage is that the doctest does not show the actual computed interval. On the other hand, the output is a neater representation of the mathematical value. For example, if you print pi to 10 digits, you should always get [3.141592654 +/- 4.11e-10] beyond a certain working precision (from 43 bits, in fact) since the actual difference between pi and 3.141592654 is 4.102...e-10.

If we go with this, I would not make a difference between normal interactive Sage output and doctests. Just print balls with fewer digits always, like we already do for RR.

I do not like the idea of always printing less information than we actually have. After all, we precisely know the error, so dilluting it seems somehow crude.

[jdemeyer]

comment:28

Replying to @cheuberg:

I do not like the idea of always printing less information than we actually have.

On the other hand, when do you really care about seeing the last few digits in a many-digit number? Like I said, we are already printing elements of RR like that and I don't think anybody ever complained about it.

[fredrik-johansson]

comment:29

Replying to @jdemeyer:

The two statements seem to contradict each other:

Replying to @fredrik-johansson:

Results can differ on 32-bit and 64-bit system but this should not happen often. Gratuitous nondeterminism is avoided

it's safest to think of arb as actually employing randomization

What I mean is that you might run through billions of values before you see a difference, but if you assume that a difference never occurs and implicitly base your code on this assumption (perhaps forgoing using the radius information before converting output to floating-point numbers), then it could potentially bite you in a serious way when it does.

[jdemeyer]

comment:30

Replying to @fredrik-johansson:

it could potentially bite you in a serious way when it does.

Well, doctests never "bite you in a serious way" so I'm willing to take my chances here.

If the problem turns out to be more common then expected, we can still change our strategy.

[fredrik-johansson]

comment:31

As for the number of digits to print, why not print many digits by default but do something like this in (some of the) doctests:

RBF = RealBallField(53, print_digits=10)

[jdemeyer]

comment:32

Replying to @fredrik-johansson:

As for the number of digits to print, why not print many digits by default but do something like this in (some of the) doctests:

RBF = RealBallField(53, print_digits=10)

-1 because it clutters the doctests for no obvious advantage.

[cheuberg]

Changed branch from u/cheuberg/libs/arb-2.6.0 to u/cheuberg/libs/arb-2.6.0-no-tolerance

[cheuberg]

Changed commit from 3b534cb to 922b674

[cheuberg]

comment:33

Replying to @jdemeyer:

Well, doctests never "bite you in a serious way" so I'm willing to take my chances here.

If the problem turns out to be more common then expected, we can still change our strategy.

There is only one way to find out how common that problem will be ...

Thus I attach a branch without tolerances, but with modified doctests (as before), plus spkg-{check,install} executable as discussed above.

---

New commits:

922b674

Trac #18560: spkg-{check,install} executable

[jdemeyer]

Reviewer: Jeroen Demeyer

[vbraun]

Changed branch from u/cheuberg/libs/arb-2.6.0-no-tolerance to 922b674