math/big: extend ProbablyPrime with Baillie-PSW test

[jfcg]

BPSW is the current popular way for probabilistic primality testing.

More detailed info about BPSW is at https://en.m.wikipedia.org/wiki/Baillie–PSW_primality_test

A patch and initial discussion is at https://groups.google.com/forum/m/#!topic/golang-dev/AOAbwvCwgwo

[minux]

I don't like the idea of use negative n in ProbablyPrime(n) to test for primality using Baillie-PSW. That will limit us to Miller-Rabin and Baillie-PSW tests forever. What if there are even better tests in the future? (Say if a future version of AKS primality test become practical or a probability version of deterministic AKS primality test.) If we want to add it to math/big, we should choose another name. But then what should the user choose? There is no proof that Baillie-PSW doesn't have false positives, but there is clear and well-understood bound for Miller-Rabin test giving false positives. Perhaps we can add it to a x/math subrepo and see how many people actually cares about the speed of primality testing and also willing to take the risk of using algorithm with unproven result.

[jfcg]

We can use BPSW for n=0 and reserve n<0 for any future improvement.

I agree that there should not be multiple apis for primality testing. In function doc we clearly inform users of what we use for each n and what the success guarantees are.
For a composite, failure rate of a single MR is 1/4. For BPSW:

BPSW is available since 1980, people have tried every available prime & composite on the planet and it stood the test of time. It has a sound well-studied algorithm, notice it is an extension of MR. No serious library/language/software uses only MR for testing. Everybody suplements MR with a Lucas(jacobi=-1).
They "complement" each other. Blindly rejecting this without reading the relevant literature is a deep ignorance & shame for the Go project.

This is probabilistic primality testing, as the name suggests. We dont claim to have a primality proof in the first place. This does not mean we can provide lousy algorithms to users. On the contrary BPSW is the state-of-the-art primality testing.

[cznic]

We can use BPSW for n=0 and reserve n<0 for any future improvement.

I agree with @minux that the existing function should be kept alone.

BPSW is available since 1980, people have tried every available prime & composite on the planet and it stood the test of time.

This is mathematically irrelevant. The lack of proof is what matters. I object to putting an unproven, possibly failing primality-checking algorithm into the stdlib.

[jfcg]

Have you read "The danger of relying only on Fermat tests" at https://en.m.wikipedia.org/wiki/Baillie–PSW_primality_test ??

[ALTree]

is a deep ignorance & shame for the Go project

Please be civil. There's no shame in wanting to understand advantages and disadvantages of a proposal before implementing it. You failed to provide some information.

without reading the relevant literature

you did not provide "the relevant literature" (wikipedia is not literature). I understand that you may expect a reviewer to go on wikipedia and then doing a literature search, but this is really something that you should help with as part of your proposal.

I have a few comments. First of all:

[Java's isPrime for bigInts] does one or more Miller-Rabin tests with random bases. If n, the number being tested, has 100 bits or more, this method also does a non-strong Lucas test

So what they do is MR for bits < 100 and MR + Lucas on bits >= 100. But this is not what you're doing in your patch. First of all there's no 100 threshold: you make the caller choose what the method does - i.e. n > 0 means MR and n <= 0 means calling sprp() && slprp() (as a side note, the function names you have choosen really do not help understanding your code). This is confusing. Java's behaviour is nice: the method configures itself. This would not be the case in go (at least not with your proposal).

When should the user call isProbablyPrime with n == 0 ? And when not? You have to explain this (since we'll have to explain it to our users if we are to implement this).

Maple's isPrime function requires no configuration. Mathematica's PrimeQ function requires no configuration.

Also Java uses a non-strong Lucas test. Your patch seems to test for strong-Lucas primality. Care to explain why?

Second:

GNU Multiple Precision Arithmetic Library's mpz_probab_prime_p function uses a Miller-Rabin test, but does not use a Lucas test.

GMPL is is a widely used bignum library and what they do is MR. I'm not saying we should use pure MR just because GMPL does that, but the fact that they do suggests that a pure MR primality test is not an horrible crime. Go big package is much much simpler and smaller than GMPL.

Third: you have been asked several times to explain what guarantees (if any) the test gives regards to the probability of false-positives or false-negatives. The current PM implementation is pretty clear about that (the usual 4^-k bound). What about BPSW?
In your patch you only write

returns true only if x is a probable prime

maybe it's me, but this mean everything and nothing.

Fourth: performances. Is this faster or slower? You only did a single benchmark with n = 100 on MR, but almost nobody will call MR with parameter 100. n = 20 is a tipical value.

[jfcg]

Alberto,
If you really cared, you would have read my patch and see the two relevant literature references as comments.
Also, yes I would appreciate if a reviewer god damn reads the .ucking wiki article. I did not expect them to do a literature research on the subject!
Just read the comments made to this issue and the original thread. Most commenters really did not care to read the wiki or the patch.

bits<100 is just a made up limit, I havent read or come up with any "bit limit" on when lucas should be used. In any case, real applications will care for primes of 200+ bits. For example EdDsa uses 250+ bits primes.

sprp: strong probable prime (standard term in number theory)
slprp: strong lucas probable prime (standard term in number theory)
Really easy definitions, you would quickly recognize them if you cared to read my patch, the wiki or the papers ;)

As you know, Go has a compatibility promise. I did not want to clutter the api, just extend the current api with how Sage/Pari/Gp does it. Notice that there is no loss of MR(n) functionality.
That's why we need 'configuration'.
Users just use whatever they want. n=0 for BPSW, n>0 if they want MR(n).

If you cared to read some basic info about primality testing, you would not ask why we use the strong versions, but i'll explain in any case: strong tests are better detectors of primes/composites. Also the existing MR code is itself a strong probable prime test ;)

Gmp is the only serious lib that does not employ lucas tests, dont know why. Why dont you ask them and let us know.

BPSW is:

• either necessary/sufficient (notice the math term;) for primality
• or it has an extremely extremely low error rate (MR's is 1/4). Nobody was able to provide a composite that fools this test since 1980! You can contruct composites that fool MR(n=64) for example.

Nobody knows which yet.

You should use BPSW, if not, you should use MR(n=40+) for a serious application. BPSW is faster than the latter case.

Probable prime means:

• either prime
• very rare composite that fooled your prp test.

That is the mathematically correct term for what all these functions produce.

I hope I covered all your questions, let me know if I missed anything.

[ALTree]

First of all: I did not study the patch carefully because we are not doing code review. Understanding if your proposed extension is fine should not require one to study (or even open) a code patch. And certainly not one with functions named sprp and slprp.

sprp: strong probable prime (standard term in number theory)
slprp: strong lucas probable prime (standard term in number theory)
Really easy definitions, you would quickly recognize them if you cared to read my patch, the wiki or the papers ;)

This is a non-argument. I quickly glanced to the diff in ProbablyPrime, and what I found is

+ return x.abs.sprp(natTwo, nm1, q, k) && x.abs.slprp(b == 2)

compare with Java SDK implementation

return passesMillerRabin(rounds, random) && passesLucasLehmer();

As you know, Go has a compatibility promise. I did not want to clutter the api, just extend the current api with how Sage/Pari/Gp does it. Notice that there is no loss of MR(n) functionality.
That's why we need 'configuration'.
Users just use whatever they want. n=0 for BPSW, n>0 if they want MR(n).

Yes, I understand that. And yet everyone who has read your proposal said that the proposed API is ugly. Maybe a new fuction would be better (as Minux said). There's no agreement, not even on the signature of the function, and yet you expect someone to carefully study the whole patch.

If you cared to read some basic info about primality testing, you would not ask why we use the strong versions, but I'll explain in any case: strong tests are better detectors of primes/composites. Also the existing MR code is itself a strong probable prime test ;)

I know the difference between strong and weak tests, thank you. My point was that you can't both write "we should do it this way because this is how everyone is doing it" and avoid being questioned about remarkable differences between your implementation and other languages/libraries/platforms implementations.

I look at Java's implementation and I see weak-Lucas, I look at GMPL and they don't even do Lucas. Mathematica and PARI/GP do exactly what you're doing in your patch. The sage reference returns 404. Perl and Python implement Lucas and BPSW in external libraries.

To sum the evidence up:

• Mathematica and PARI/GP do BPSW out of the box for isPrime
• Java does MR + weak-Lucas after a certain threshold
• GMPL does MR only
• Other languages provide Lucas and BPSW in external libraries

You should use BPSW, if not, you should use MR(n=40+) for a serious application. BPSW is faster than the latter case.

How much faster for n=40 ?

Probable prime means:
either prime
very rare composite that fooled your prp test.
That is the mathematically correct term for what all these functions produce.

I was specifically asking if we do have a bound. You write "very rare", I deduce that a bound is not known. That's fine.

IMHO the first thing to do is decide how the API will look if this goes in (new function? n <= 0 trick? Out-of-the-box Lucas in isProbablyPrime?).

[rsc]

The docs are very clear:

// ProbablyPrime performs n Miller-Rabin tests to check whether x is prime.
// If x is prime, it returns true.
// If x is not prime, it returns false with probability at least 1 - ¼ⁿ.
//
// It is not suitable for judging primes that an adversary may have crafted
// to fool this test.

While we could start overloading non-positive values of n, that will be confusing to read and also confusing when you compile code expecting the new meaning of (say) 0 but instead it panics saying 0 is not a valid argument.

There are two possible ways forward for adding other primality test algorithms:

1. If a test makes sense to do all the time or in some limited context (such as small numbers), just do it, in addition to the currently-guaranteed Miller-Rabin tests.
2. Add a new method.

There have been a few things suggested here, including BPSW and various Lucas variants. I would like to see, for each test being proposed, a crisp statement containing:

• the proposed new test
• the class of numbers being tested for which the test makes sense over the current implementation, and
• the cost compared to one Miller-Rabin iteration.

[rsc]

@jfcg, you wrote:

Blindly rejecting this without reading the relevant literature is a deep ignorance & shame for the Go project.

to which @ALTree reasonably replied:

Please be civil. There's no shame in wanting to understand advantages and disadvantages of a proposal before implementing it. You failed to provide some information. ... you did not provide "the relevant literature" (wikipedia is not literature). I understand that you may expect a reviewer to go on wikipedia and then doing a literature search, but this is really something that you should help with as part of your proposal.

To which you replied:

If you really cared, you would have read my patch and see the two relevant literature references as comments. Also, yes I would appreciate if a reviewer god damn reads the .ucking wiki article.

This is not really a productive way to have a conversation. You're not doing anything to help your case.

[jfcg]

Hi,
I directly time each call x.ProbablyPrime(n) with my patch on core i5 4210m laptop with ubuntu 14.04, 3.19 kernel.
For starters, I got these numbers. Separated by type:

// composite fermat/mersenne numbers, strong pseudoprimes to base 2
// 2^256+1, 2^512+1, 2^257-1, 2^509-1
n= 40
257 bits took 207.514µs
513 bits took 584.49µs
257 bits took 69.83µs
509 bits took 207.916µs
total: 1.06975ms

n= 0
257 bits took 911.742µs
513 bits took 1.25268ms
257 bits took 295.564µs
509 bits took 837.612µs
total: 3.297598ms

// 5 primes with 256 256 697 730 995 bits
n= 40
256 bits took 1.745521ms
256 bits took 1.719863ms
697 bits took 18.613694ms
730 bits took 21.783088ms
995 bits took 46.10569ms
total: 89.967856ms

n= 0
256 bits took 517.818µs
256 bits took 509.545µs
697 bits took 3.270944ms
730 bits took 4.458831ms
995 bits took 7.606551ms
total: 16.363689ms

// pairwise products of 5 primes
n= 40
512 bits took 229.017µs
953 bits took 959.979µs
986 bits took 1.142523ms
1251 bits took 2.032602ms
953 bits took 946.264µs
986 bits took 1.110485ms
1250 bits took 2.087854ms
1427 bits took 3.505977ms
1691 bits took 4.834519ms
1725 bits took 4.44336ms
total: 21.29258ms

n= 0
512 bits took 496.858µs
953 bits took 1.881229ms
986 bits took 2.041718ms
1251 bits took 4.105593ms
953 bits took 1.907105ms
986 bits took 2.037402ms
1250 bits took 3.454044ms
1427 bits took 5.377545ms
1691 bits took 7.636516ms
1725 bits took 7.461523ms
total: 36.399533ms

I was wrong for composites, MR(n=40) works faster for them. I think that sounds right because for a prime, MR needs to try all the 40 bases, for a composite there are chances to get eliminated for each base.

I think MR enjoys a well implemented expNN, lucasMod is bare minimum.

There are constant costs as well. I'll make polynomial fits to these data & come back :)

[minux]

Speed is not the only concern. The major client for ProbablyPrime is crypto/rsa. I did a survey of major open source crypto libraries to see how do they test for primes when generating RSA keys: OpenSSL: https://www.openssl.org/docs/manmaster/crypto/BN_generate_prime.html Only uses Miller-Rabin probabilistic primality test, with bounded false positive rate of at most 2^-80 for random input. NSS: function mpp_make_prime in http://hg.mozilla.org/projects/nss/file/tip/lib/freebl/mpi/mpprime.c#l392 uses Fermat test followed by Miller-Rabin tests. CryptoPP: RSA key generation uses the VerifyPrime function at https://github.com/weidai11/cryptopp/blob/master/nbtheory.cpp#L249 and VerifyPrime uses strong lucas test followed by Miller-Rabin tests. It's interesting to see that: 1. all of them uses MR test in the last step (ranging from 2 rounds to 40 rounds, our crypto/rand.Prime uses 20 rounds) 2. none of them uses Baillie-PSW, 3. only CryptoPP uses strong lucas test.

[rsc]

Let's take cryptography explicitly off the table. This is about math/big.ProbablyPrime, not crypto/rand.Prime. There are many other concerns for the generation of large random primes in a cryptographic context, and in fact in that case only a few rounds of Miller-Rabin are required. See Menezes et al., Handbook of Applied Cryptography, 1997, pp. 145-149. If you can't find that, see Appendix F of FIPS 186-4, which has some of the same information.