Test suite runs for over an hour

[peti]

The RSA test suite runs for over an hour on fairly modern hardware. This feels a little excessive. Is there a way to build the test suite so that it performs testing that's a little less thorough but completes in, say a minute or so?

[acw]

This is going to be a very long answer to your question.

Right now, a number of tests are very slow because they randomly generate a new key pair, between 512 and 4096 bits, for each test. Which I like because it gives a lot of coverage, but also makes things ridiculously slow.

I'd certainly be happy to see a version that pre-generated (either via hard coding or memorization) a few keys and then just used them over and over for the tests. You could even restrict some of these to smaller bit sizes to make the math faster. However, I'd love to keep the slow version for completeness. I actually have had some failures pop up that I wouldn't have caught without this feature.

If someone were to implement the pregeneration option, I'd imagine it would take the form of a cabal flag (-fvery-long-tests) that would set a CPP flag (VERY_LONG_TESTS), and then have the Arbitrary instances for LargePrime and KeyPair in Test.hs selected based on that option (the current version if VERY_LONG_TESTS, an invocation of elements if not). I'm happy for False to be the default for the flag; I can switch my computer and test servers to set it.

I'd be very happy to accept a patch to do the above. :) Otherwise, it might take awhile to get in, but I'll leave the ticket open as a reminder.

[peti]

I maintain several Haskell packages in NixOS. Usually, we run the test suite as part of every build (and report failures upstream) to ensure that our distribution is in a good state. With the RSA package, however, the time it takes to run the test suite is approx. 60 times longer than the time it takes to compile the entire package! We cannot afford to block our build machines for such a long time for a relatively small package like RSA. (We do run some very extensive test suites for packages like autoconf, but that's a different matter, IMHO.) For the time being, I've simply disabled the check phase in our builds.

Having a Cabal flag to choose a faster version of the suite for our purposes would be great! If that feature becomes available, then I'll re-enable the test suite.

[nomeata]

Also affects Debian, of course. Having a sensible test suite by default would be great.

[acw]

I've committed a version that should take much less time, although it still takes a few minutes. Tell me if it works for you.

It has not yet been uploaded to Hackage due to a Travis issue I'd like to fix before I bump version numbers again.

[acw]

I'm going to assume that the lack of response means "works for me" ...

[nomeata]

In this case it means that Debian is lagging behind with updating, but I’ll raise my voice if it is still a problem when we upgraded to the latest release.

[peti]

The test suite of RSA-2.1.0.1 still exceeds a 2 hour timeout on our build farm: http://hydra.nixos.org/build/25329830/nixlog/1/raw.

[acw]

I will look at it again, but understand that the test suite takes 30 minutes on my three+ year old laptop, and testing large number math is fundamentally time consuming.

If I added a flag (-fminimal-tests), is that the sort of thing that's easy for you to add?

[peti]

Yes, a Cabal flag that enables a shorter test run would help us greatly!

[acw]

Looks like the bigger problem was that I never uploaded my earlier fix to Hackage. Ugh. Sorry about that.

RSA-2.1.0.2 and 2.1.0.3, which should've been released in mid-April, should fix your problems. The test suite now takes under a minute on my laptop. It does not require any special flags.

[peti]

OK. Thank you for taking care of this.