faster rand!(::MersenneTwister, ::Array{T}) for IntTypes and Float16/32

[rfourquet]

This PR tries to waste less random bits for array filling, and also uses dsfmt_fill_array_close1_open2 function on non-Float64 arrays.

Edit: better timings using this function:

function arr(T, n)
    a = Array(T, n)
    m = MersenneTwister()
    tic()
    rand!(m, a)
    toc()
end
for T in [Int8, Int16, Int32, Int64, Int128, Float16, Float32]
    arr(T, 10^8÷sizeof(T))
end

Type	PR	master (64aa84d)	master/PR
Int8	0.364	3.74	10.3
Int16	0.367	1.94	5.3
Int32/Int64/Int128	0.360	1.05	2.9
Float16	2.71	5.37	2.0
Float32	0.455	1.27	2.8

The results are similar for unsigned types (and also a 3x speedup for rand!(::BitArray)).
(I would be interested if someone finds very different speedups).

[ViralBShah]

Should we have some more tests for all the new stuff?

[ViralBShah]

It would be great to get a few more eyes on this.

[ViralBShah]

@rfourquet The refactoring in this PR is perfect and can be merged right away. The other stuff I would like some eyeballs on. In general, what would help is if you can make smaller PRs that are focussed on one particular feature, so that it is easy to go with the merges.

[rfourquet]

OK, will try! (I admit I didn't expect that a refactoring without added functionality would be welcome).
This definitely need some eyeballs, there is room for mistakes.
I'm not sure if this need some more tests, as it's only internal implementation changes (no modified public API).

[staticfloat]

The only test I can think of is running Bigcrush on this to ensure that we're still providing the same level of randomness as before.

[ivarne]

We should at the bare minimum have tests that ensure that all the code paths are executed, and that no exceptions are raised. It is really sad to get a UndefVarError in a bug report because you had a typo in a code path that you never executed.

We can potentially also check return type and range.

[andreasnoack]

I have just tried BigCrush on the scalar and array versions and they still pass all tests but LinearComp which they consistently fails (which is well known for the MT).

[ViralBShah]

@rfourquet Refactoring that makes the code more readable or clear in any way is always acceptable. Also, we generally do not have enough tests for all the random stuff - the existing tests are in no way sufficient. So, as suggested, we should try to have more tests to exercise all code paths and whatever we can test of RNGs in the test framework.

@andreasnoack Does your BigCrush test suite run tests for all the float and int precisions?

[rfourquet]

I added some tests which should exercise most code paths for added rand! methods (suggestions welcome, I'm a beginner at writing tests).

[rfourquet]

I finally learnt how to use RNGTest, so I could check that BigCrush tests pass for all int precisions.
But for floats I don't understand: testu01 tests only double precision floats, so testing directly the raw FLoat32 (via Float64(rand(Float32))) fails many tests.
Then following the description of the function unif01_CreateDoubleGen (at page 11 of the manual), I tried to obtain 53 bits precision numbers from 3 Float32 (wich have 24 bits of precisision) with f53() = (rand(Float32)+rand(Float32)/exp2(24)+rand2(Float32)/exp2(48)) % 1.0. But testing f53 passes BigCrush!
So what would be a good method to test Float32 and Float16 without finding them passing more tests than FLoat64?

[andreasnoack]

I think your methods is fine. Why is not okay that the tests pass?

[rfourquet]

As the rand! method for Float16 and Float32 use the Float64 one, it seemed suspicious to me that they pass more tests (i.e. scomp_LinearComp) than raw Float64. But now I realize that one Float64 (52 bits of entropy) is used to initialize two Float32 (which need only 2x23=46 bits en entropy) or 4 Float16 (which need 4x10=40 bits of entropy), so that may be okay after all. Speaking of what, would we want an option to generate a Float64 which pass all BigCrush, which consumes more that one FLoat64 internally?
@andreasnoack I seem to remember seeing somewhere that you tested randn with RNGTest, is that possible?

[andreasnoack]

I define testrandn()=cdf(Normal(), randn()) and pass that function to BigCrush. The surprising result is that is passes the tests even though rand doesn't.

I don't think we should modify our Mersenne-Twister to pass the scomp_LinearComp. Instead I think it would be better to supply alternative RNGs that pass all tests, e.g. the xorshift*, which we already have several gist versions of.

[rfourquet]

Ah thanks!

[ViralBShah]

It would certainly be interesting to see how to make Float64 pass BigCrush, and what it means for performance and APIs.