Skip to content

Commit

Permalink
DOC: Additional QA results
Browse files Browse the repository at this point in the history
  • Loading branch information
@bashtage
bashtage committed Jun 29, 2020
1 parent 5ca2cca commit 3c5ff88
Show file tree
Hide file tree
Showing 9 changed files with 592 additions and 419 deletions.
113 changes: 59 additions & 54 deletions doc/source/performance.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ Recommendation
**************
The recommended generator for single use is :class:`~randomgen.pcg64.PCG64DXSM`
although :class:`~randomgen.sfc64.SFC64` and :class:`~randomgen.xoshiro256.Xoshiro256`
are both excellent alternatives.
are both excellent alternatives. :class:`~randomgen.romu.Romu` is a newer generator that
is also very fast.

For very large scale
applications -- requiring 1,000+ streams --
Expand Down Expand Up @@ -37,33 +38,35 @@ it requires 2 32-bit values to equal the output of the faster generators.
:header: Bit Gen,Uint32,Uint64,Uniform,Expon,Normal,Gamma
:widths: 30,10,10,10,10,10,10

SFC64,2.3,2.5,2.4,5.1,10.6,18.7
Xoshiro256,2.5,2.5,2.5,5.2,9.7,19.1
PCG64DXSM,2.3,2.8,3.0,5.3,11.0,20.7
JSF,2.3,3.0,3.0,5.7,10.2,20.0
EFIIX64,2.5,3.0,3.0,5.5,10.7,20.8
PCG64,2.3,3.1,3.1,5.8,11.2,21.4
Xoshiro512,2.6,3.5,3.3,5.8,10.3,20.3
SFMT,2.9,3.3,3.1,6.3,10.9,20.8
LXM,2.6,3.5,3.5,6.3,11.3,21.9
PCG64(variant="dxsm-128"),2.8,3.3,3.5,6.1,12.5,24.3
DSFMT,3.0,4.2,2.7,7.0,12.1,21.6
MT64,2.8,4.0,4.2,6.8,12.8,23.7
JSF32,3.0,4.3,4.3,6.9,11.2,22.9
"Philox(n=2, w=64)",3.3,4.7,5.1,8.1,14.7,27.4
Philox,3.9,5.9,6.0,8.8,13.6,26.9
AESCounter,4.4,6.0,5.9,9.1,14.5,27.4
MT19937,3.8,6.2,7.2,9.2,14.8,28.7
"ThreeFry(n=2, w=64)",4.1,6.5,6.9,9.4,15.7,29.8
NumPy,3.0,4.7,5.8,16.7,20.8,40.2
HC128,4.1,7.2,7.2,10.5,16.6,31.6
"Philox(n=4, w=32)",4.2,7.6,8.5,10.7,16.5,32.6
SPECK128,5.4,8.1,9.6,11.3,17.0,33.3
ThreeFry,5.8,9.1,9.3,12.0,16.7,34.9
ChaCha(rounds=8),6.7,10.2,10.2,13.2,18.0,36.4
ChaCha,9.6,16.4,16.4,19.6,24.3,48.9
"ThreeFry(n=4, w=32)",9.0,16.3,17.6,20.7,24.4,53.4
RDRAND,129.9,133.1,130.7,139.0,140.8,298.6
"Romu(variant=""trio"")",2.3,2.1,2.0,5.2,9.3,18.2
SFC64,2.3,2.5,2.4,5.1,9.6,18.7
Romu,2.5,2.5,2.4,5.0,9.5,18.8
Xoshiro256,2.5,2.5,2.5,5.2,9.8,19.2
PCG64DXSM,2.3,2.9,3.0,5.3,11.0,20.7
JSF,2.4,3.0,3.0,5.8,10.2,20.0
EFIIX64,2.5,3.0,3.0,5.4,10.5,20.4
PCG64,2.3,3.1,3.1,5.9,11.3,21.4
Xoshiro512,2.7,3.5,3.3,5.8,10.3,20.3
SFMT,2.9,3.3,3.1,6.3,11.0,20.9
LXM,2.6,3.5,3.5,6.3,11.3,21.9
"PCG64(variant=""dxsm-128"")",2.8,3.4,3.5,6.1,12.5,23.1
DSFMT,3.0,4.2,2.7,7.0,12.2,21.7
MT64,2.8,4.0,4.2,6.9,12.8,23.7
JSF32,3.0,4.3,4.3,6.9,11.2,22.9
"Philox(n=2, w=64)",3.2,4.7,5.1,8.1,14.7,27.4
Philox,4.0,5.9,6.1,8.8,13.6,27.0
AESCounter,4.4,6.0,5.7,9.1,14.4,27.4
MT19937,3.8,6.3,7.2,9.2,14.8,28.8
"ThreeFry(n=2, w=64)",4.1,6.5,6.9,9.5,15.7,30.6
NumPy,3.0,4.6,5.8,14.4,20.1,39.8
HC128,4.1,7.2,7.2,10.5,16.6,31.5
"Philox(n=4, w=32)",4.2,7.6,8.7,10.8,16.5,32.6
SPECK128,5.4,8.1,9.7,11.4,17.0,33.4
ThreeFry,5.9,9.1,9.3,12.0,16.7,34.9
ChaCha(rounds=8),6.7,10.4,10.3,13.2,18.1,36.4
ChaCha,9.7,16.6,16.4,19.6,24.4,49.0
"ThreeFry(n=4, w=32)",9.0,16.8,17.5,20.7,24.3,54.1
RDRAND,129.5,129.9,129.6,136.9,139.6,287.4

The next table presents the performance relative to NumPy's ``RandomState`` in
percentage. The overall performance is computed using a geometric mean.
Expand All @@ -72,32 +75,34 @@ percentage. The overall performance is computed using a geometric mean.
:header: Bit Gen,Uint32,Uint64,Uniform,Expon,Normal,Gamma,Overall
:widths: 30,10,10,10,10,10,10,10

SFC64,128,190,246,283,210,211,205
Xoshiro256,120,186,229,280,206,207,198
JSF,126,155,193,251,197,198,183
EFIIX64,120,154,193,265,191,194,181
PCG64DXSM,128,163,191,248,164,192,177
PCG64,129,149,185,247,179,185,175
Xoshiro512,111,132,177,251,195,195,171
SFMT,104,142,186,229,183,190,167
LXM,114,135,165,228,177,181,163
PCG64(variant="dxsm-128"),105,139,165,235,161,172,158
DSFMT,100,110,214,207,166,184,157
JSF32,99,108,134,209,179,173,145
MT64,105,117,137,211,157,167,145
"Philox(n=2, w=64)",89,98,114,177,137,145,123
Philox,75,79,95,165,148,147,112
AESCounter,67,77,102,159,139,144,109
MT19937,77,74,80,157,135,138,105
"ThreeFry(n=2, w=64)",72,71,84,153,128,133,102
HC128,72,65,80,137,121,125,96
"Philox(n=4, w=32)",71,61,68,134,122,122,92
SPECK128,55,57,60,127,118,119,83
ThreeFry,51,53,63,120,120,114,81
ChaCha(rounds=8),45,46,56,109,111,109,73
ChaCha,31,28,35,74,83,81,50
"ThreeFry(n=4, w=32)",33,28,33,70,83,76,49
RDRAND,2,3,4,10,14,13,6
"Romu(variant=""trio"")",128,219,282,278,215,219,217
SFC64,129,189,244,283,210,213,205
Romu,118,185,242,285,212,212,202
Xoshiro256,119,185,229,279,205,208,198
PCG64DXSM,128,163,190,271,182,192,183
JSF,126,155,193,250,197,199,182
EFIIX64,120,154,193,264,190,195,181
PCG64,128,149,184,246,178,186,175
Xoshiro512,111,131,176,249,195,196,170
SFMT,104,143,185,228,183,190,167
LXM,114,135,164,227,177,182,162
"PCG64(variant=""dxsm-128"")",105,138,165,234,160,172,158
DSFMT,100,110,213,206,165,184,156
MT64,104,117,136,210,156,168,145
JSF32,99,108,134,208,178,174,145
"Philox(n=2, w=64)",91,98,114,177,136,145,124
Philox,75,79,95,163,147,147,112
AESCounter,67,77,102,158,139,145,109
MT19937,78,74,80,156,135,138,105
"ThreeFry(n=2, w=64)",72,71,84,152,128,130,101
HC128,72,64,80,136,121,126,96
"Philox(n=4, w=32)",71,61,66,134,122,122,91
SPECK128,55,57,59,127,118,119,83
ThreeFry,51,51,62,120,120,114,80
ChaCha(rounds=8),45,45,56,109,111,109,73
ChaCha,30,28,35,73,82,81,49
"ThreeFry(n=4, w=32)",33,28,33,69,82,73,48
RDRAND,2,4,4,10,14,14,7

.. note::

Expand Down
110 changes: 55 additions & 55 deletions doc/source/test-results.txt
View file
Original file line number Diff line number Diff line change
@@ -1,55 +1,55 @@
+------------------------------+-----------------------------------+-----------------------+
| Method | Seed Sequence | Jumped |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| Streams | 1 | 4 | 8196 | 4 | 8196 |
+==============================+===========+===========+===========+===========+===========+
| AESCounter | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| ChaCha(rounds=20) | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| ChaCha(rounds=8) | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| DSFMT⁴ | 4TB | FAIL¹ | 4TB | FAIL¹ | FAIL¹ |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| EFIIX64 | 1TB | 1TB | 1TB | -- | -- |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| HC128 | 4TB | 4TB | 4TB | -- | -- |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| JSF | 4TB | 4TB | 4TB | -- | -- |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| JSF(seed_size=3) | 4TB | 4TB | 4TB | -- | -- |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| LCG128Mix(output=upper) | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| LXM | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| MT19937⁴,⁵ | 4TB | FAIL¹ | 4TB | FAIL¹ | 4TB |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| PCG64DXSM² | 4TB | 1TB | 4TB | 1TB | 1TB |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| PCG64(variant=dxsm-128) | 1TB | 1TB | 4TB | 4TB | 1TB |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| PCG64⁵ | 4TB | 4TB | 1TB | 4TB | 4TB |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| Philox⁵ | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| Romu | 128GB | 128GB | 128GB | -- | -- |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| Romu(variant=trio) | 128GB | 128GB | 128GB | -- | -- |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| SFC64⁵ | 1TB | 4TB | 4TB | -- | -- |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| SFC64(k=3394385948627484371) | 1TB | 1TB | 1TB | -- | -- |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| SFC64(k=Weyl)³ | 4TB | 1TB | 1TB | -- | -- |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| SFMT⁴ | 1TB | FAIL¹ | 4TB | FAIL¹ | 1TB |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| SPECK128 | 4TB | 1TB | 4TB | 1TB | 4TB |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| ThreeFry | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| Xoshiro256 | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
| Xoshiro512 | 4TB | 4TB | 1TB | 4TB | 4TB |
+------------------------------+-----------+-----------+-----------+-----------+-----------+
+------------------------------+-----------------------------------------------+-------------------------------+
| Method | Seed Sequence | Jumped |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| Streams | 1 | 4 | 8196 | 4 | 8196 |
+==============================+===============+===============+===============+===============+===============+
| AESCounter | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| ChaCha(rounds=20) | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| ChaCha(rounds=8) | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| DSFMT⁴ | 4TB |FAIL at 64 GB¹ | 4TB |FAIL at 64 GB¹ |FAIL at 64 GB¹ |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| EFIIX64 | 4TB | 4TB | 4TB | -- | -- |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| HC128 | 4TB | 4TB | 4TB | -- | -- |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| JSF | 4TB | 4TB | 4TB | -- | -- |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| JSF(seed_size=3) | 4TB | 4TB | 4TB | -- | -- |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| LCG128Mix(output=upper) | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| LXM | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| MT19937⁴,⁵ | 4TB |FAIL at 64 GB¹ | 4TB |FAIL at 64 GB¹ | 4TB |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| PCG64DXSM² | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| PCG64(variant=dxsm-128) | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| PCG64⁵ | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| Philox⁵ | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| Romu | 4TB | 4TB | 4TB | -- | -- |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| Romu(variant=trio) | 4TB | 4TB | 4TB | -- | -- |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| SFC64⁵ | 4TB | 4TB | 4TB | -- | -- |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| SFC64(k=3394385948627484371) | 4TB | 4TB | 4TB | -- | -- |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| SFC64(k=Weyl)³ | 4TB | 4TB | 4TB | -- | -- |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| SFMT⁴ | 4TB |FAIL at 64 GB¹ | 4TB |FAIL at 64 GB¹ | FAIL at 4 TB¹ |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| SPECK128 | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| ThreeFry | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| Xoshiro256 | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
| Xoshiro512 | 4TB | 4TB | 4TB | 4TB | 4TB |
+------------------------------+---------------+---------------+---------------+---------------+---------------+
47 changes: 42 additions & 5 deletions doc/source/testing.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,12 @@ Quality Assurance
=================

A values below are the maximum output size where a bit generator or sequence of bit generators
has passed PractRand_.ꭞ A -- indicates that configuration is not relevant. Failures are marked
has passed PractRand_. A -- indicates that configuration is not relevant. Failures are marked
with FAIL. Most bit generators were only tested in their default configuration.
Non-default configurations are indicated by listing the keyword arguments to the bit generator.
Two sets of tests were performed. The first tested all configurations using 128GB of data using
PractRand's extended set of tests and additional bit folding. The second set of tests used
4TB of data wit the standard set of tests and folding.

All bit generators have been tested using the same :class:`~numpy.random.SeedSequence`
initialized with the same 256-bits of entropy taken from random.org.
Expand All @@ -14,18 +17,52 @@ initialized with the same 256-bits of entropy taken from random.org.

Notes
-----
ꭞ Testing is an on-going process. All generators will be tested to at east 4TB.

¹ Failed at 512GB.
¹ Failures at or before 128GB were generated by tests that used the expanded
set of tests and extra bt folds (``-te 1`` and ``-tf 2``). Failures at sample
sizes above 128GB were produces using the default configuration
(``-te 0`` and ``-tf 0``).

² PCG64DXSM and PCG64(variant=dxsm) are identical and so the latter not separately reported.

³ SFC64(k=weyl) uses distinct Weyl increments that have 50% or fewer non-zero bits.

⁴ The Mersenne Twisters begin to fail at 256GB. This is a known limitation of MT-family
⁴ The Mersenne Twisters begin to fail at 64GB. This is a known limitation of MT-family
generators. These should not be used in large studies except when backward compatibility
is required.

⁵ Identical output to the version included in NumPy 1.19.

.. _PractRand: http://pracrand.sourceforge.net/

Example Configuration
---------------------
All configurations are constructed using the same template. The code below tests a
configuration using 8,196 streams of :class:`~randomgen.aes.AESCounter`. The other
configurations simply make changes to either ``JUMPED`` or ``STREAMS``.

::

import numpy as np

import randomgen as rg

ENTROPY = 86316980830225721106033794313786972513572058861498566720023788662568817403978
JUMPED = False
STREAMS = 8196
BIT_GENERATOR_KWARGS = {}

SEED_SEQ = np.random.SeedSequence(ENTROPY)


BASE_GEN = rg.AESCounter(SEED_SEQ, **BIT_GENERATOR_KWARGS)
if STREAMS == 1:
bit_gens = [BASE_GEN]
elif JUMPED:
bit_gens = [BASE_GEN]
for _ in range(STREAMS - 1):
bit_gens.append(bit_gens[-1].jumped())
else:
bit_gens = []
for child in SEED_SEQ.spawn(STREAMS):
bit_gens.append(rg.AESCounter(child, **BIT_GENERATOR_KWARGS))
output = 64

0 comments on commit 3c5ff88

Please sign in to comment.