Skip to content
Permalink
Browse files
Correct longAsInt output in the stress test application
RNG-171 changed the cached int source to output a long as two ints using
the low 32-bits, then high 32-bits.

Updated the RngDataOutput to support HiLo or LoHi. Added tests for each
implementation.

Dropped high-bits and low-bits command options. Added source64 option
with enum for high, low, high-Low, low-high, long, int. The default for
stress testing matches the caching implementation.

Update the test documentation to reflect the changes.
  • Loading branch information
aherbert committed Mar 29, 2022
1 parent 3c4184b commit cc6b0c0b40499fb698730d94dc9c2d66f4176ebe
Showing 10 changed files with 635 additions and 125 deletions.
@@ -59,12 +59,12 @@ This will create a single jar file with all the dependencies in the `target` dir
contains a simple command to output data from a random generator. To output data in both text
and binary format use the following commands:

java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 100000 \
java -jar target/examples-stress.jar output KISS -s 1 -n 2048000 \
-f DIEHARDER -o test.dh
java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 100000 \
-f BINARY -o test.big -b big_endian
java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 100000 \
-f BINARY -o test.little -b little_endian
java -jar target/examples-stress.jar output KISS -s 1 -n 1000 \
-f BINARY -o test.big -b big_endian --buffer-size 8192
java -jar target/examples-stress.jar output KISS -s 1 -n 1000 \
-f BINARY -o test.little -b little_endian --buffer-size 8192

This should produce the following output files:

@@ -74,6 +74,8 @@ This should produce the following output files:
| test.big | Binary file using the big-endian format |
| test.little | Binary file using the little-endian format |

Note that the `-n` parameter is the count of numbers in text output but the count of buffers in binary output. The example above has 2048 4-byte integers per 8192 buffer. The `-n` parameter has been adjusted so the output numbers are the same.

The data can then be used to run a test within **Dieharder**:

dieharder -g 202 -d 0 -f test.dh
@@ -89,10 +91,10 @@ written direct to dieharder. For the birthdays test (`-d 0` option) 20,000,000 s
In the following example the stress test application directly writes to `stdout` which is then
piped to the `dieharder` application which reads using `stdin` (`-g 200` option):

java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 20000000 \
java -jar target/examples-stress.jar output KISS -s 1 -n 20000000 \
-f DIEHARDER -o test.dh
dieharder -g 202 -d 0 -f test.dh
java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 20000000 \
java -jar target/examples-stress.jar output KISS -s 1 -n 10000 \
-f BINARY -b little_endian | dieharder -g 200 -d 0

If the results are not the same then the second command can be repeated with `-b big_endian`.
@@ -150,7 +152,7 @@ integer value and the signed integer value. The contents will be similar to:
01011100 10001111 11110001 11000001 1552937409 1552937409
10110000 01110101 10010011 00011100 2960495388 -1334471908

The `stdin2testu01` has been written to output the same format when using the `raw32` mode.
The `stdin2testu01` has been written to output in the same format when using the `raw32` mode.
If the data has been correctly read the `bridge.data` and `bridge.out` should match.
If the endianess is incorrect then the data sent by the Java application will not match the
data read by the sub-process. For example to swap the endianness use the `-b` option:
@@ -215,7 +217,7 @@ The number data in the file should be identical if the endianness was correctly
The only difference should be the the header and footer added by the `stress` command
to the results file. If the endianness is incorrect for the `output` command then the number
data will not match. Note that the `output` command by default uses big endian for consistency
across all platforms.
across all platforms; the `stress` command uses the native byte order of the platform.

An equivalent 64-bit output would use the `--raw64` option for each command:

@@ -131,8 +131,8 @@ public int next() {
// Open the stdin of the process and write to a custom data sink.
// Note: The 'bridge' command only supports 32-bit data in order to
// demonstrate passing suitable data for TestU01 BigCrush.
final boolean raw64 = false;
try (RngDataOutput sink = RNGUtils.createDataOutput(rng, raw64,
final Source64Mode source64 = null;
try (RngDataOutput sink = RNGUtils.createDataOutput(rng, source64,
testingProcess.getOutputStream(), buffer.capacity() * 4, byteOrder)) {
sink.write(rng);
}
@@ -126,25 +126,22 @@ class OutputCommand implements Callable<Void> {
description = {"Reverse the bits in the data (default: ${DEFAULT-VALUE})."})
private boolean reverseBits;

/** Flag to use the upper 32-bits from the 64-bit long output. */
@Option(names = {"--high-bits"},
description = {"Use the upper 32-bits from the 64-bit long output.",
"Takes precedent over --low-bits."})
private boolean longHighBits;

/** Flag to use the lower 32-bits from the 64-bit long output. */
@Option(names = {"--low-bits"},
description = {"Use the lower 32-bits from the 64-bit long output."})
private boolean longLowBits;

/** Flag to use 64-bit long output. */
@Option(names = {"--raw64"},
description = {"Use 64-bit output (default is 32-bit).",
"This is ignored if not a native 64-bit generator.",
"In 32-bit mode the output uses the upper then lower bits of 64-bit " +
"generators sequentially."})
"Set to true sets the source64 mode to LONG."})
private boolean raw64;

/** Output mode for 64-bit long output. */
@Option(names = {"--source64"},
description = {"Output mode for 64-bit generators (default: ${DEFAULT-VALUE}).",
"This is ignored if not a native 64-bit generator.",
"In 32-bit mode the output uses a combination of upper and " +
"lower bits of the 64-bit value.",
"Valid values: ${COMPLETION-CANDIDATES}."})
private Source64Mode source64 = RNGUtils.getSource64Default();

/**
* The output mode for existing files.
*/
@@ -166,12 +163,20 @@ public Void call() {
final Object objectSeed = createSeed();
UniformRandomProvider rng = createRNG(objectSeed);

// raw64 flag overrides the source64 mode
if (raw64) {
source64 = Source64Mode.LONG;
}
if (source64 == Source64Mode.LONG && !(rng instanceof RandomLongSource)) {
throw new ApplicationException("Not a 64-bit RNG: " + rng);
}

// Upper or lower bits from 64-bit generators must be created first.
// This will throw if not a 64-bit generator.
if (longHighBits) {
rng = RNGUtils.createLongUpperBitsIntProvider(rng);
} else if (longLowBits) {
rng = RNGUtils.createLongLowerBitsIntProvider(rng);
// Note this does not test source64 != Source64Mode.LONG as the full long
// output split into hi-lo or lo-hi is supported by the RngDataOutput.
if (rng instanceof RandomLongSource &&
(source64 == Source64Mode.HI || source64 == Source64Mode.LO || source64 == Source64Mode.INT)) {
rng = RNGUtils.createIntProvider((UniformRandomProvider & RandomLongSource) rng, source64);
}
if (reverseBits) {
rng = RNGUtils.createReverseBitsProvider(rng);
@@ -305,9 +310,9 @@ private static void stripArrayFormatting(List<String> arguments) {
*/
private UniformRandomProvider toOutputFormat(UniformRandomProvider rng) {
UniformRandomProvider convertedRng = rng;
if (rng instanceof RandomLongSource && !raw64) {
if (rng instanceof RandomLongSource && source64 != Source64Mode.LONG) {
// Convert to 32-bit generator
convertedRng = RNGUtils.createIntProvider(rng);
convertedRng = RNGUtils.createIntProvider((UniformRandomProvider & RandomLongSource) rng, source64);
}
if (byteOrder == ByteOrder.LITTLE_ENDIAN) {
convertedRng = RNGUtils.createReverseBytesProvider(convertedRng);
@@ -421,7 +426,7 @@ private void writeBinaryData(final UniformRandomProvider rng,
// If count is not positive use max value.
// This is effectively unlimited: program must be killed.
final long limit = (count < 1) ? Long.MAX_VALUE : count;
try (RngDataOutput data = RNGUtils.createDataOutput(rng, raw64, out, bufferSize, byteOrder)) {
try (RngDataOutput data = RNGUtils.createDataOutput(rng, source64, out, bufferSize, byteOrder)) {
for (long c = 0; c < limit; c++) {
data.write(rng);
}

0 comments on commit cc6b0c0

Please sign in to comment.