[Feature Request]: randomStream serializer / deserializer

[vasslitvinov]

Summary of Feature

As a Chapel user, I want to be able to write out a randomStream then read it in later to resume where I left off with it. I want to write it out in binary format for space efficiency.

Is this issue currently blocking your progress? No.

Code Sample

use IO, Random;

proc main() {
  var f = openMemFile();
  type eltType = int;

  // serialize it out
  {
    var w = f.writer(serializer=new binarySerializer());
    var rs = new randomStream(eltType, 314);
    w.write(rs);
  }

  // print binary contents of file
  {
    var r = f.reader();
    var data = r.readBytes(max(int));
    for b in data do
      writef("%xu ", b);
    writeln();
  }

  // deserialize it back in
  {
    var r = f.reader(deserializer=new binaryDeserializer());
    var rs = r.read(randomStream(eltType));
    writeln("read: ", rs);
  }
}

Thanks @benharsh for the code sample.

[vasslitvinov]

Implementation notes + discussion

Currently randomStream has a serializer that writes out only the seed, and no deserializer. This serializer, called writeThis at the time, was added in r20924 merely to avoid a deadlock. Recently, in #23752, the serializer got a comment "serialize the randomStream as a record with two fields: eltType and seed". However, I do not read that as a conscious design decision "a random stream shall not be saveable+restorable."

Since randomStream is POD, the compiler-generated de/serializers will suffice, so removing the existing one is a good first step. However, since randomStream defines its own initializers, the compiler does not generate a deserializing initializer , which needs to be defined. Also randomStream needs to be defined to implement certain interfaces other than or instead of the currently-declared writeSerializable -- other relevant interfaces are readDeserializable, initDeserializable, serializable (combines all three). We might also want a feature request to allow the user to opt into the default de/serialization behavior even in the presence of user-defined initializers.

See for example the tech note and the IO module spec.

[mppf]

IMO a reasonable alternative would be to serialize eltType seed and the current offset in the random stream (which we might call index or count). We can read this to reconstruct the state by starting with the seed and then having the RNG skip ahead to that offset.

I would say that the reason it doesn't just serialize the details as-is is that it's an attempt to avoid implementation-specific details. In the same way, using eltType + seed + offset would allow for the details of the RNG to change without making it impossible to read a file containing the RNG.

If random streams included an offset accessor method, this could even be implemented outside of the Random module entirely.

[lydia-duncan]

This serializer, called writeThis at the time, was added in r20924 merely to avoid a deadlock. [...]
Since randomStream is POD, the compiler-generated de/serializers will suffice, so removing the existing one is a good first step.

I feel like I'm missing something with this proposal. Has the implementation of randomStream changed sufficiently that the deadlock would no longer be a possibility? That seems plausible to me given the focus on the particular fields mentioned and the good work Jeremiah put in on this module, but it would have been nice to be explicit about that.

[vasslitvinov]

Has the implementation of randomStream changed sufficiently that the deadlock would no longer be a possibility?

Correct. When writeThis was introduced, randomStream (under a different name back then) contained a sync variable, which the default "writeThis" would try to read, causing an unintended synchronization operation. randomStream is no longer parallel safe, with the upshot of not containing any synchronization-related fields to worry about.

Michael, thank you for the good guidance. Even though we could write a de/serializer outside the module, we should do it within the module as we want this to be part of built-in functionality.