serialisation - add FREEZE and THAW support

[shawnlaffan]

Serialisation using Sereal, and presumably other such systems, results in fatal errors. Reproducer below, along with its output.

Fixing this should (one hopes) just be a case of adding FREEZE and THAW methods, as documented at https://metacpan.org/pod/Sereal::Encoder#FREEZE%2FTHAW-CALLBACK-MECHANISM

Similar code has already been added for Storable via PDL::IO::Storable so it should be a case of reusing that.

I'll run some testing and if it works then I'll raise a PR.

#  reproducer
use strict;
use warnings;
use 5.014;
use PDL;
use Sereal qw/decode_sereal encode_sereal/;
my $x = decode_sereal encode_sereal (xvals(25,25));

say 'Result:';
say '=====';
say $x;
say '=====';
say 'Script completed';

Result:
=====
Stringizing problem: Fatal error: argument is probably not an ndarray, or magic no overwritten. You're in trouble, guv: 2453f26e5a8 245412ebf40 1111638597
 at C:/perls/5.40.0.1_PDL/perl/vendor/lib/PDL/Core.pm line 3294.
        PDL::string(PDL=SCALAR(0x2453f26e5a8), undef, "") called at serialise.pl line 9
        (in cleanup) Fatal error: argument is probably not an ndarray, or magic no overwritten. You're in trouble, guv: 2453f26e5a8 245412ebf40 1111638597

[mohawk2]

Happy New Year! Did the testing work?

[shawnlaffan]

The initial checks did not work. I'll get back to it soonish.

[shawnlaffan]

I think I have it working now.

Work-in-progress code is at https://github.com/shawnlaffan/pdl/tree/freeze_thaw

It needs a cleanup before any PR is raised.

[shawnlaffan]

I've now also checked the approach works with CBOR::XS.

However, it fails with JSON::MaybeXS and presumably other JSON serialisers that support FREEZE and THAW methods.

The issue is that the data is default encoded to unicode (see Cpanel::JSON::XS docs). This leads to incorrect sizes or simply incorrect data, depending on the ndarray.

While one could decode the data from unicode before unpacking it into the ndarray, the encoding permutations available to JSON serialisers likely make this a somewhat fraught exercise.

An alternative is to freeze JSON data using PDL::IO::Dumper. The issue is that it uses a string eval to deserialise data.

Suggestions welcome.

[mohawk2]

Thank you for your efforts on this! Is it possible to find (and then document) some combination of settings for the two not-yet-working encoder/decoder so it does work?

As an alternative, would it work to just stringify (with a local $PDL::toolongtoprint = $largenumber) then decode with PDL->pdl? We're after making it work, not necessarily super elegance! Slightly better might be to instead encode the top level as a two-element array, first is the type name, second either an unpdl-ed array, or the stringified version.

[shawnlaffan]

Thanks for the idea.

The Types::Serialiser protocal works with arrays of data, which means storing the type with unpdl'd data works for non-complex types (people can see the relevant commit if interested).

Complex data have substructures which either also need freeze/thaw methods or to be processed in some way after unpdl is called.

perl -MData::Dump=dump -MPDL -E"my $x = cdouble 5..6; my $u = $x->unpdl; say dump $u;"
[
  bless({
    c_dirty        => 0,
    cartesian      => [5, 0],
    display_format => { polar_pretty_print => 1, style => "cartesian" },
    p_dirty        => 1,
  }, "PDL::Complex::Overloads"),
  bless({
    c_dirty        => 0,
    cartesian      => [6, 0],
    display_format => { polar_pretty_print => 1, style => "cartesian" },
    p_dirty        => 1,
  }, "PDL::Complex::Overloads"),
]

[shawnlaffan]

Complex data have substructures which either also need freeze/thaw methods or to be processed in some way after unpdl is called.

Or is it worth extending unpdl to work with complex types?

[shawnlaffan]

Complex data have substructures which either also need freeze/thaw methods or to be processed in some way after unpdl is called.

Or is it worth extending unpdl to work with complex types?

Some experimentation shows it already handles them. The issue seems to be that the JSON serialiser looks for FREEZE/THAW methods on any object it sees.

[shawnlaffan]

Adding FREEZE and THAW methods to PDL::Complex::Overloads seems to work.

See commits for code and tests.