Remove type-guessing for scalars.

[exhuma]

Getting data from Perl into Python has one major challenge: The Scalar type
cannot be mapped 1:1 into Python. For exapmple a scalar can be an integer, a
float or even a string. The scalar deserializers in python-storable
(SC_SCALAR, SX_LSCALAR and SX_TIED_SCALAR) do not properly determine the
actual type. Instead the type is guessed. This is bound to cause bugs
down-stream. For example if a user of the library stored the string "23" in
Perl and deserialises in Python, the value will become the integer value 23!

Blindly reading the values as bytes is also impractical, and caused issues
with the existing unit-tests.

Personally, I have not enought experience with Perl (and the storable format)
yet to provide a fix with enough certainty of correctness, so any help is
greatly appreciated.

With agreement of the involved parties, I'll summarise the previous discussion
about this topic:

exhuma Feb. 22. 2017

I've contacted the Perl maintainers to get some information about the
"scalar" type. And the main issue stems from the fact that in Perl, numbers
are actually subclasses from strings. Which means that a scalar can either be
a string, an int or a float/double which makes decoding into Python
impossible without type guessing. The best one could do would be to return
the values as plain bytes. But that's kind of impractical.

mhart Feb. 23. 2017

Perl is, kinda, typeless. It's a very hard thing to work with when moving
data into a typed language like Python (or between Perl and JSON even...).
The scalar type is a C struct which can hold multiple value types and a set
of flags indicating which of those entries can be read to extract the current
value held. This means that if you read in the string value "123abc" and then
tried to use that in an integer context ($var + 0) you'd get the value 123
returned, but then the struct of your original var would also hold the
integer 123 in the struct. Magic... :|
http://perldoc.perl.org/perlguts.html#Working-with-SVs and
http://cpansearch.perl.org/src/RURBAN/illguts-0.49/index.html might help you,
I'm not sure.

Part of the type-guessing you're working on might want to decide which of the
flags to give precedence to. For example, if a number in the value-part of a
perl hash were used as a string at some point, when outputting the Python
dict equivalent would you output a string or int value? My preference is for
the number, but that's mostly because even when debugging Perl you can
accidentally cause the string value of a scalar to be created which would
affect the types you get. See the Perl JSON modules for example, or this kind
of question on SO

[...]

mhart 28. Feb. 2017

I think people make poor assumptions about types in Perl. If a value contains
something that looks like a number, it can be treated as such. If it is
treated as a number then it acts like one. That does not, however, make it a
number. It's just a scalar with some special flags. Working out how to best
use that is hard. The JSON encoders in Perl prove this, where numbers may
be rendered as strings because someone tried printing the variable earlier,
etc. :( Perl doesn't appear to support bigint by default, so expecting
b'112233445566778899' != 112233445566778899 to do "the right thing" won't
work either.

However... there may also have been some poor decoding in our storable.py
module where we aren't doing the right thing with the flags associated with a
scalar, you remember that perlguts page I referred you to earlier? I think we
might need to revisit that to determine what the type should be in Python
in case we've made a mistake somewhere here.

[CowboyTim]

This is one of the powerful features if perl IMO, together with the regular expressions mixed in the syntax and a very posix-like functional programming feeling and only a tiny bit of OO.

If you think as a perl dev, you don't really care, perl does the right thing and by extension, Storable does too. So, let's say I have a program that first does a few things, then saves a storable state, which is read in later on by e.g. a different process:

1. program A makes a storable

   my $abc = "10";
   my $def = 0.5;

When saved as a storable, the first variable is a string scalar, the second a double scalar.

2. program B reads the storable and calculates extra things

   my $diff =  $stor->{abc} * "1.1"- $stor->{def};

Perl transforms the $abc from a string scalar to a double and when storable saves this, it says it's SX_DOUBLE while we didn't even change the variables themselves, we only calculated a new variable from them.

From a perl programmer perspective, this works fine. It does what you want. Unless you didn't print the storable in net-mode, then it's a problem when endiannes or cpu architecture changes when e.g. program A and B don't run on the same machine. "net-mode" stringyfies the double in a readable always correct format.

In python, there are real types and the types used by storable mapped 1-1 on python's types in storable.py. This makes for a very easy solution that more or less always work if:

1. you're a decent perl programmmer
2. you both have the python and perl components under your control - this was btw also the initial reasoning on writing the python-storable class

The "decent perl programmer" argument is also next to impossible to be, as any cpan module can change this as perl does it on-the-fly in hidden places. It more means: if you're about to save a storable for a python class, make sure that what you're about to save is correct - from a perl->python conversion perspective.
Like:

  $stor->{abc} += 0.0;

This mostly boils down to numbers, as regular strings rarely can be upgraded to a number unless your entire string is e.g. accidentally 0-9a-f for hex, or only 0-9 for decimal.

My 2cents is to keep it a simple mapping as much as possible - as "guessed" by Storable from what perl says/thinks it is at the moment of making the storable.

[exhuma]

if you're about to save a storable for a python class, make sure that what you're about to save is correct - from a perl->python conversion perspective.

This is unfortunately not always possible. For example, right now, I am working on a tool to load storable data which was written by another program and not by myself (munin in this case).

There are certainly other cases where you don't have control over how the storable is written.

[CowboyTim]

Yes, for such cases I don't really have an answer.

For instance in JSON:

{ "abc" : "10", "def" : 0.5 }

Is possible and it means something different semantically then:

{"abc":10, "def" : "0.5" }

So maybe we should take a look on how this json is handled in python (and perl)? They both have good json modules, although the one for perl comes with quite some documentation on all the quirks too.

[exhuma]

I think the implementation in Python is straightforward, as the Python types map very well with the JSON types. It's Perl which adds a bit of complexity. From what I understand, even the Perl implementation for JSON parsing is nondeterministic. Don't quote me on that though, I have not verified/tested this.

Unfortunately my time is limited, which is the main reason I opened this issue. So that it's visible to the public that there's an outstanding issue. Once the release is done I have to move on to a different project (which has python-storable as dependency), and the scalar issue may or may not prove to be a problem. If it is, I may need to come back to this issue.

Given the difficulty of mapping a Perl scalar to Python it may happen that the most sensible course of action is to properly document this behaviour and let the end-users decide how to handle this.

There are other options:

• Implement a "scalar" class in Python which copies the behaviour of Perl.
• Return scalars as plain bytes (which is less interesting as it exposes/leaks internals beyond the abstraction that this library should provide).

[quasarj]

Implementing a "scalar" class in Python would be an interesting way to handle this. I'm not sure I would like it in the end, but it does seem like a good way to handle the uncertainty, and push the decision making onto the user of the module. I'm a fan of keeping guessing to a minimum.