Servers and clients need a way to skip flattening/unflattening if not needed

[maffoo]

For servers that just want to pass data along or store it in a lossless way, it would be nice to be able to avoid incurring the cost of flattening/unflattening data.

[pomalley]

Dare I suggest: LazyList?

[maffoo]

LazyList was a half-hearted attempt to do something like this, but it only skipped unflattening lists. I'm thinking of something more drastic here, more inline with how the delphi and java apis work, where you really just have an opaque Data object that is essentially just raw bytes.

[ejeffrey]

Why not just make flattening and unflattening cheap enough that it doesn't
matter? Austin's stuff should help with that.

@pomalley: very funny.

I think the right way to do this is to make it explicit. Make a variant of
the @setting decorator that allows requesting some or all of the arguments
to explicitly stay flattened, wrapped up in an opaque "LabradData" type.
Then the server code can either manually extract the data out of it, or
just pass it on to an outbound call.

On Wed, Feb 11, 2015 at 7:10 PM, Matthew Neeley notifications@github.com
wrote:

For servers that just want to pass data along or store it in a lossless
way, it would be nice to be able to avoid incurring the cost of
flattening/unflattening data.

—
Reply to this email directly or view it on GitHub
#65.

[maffoo]

Even if unflattening is cheap, what are you unflattening to? You need a lossless representation of the labrad data, so why not just operate on the bytes directly. Also, as mentioned in #66, it would be nice to uncouple the python representation so the user can unflatten in different ways.

Agree that explicit is better than implicit, and I was thinking along the same lines of modifying the setting decorator. Similarly, could have some kind of flag for clients who make a request and want the result to not get unflattened.

[ejeffrey]

Well, we want a low overhead way to do the flatten/unflatten paths in any
case, as that is the 'normal' code path. If we can make that fast enough
that we aren't limited by it, and if we can make sure that
unflatten+flatten is lossless, then I don't see the point in providing an
alternate code path.

On Wed, Feb 11, 2015 at 7:22 PM, Matthew Neeley notifications@github.com
wrote:

Even if unflattening is cheap, what are you unflattening to? You need a
lossless representation of the labrad data, so why not just operate on the
bytes directly. Also, as mentioned in #66
#66, it would be nice
to uncouple the python representation so the user can unflatten in
different ways.

Agree that explicit is better than implicit, and I was thinking along the
same lines of modifying the setting decorator. Similarly, could have some
kind of flag for clients who make a request and want the result to not get
unflattened.

—
Reply to this email directly or view it on GitHub
#65 (comment)
.

[DanielSank]

You need a lossless representation of the labrad data, so why not just operate on the bytes directly

While this will solve the immediate problem with the datavault and will also help in the case where a server just wants to forward a request or response elsewhere, I think we may be conflating two ideas here. Independently of whether or not we keep the underlying binary data, I'd think we want to guarantee that flatten(unflatten(binary_data)) == binary_data, i.e. promise Losslessness^TM.

Keeping the binary string available seems like a totally independent consideration (although it does provide Losslessness^TM by construction). It's independent because users aren't going to want binary data, they want python objects, so there has to be a lossless binary->python and python->binary mapping anyway.

[ejeffrey]

If we do this, I propose only supporting it in the asynchronous
(server)client. That should cover 90% of the use cases, and we can then
build it into the packet data structure:

This is what I propose as a syntax for this:

@setting(10, _raw=True, accepts=['_s', '_v[GHz]'], returns=['_i_v[]])
def do_stuff_proxy(self, c, data):
p = remote_server.packet()
p.do_stuff(data, _raw=True)
result = yield p
returnValue(p['do_stuff'])

I haven't looked in pylabrad enough recently to know how ugly this is. It
definitely crosses all the code in pylabrad, if I recall correctly the
unflattering of record data happens already in the packet dissector.

On Wed, Feb 11, 2015 at 10:25 PM, Daniel Sank notifications@github.com
wrote:

You need a lossless representation of the labrad data, so why not just
operate on the bytes directly

While this will solve the immediate problem with the datavault and will
also help in the case where a server just wants to forward a request or
response elsewhere, I think we may be conflating two ideas here.
Independently of whether or not we keep the underlying binary data, I'd
think we want to guarantee that flatten(unflatten(binary_data)) ==
binary_data, i.e. promise LosslessnessTM.

Keeping the binary string available seems like a totally independent
consideration (although it does provide LosslessnessTM by construction).
It's independent because users aren't going to want binary data, they want
python objects, so there has to be a lossless binary->python and
python->binary mapping anyway.

—
Reply to this email directly or view it on GitHub
#65 (comment)
.

[DanielSank]

Why the underscore in _raw?

[maffoo]

@ejeffrey why only for the asynchronous interfaces and what do you mean by "build it into the packet data structure"?

[ejeffrey]

Well, currently the async and syncronous interfaces are largely separate
(they use different ServerWrapper and PacketWrapper classes). So I think
we should do the async version first at least -- that is where the majority
of the use cases are, and that interface is more featureful.

I just mean that we can require using the explicit packet syntax to request
non-flattening of return values, rather than allowing it in the direct call
syntax. Actually, it could be requested when you construct the packet
object if we don't need per-record control over flattening.

I used _raw to prevent confusion with argument names, but I actually think
that isn't necessary. In the setting decorator we can tell by context, or
we can just have a different decorator. If we put the flag for client
calls into packet() it is explicitly separate from arguments.

Evan

On Thu, Feb 12, 2015 at 8:35 AM, Matthew Neeley notifications@github.com
wrote:

@ejeffrey https://github.com/ejeffrey why only for the asynchronous
interfaces and what do you mean by "build it into the packet data
structure"?

—
Reply to this email directly or view it on GitHub
#65 (comment)
.

[maffoo]

Ah, makes sense. When you said "packet data structure" I thought you were referring to bytes on the wire.