Make packed substructs auto-serialize to []byte

[riannucci]

Even though slice-of-slice isn't reasonable to implement, a substructure marked as noindex could /totally/ be serialized as just a normal []byte. This would allow the common nested-structs use case trivially handled by adding noindex. Folks that want indexes for their deeply nested stuff will have to denormalize or something, but I'm not too worried about that.

[vadimsht]

You'd need to support both variants. Imaging adding "noindex" to existing struct. Requiring map reduce to convert entities is unreasonable in this case.

Consider adding new meta tag ",packed" (or something) instead. So the magic is visible to users.

[riannucci]

yeah ok, that sounds like a good idea. maybe packed implies noindex, but noindex doesn't imply packed?

[vadimsht]

Yes :) And a big red warning that adding\removing "pack" requires running map reduce to update entities (or something).

In worst case developers can do some stuff like

OldCrap S gae:"Crap,noindex"
Crap S gae:"PackageCrap,packed"

and then read from Crap, falling backto OldCrap if needed.

[riannucci]

e.g.

type Something struct {
  Nums    []int
  Strings []string
}

type Group struct {
  Things []Something `gae:",packed"`
}

Would normally fail, but now Group would essentially become a

type Group struct {
  Things []byte `gae:",noindex"`
}

Which means that Group would also itself be embeddable now.

edit: gofmt pedantics

[riannucci]

I could use gob to pack, probably?

[riannucci]

Using gob would also allow a bit of flexibility when adding new fields and such, or embedding pointers or whatnot.

[riannucci]

Actually, I could make removing pack transparent. Basically if I try to unpack []byte into a struct, use gob to do it, and if I have it defined as packed, but see struct data, I can use the existing substruct serialization code (which would still imply that nested slices are banned, but that's fine).

[vadimsht]

I've never used gob.

I've used encoding/binary, and it won't work if structs have slices (seems like you want to support this case).

So gob, yeah... Does it mean each struct type would have to be explicitly added to gob registry?

[riannucci]

gob registers all the basic types automatically, I think... it's only if you have an interface type that you need to register?

[riannucci]

Oh, or internal types maybe.

[riannucci]

"Only types that will be transferred as implementations of interface values need to be registered." https://golang.org/pkg/encoding/gob/#Register

[vadimsht]

It's like magic! ✨

[riannucci]

was thinking about this a bit more. I think we could use a syntax like

type Foo struct {
  Field []ComplexType `gae:",pack=gob|gzip"`
}

So it would be [name],pack[={gob,json,bytes}|{gzip|plain}]. For compatibility, "json|gzip" would be the default if the options aren't specified, but json|plain, gob|gzip all seem like reasonable choices too. In the case of bytes|gzip it would consume fields of type []byte or string and compress them.

I think for compatibility purposes, the first 4 bytes would be [version=1][reserved][compress_mode][pack_mode]. This would allow flexibility in the future, including schema upgrades and such, like switching from plain to gzip without hassle.