Consider more efficient representation for HKD

[axman6]

I've watched the higgledy from scratch videos, which were fantastic. The first thing that jumped out at me is that in memory representation of HKD Foo is going to be quite in efficient if one would like to do anything other than immediately convert back to a Foo; lots of pointer chasing to get to each field, and the more fields a structure has the worse this would get. I wondered how difficult having HKD(_) internally produce something similar to SuperRecord's FromNative version of Foo, where accessing each field would become a constant time operation, so performance should ideally approach the hand written HKD version of Foo.

I would personally simplify things a little over what SuperRecord does where it sorts the fields, this seems unnecessary.

I'm mostly opening this issue for discussion of the pros and cons, I'm not tied to the exact choice of library (SuperRecord is the only one I have experience with of the anonymous record libraries, and it has excellent performance).

[i-am-tom]

This is interesting! I'm pretty busy this weekend, but next week I'll take a look at some core - I've always (optimistically) assumed that GHC has some generics-specific specialising that takes place, but maybe that was naïve.

I'd also be keen to hear more about your use case - I've also assumed that the O(log n) access times for a generic rep would always be fast enough, as I thought the compiler requires that n < 62. If your application involves a lot of read/writes, though, I can imagine this might start to build up! Maybe there are some other things we can do to fuse operations?

Thanks for opening up this discussion!

[axman6]

Yeah as far as I know unless you're directly producing something Rep like and then consuming it GHC should go a good job deforesting the intermediate structure, but if you expect to work with something of type HKD f a for more than that then the tree representation would have to exist in memory, which would probably be a bad thing. Particularly for memory usage, there's a hell of a lot of pointer overhead with such a nestled representation (we actually have quite a few data types at work which have dozens of fields, so if we were to use this that would affect us a lot).

We don't currently have a use case for Higgledy, I'm just really interested in the technique and could probably put it to good use in some other projects down the line.

I'll see if I can find some time on the weekend to have a play with it.