Feature request: extend Haxl to support streaming throughout the tree

[gleber]

If Haxl is implemented to support streaming throughout the tree, from each leaf up to the root. With this feature, Haxl will (?) be usable as a framework to perform operations on a data which does not necessary fit into the memory.

[simonmar]

I have no idea what streaming would look like. That doesn't necessarily mean it's a bad idea, just that it's not at all clear (to me) what the design is. Maybe you have something in mind?

[evincarofautumn]

Seems like the biggest challenge would be reconciling streaming with caching. Either streaming fetches should not be cached, in which case you only benefit from concurrency, or Haxl needs to somehow provide the guarantee that anything not yet seen by all readers be cached. If the data doesn’t necessarily fit into memory, you never want to cache all of it, but that would seem too easy to do accidentally:

do
  (sx, sy) <- liftA2 (,) fetchThings fetchThings
  x <- consume sx
  -- Here, the entire contents of the shared stream are in
  -- the cache because sy hasn’t been consumed.
  y <- consume sy
  return (x + y)
where
fetchThings :: Haxl (Stream Thing)
consume :: Stream Thing -> Haxl Int

[simonmar]

I don't think it's useful to keep this issue open given that we don't have a concrete plan or design, and there's been no discussion for months.