Change definition of StreamGenerator

[jvanbruegge]

Currently it is not possible to write an instance for StreamGenerator for conduit, despite the comment saying otherwise.
Furthermore you cannot use Streams that use a ReaderT IO stack.
This PR trys to solve that
DISCLAIMER: I have to idea what I'm doing

[jvanbruegge]

I don't think the build failure is caused by my code

[gbaz]

Changing the class to use a fundep makes sense, as you explained. But, I don't see what the move to a StreamGenerator parameterized over MonadUnliftIO buys you? Especially since the usage you make of it seems to all be in the case where it is specialized back to IO anyway?

[jvanbruegge]

The instance I have there currently is just for easy usage with IO. My issue is because my App base monad is a ReaderT IO and I need the monadic context for my response. With MonadUnliftIO I can do that

[gbaz]

I confess I don't understand why ReaderT IO can't be handled with the IO version directly.

[jvanbruegge]

Because the return value of the function is IO (getStreamGenerator :: a -> IO ()) -> (a -> IO()) -> IO ())
If I try to define an instance for to StreamGenerator for Conduit (MonadUnliftIO implies MonadIO):

instance MonadUnliftIO m => ToStreamGenerator (ConduitT () o m ()) o where
     toStreamGenerator s = StreamGenerator $ \a b -> runConduit $ s .| op a b
        where op :: (o -> IO ()) -> (o -> IO ()) -> ConduitT o Void m ()
              op = undefined

runConduit returns m () not IO (). Thus the function is not assignable.
Maybe I am missing something here

[jvanbruegge]

Yeah, UnliftIO is not needed when you let the functions return plain IO
But this actually leads to compiler errors in the code using the class:

Couldn't match type ‘m’ with ‘m1’
      ‘m’ is a rigid type variable bound by
        the instance declaration at src/MediaGoggler/Raw.hs:20:10-74
      ‘m1’ is a rigid type variable bound by
        a type expected by the context:
          forall (m1 :: * -> *).
          MonadIO m1 =>
          (o -> IO ()) -> (o -> IO ()) -> m1 ()
        at src/MediaGoggler/Raw.hs:21:27-79

This means the ToStreamGenerator class would have to pass the monad too

[gbaz]

Right. You won't be able to write the instance in general. You can only write it for specific m. So you'll need one for ConduitT over IO, etc.

I suspect that what you've written doesn't do how you expect. The only conduits you will be able to use, because of the universal quantification, are things that work over any m that has the typeclass. This isn't going to be what you want :-)

(i.e. what you ran into above).

[gbaz]

Oh and I see the Reader issue now. Yeah, you won't be able to pass in a conduit over ReaderT IO. There's no place to thread through the reader value to use. You'll need to convert that to a conduit over IO directly. You can do that in your ambient monad using the toIO function, I guess.

[jvanbruegge]

I see, thank you. I will try that

[alpmestan]

Regarding CI, yes the build fails with the oldest GHC that we test on but that's been the case for some time, it's not because of your patch. We will soon slide the window, getting rid of that one and adding 8.4.

[jvanbruegge]

I squashed my commits as I was now able to write a working conduit instance that streamed a video over a normal servant StreamGet endpoint:

instance ToStreamGenerator (ConduitT () o (ResourceT IO) ()) o where
    toStreamGenerator s = StreamGenerator $ \a b -> runConduitRes $ s .| op a b
        where op :: (o -> IO ()) -> (o -> IO ()) -> ConduitT o Void (ResourceT IO) ()
              op start rest = do
                takeC 1 .| mapM_C (liftIO . start)
                mapM_C $ liftIO . rest

Code using the instance:

type FileStream = ConduitT () ByteString (ResourceT IO) ()

serveFile :: Id -> Server (StreamGet NoFraming OggVideo FileStream)
serveFile id = except404 (getFilePath id) >>= pure . Conduit.sourceFile

getFilePath :: (DB.MonadDB m) => Id -> m (Either Text FilePath)
getFilePath id = DB.getVideoFile id >>= \case
    Left e -> pure $ Left e
    Right (DBEntry _ VideoFile{ path }) -> pure . Right . toFilePath . prefixPath $ path

[jvanbruegge]

And CI passes except for the oldest GHC 🎉

[jvanbruegge]

Streaming files requires a basic NoFraming strategy. Should I add it to this PR? I think it is useful in general for raw data.

[alpmestan]

@jvanbruegge Yes, please do feel free to add it. In fact, in previous discussions I came to realize that for many many common streaming scenarios, this is the "framing strategy" we would need. So this PR looks to me like the perfect excuse for adding it. :-)

[jvanbruegge]

Done, though I am not entirely sure about FramingUnrender. Test do pass.

[alpmestan]

@gbaz How does this PR look to you now?

[jvanbruegge]

I noticed another problem when trying to implement Range Requests over a streaming endpoint. The stream data type does not allow to specify a result code, it is always 200: https://github.com/jvanbruegge/servant/blob/db22df1236c7b02035a1ef84ba4c5ae4523864c2/servant-server/src/Servant/Server/Internal.hs#L326
Should I try to change that in this PR too, or rather open a new one?

[phadej]

I managed to fix master CI, @jvanbruegge could you rebase on top of it?

---

I'd prefer things which can be separate in separate PRs.

[gbaz]

This looks good to me, and what I'd expect. Glad you're doing this!

(Also, the response code stuff should def be in a different PR)

[jvanbruegge]

Rebased

[jvanbruegge]

And CI is green 🎉