Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs explaining how to actually create a full combinator (ex. one to create/store a DB connection) #704

Closed
t3hmrman opened this issue Feb 26, 2017 · 33 comments

Comments

Projects
None yet
6 participants
@t3hmrman
Copy link

commented Feb 26, 2017

I've been stuck on this for a while and would appreciate some help/someone to enshrine how to do this axiomatically in Servant. All I'm trying to do is add a DB connection that goes along with the request. In trying to avoid global state, there are only a few places such a resource can go, namely:

  1. Vault (literally on the request)
  2. App Context (inside servant app machinery)

Making a warp middleware to do (1) was pretty easy and intuitive. Then came the problem of storing the lookup key to the vault (which seems to require (2) ). At this point I tried to do it axiomatically within servant, and do (2) outright -- creating a combinator that supplies the a "backend".

So far what I have is:

-- Servant combinator for including the database handler in subsequent requests                                                                                                                                                                                                   
data WithBackend a
    deriving Typeable

instance (HasContextEntry context a, Backend a, HasServer api context) => HasServer (WithBackend a :> api) context where
    type ServerT (WithBackend a :> api) m = ServerT api m

    route Proxy ctx subserver = route api ctx subserver
        where
          api = (Proxy :: Proxy api)

This example is a combinator that doesn't do anything (just uses the sub server) -- however, when I change the type ServerT to a -> ServerT api m, the types stop matching up and it's unclear what to do next. I could modify the context and use the modified context, but I was hoping to be able to make my handlers of type SqliteBackend -> Handler x for example, with this combinator being the thing to add the backend.

I haven't been able to find any explanation of how to do this, everyone handwaves it and says it would be easy, but I'm finding it very difficult. I found the documentation easy to follow but an explanation of HasServer and how to roll your own NON-AUTH combinators might make it perfect.

@t3hmrman

This comment has been minimized.

Copy link
Author

commented Feb 26, 2017

I think I figured it out:

-- Servant combinator for including the database handler in subsequent requests                                                                                                                                                                                                   
data WithBackend a
    deriving Typeable

instance (HasContextEntry context a, SQLBackend a, HasServer api context) => HasServer (WithBackend a :> api) context where
    type ServerT (WithBackend a :> api) m = a -> ServerT api m

    route :: Proxy (WithBackend a :> api)
          -> Context context
          -> Delayed env (Server (WithBackend a :> api))
          -> Router env
    route Proxy ctx delayed = route api ctx delayed'
        where
          api = (Proxy :: Proxy api)
          db = getContextEntry ctx :: a
          delayed' = SI.passToServer delayed (\_ -> db)

What helped me was looking at the source code for passToServer.

If I were to try and explain what's happening (maybe this will elucidate the servant beginner's mindset):

  • While I thought the last argument to route could be modified easily by just returning a partial function, it's infact a Delayed object

  • After looking at the delayed documentation/code in RoutingApplication.hs, I thought I needed to add a "Capture", so at first I tried to write something similar to addCapture

  • The types didn't make sense to me (why would I need to provide a function that has the captured thing and returns a route/delayed?) -- I already have the thing I want, I just want to pass it along to the server.

  • Looking at passToServer showed that serverD is what I needed to be trying to change. Types match what is expected, I can just ignore the request in the lambda and put in the thing I pulled from the context.

As far as improving the documentation/adding an example goes, maybe this code could be used, and a more principled explanation be given

@phadej

This comment has been minimized.

Copy link
Member

commented Feb 26, 2017

Have you read thru #700, i.e. why enter with Reader or partially applying the handlers isn't working for you?

Looks like a common problem, we should add this to the tutorial more explicitly.

@t3hmrman

This comment has been minimized.

Copy link
Author

commented Feb 26, 2017

Hey @phadej , thanks for the quick response. I actually didn't read about enter, I ran across enter while looking through the source, but wasn't sure what it was for/how to use it and the type annotation scared me away.

I see that I'm indeed using Contexts wrong -- what I was trying to do should have been done with by making a bigger monad stack, if I am understanding correctly. I will try to refactor my code to do so, I'm coming off a fresh read of RWH (Real World Haskell) but I'm not 100% comfortable with mtl just yet

@alpmestan

This comment has been minimized.

Copy link
Contributor

commented Feb 27, 2017

Feel free to keep using this ticket to ask for help if needed untim your problem is solved

We do plan on improving the tutorial, I want to add a section about custom combinators bit it turns out indeed that your issue can be solved without it and it would be nice to have DB connectiond as an example/motivation for enter.

@t3hmrman

This comment has been minimized.

Copy link
Author

commented Feb 28, 2017

hey @alpmestan , thanks -- I will.

I'm currently trying to wrap my head around how to use enter and will try to attach a commit to this issue to be scooped up/considered since the docs are in this repo.

I think it would be good (at least until the documentation changes in this area) to show a case study where soemone adds a db connection with context & and a custom combinator, and then explain how that's not the preferred approach, and creating a different monad stack, and using enter is preferred.

@phadej

This comment has been minimized.

Copy link
Member

commented Mar 1, 2017

I'm 👎 on showing how to do things wrong. I'm 👍 on showing how to do things right, explain alternatives (and what's wrong with them).

I'm not sure where to put my rule of thumb:

  • use enter to pass values to Handlers (or partial application)
  • use Context machinery to implement new combinators (it's a way to solve implicit configuration problem, see e.g. http://okmij.org/ftp/Haskell/tr-15-04.pdf for an alternative)
  • don't implement new combinators to pass values to handlers (which aren't originated from the request)
@lthms

This comment has been minimized.

Copy link

commented Mar 1, 2017

I actually like the idea of having a DB combinator for one reason: allow one to know wether a route can or cannot have an effect on the database. But one may argue that this is already the object of the POST request type…

@phadej

This comment has been minimized.

Copy link
Member

commented Mar 1, 2017

@lethom in this case you'd have DB combinator also on GET routes, otherwise you wouldn't be able to fetch anything from the DB.

@alpmestan

This comment has been minimized.

Copy link
Contributor

commented Mar 1, 2017

@lethom keep in mind that most things (if not everything) that are in the API type belong to a central description of the API that show say enough about it for anyone to interact with it in a way or another. The server interpretation checks that the provided handlers match what's expected for this API type, the various client interpretations use that description to generate clients that can interact well (by construction) with each individual endpoint, and so on. I'm not sure how a DB combinator would help there, as it's useless to everyone but the server. The breakdown by @phadej in #704 (comment) is most definitely what I would recommend to follow as well, it's the least painful path to getting things done, aka gradually reaching for fancier things as your needs grow, never when it's not required. This is of course just my opinion, but it has worked well for me and is the best approach as far as I can tell after a couple of years using servant.

@t3hmrman

This comment has been minimized.

Copy link
Author

commented Mar 2, 2017

Hey all, I found a new place to get stuck at... I scrapped the combinator approach in favor of trying to use enter and here is where I am:

app :: SqliteBackend -> Application
app b = serve api appWithDB
    where
      addDB = evalStateTLNat b :: StateT SqliteBackend a :~> Server a
      appWithDB = enter addDB server

A little explanation for why this code is as it is -- when following the servant tutorial the app was just of type Application. After using the context approach, I found it necessary to change the function to instead take the relevant DB, put it in the context, and return the resulting Application (so I could then use serveWithContext with some created context.

I though that point would be a good place to transform the underlying server (that should NOT be running in ExceptT but in StateT SqliteBackend in my case) into what I want. From reading the tutorial, it makes sense that what I need is a translation function from StateT SqliteBackend a to Handler a. The code at the beginning of this comment does unfortunately error

Few questions:

  • What's the difference between evalStateTLNat and evalStateTSNat?

  • Should I be using liftNat instead? evalStateTLNat seems like just a specialization of liftNat for StateT

  • How does one read a type signature like: Monad m => s -> (:~>) * (StateT s m) m?
    it looks like a function that takes a piece of the state you're trying to save, and produces a natural translation function which converts something of type StateT yourmonad anothermonad to anothermonad. If that interpretation is correct, then it's the right thing I need.

As I was writing this post, I realized the issue might be with the type of my server:

server :: Handler API
server = users

The users function was still of type Handler [User], which indeed doesn't make sense, if I'm doing the translations powered by enter in app down below. Assuming my app is running instead in a StateT SqliteBackend context. After cheating a little bit and removing the type annotation to find out what it should be, then changing it, I get:

server :: StateT SqliteBackend Handler [User]

This works, if I use just one endpoint of the actual app -- ironically, one that doesn't even use the backend. I ran into a problem when trying to add in the other commented-out branches of the API, but I'll try and think about that a little more on my own first before asking here.

@phadej

This comment has been minimized.

Copy link
Member

commented Mar 2, 2017

the

users :: StateT SqliteBackend Handler [User}

is correct. The idea of enter is to remove (or change) the StateT SqliteBackend Handler to just Handler
MyMonad a is StateT SqliteBackend Handler a in this case.

Note the state isn't preserved across the requests.

Then, the rest of your questions:

  • you have the wrong signature for addDB, the correct one is StateT SqliteBackend Handler :~> Handler.
  • liftNat is not the same as evalStateTLNat, liftNat doesn't take initial state argument (lift vs evalStateT)
  • the L and S in evalStateTLNat and evalStateTSNat stand for lazy and strict versions of StateT transformer
@t3hmrman

This comment has been minimized.

Copy link
Author

commented Mar 2, 2017

@phadej Thank you for the explanations & answers!

@t3hmrman

This comment has been minimized.

Copy link
Author

commented Mar 2, 2017

It looks like I have it working!

Here is (most of) the relevant code:

type API = "users" :> Get '[JSON] [User]
    :<|> "users" :> Capture "email" Email :> Get '[JSON] User

-- omitted the type sig here becuase it looks like I have to enumerate all the endpoints, can't just use "API" anymore, which is weird since the API matches, it's almost like the StateT SqliteBackend Handler doesn't distribute properly
server = users
         :<|> findUserByEmail

app :: SqliteBackend -> Application
app b = serve api appWithDB
    where
      addDB = evalStateTLNat b
      appWithDB = enter addDB server

findUserByEmail :: Email -> StateT SqliteBackend Handler User
findUserByEmail email = do
  b <- get

  user <- liftIO $ getUserByEmail b email
  case user of
    Nothing -> fail "Failed to find user"
    Just u -> return u

Thanks for your help! I'm still confident that I don't completely understand everything that's happening but I can at least start using this pattern for the rest of the endpoints

@krakrjak

This comment has been minimized.

Copy link

commented Mar 3, 2017

I've been working on variations of this for over a week and the biggest hint I got was from this bit of code or rather comment!

-- omitted the type sig here becuase it looks like I have to enumerate all the endpoints, can't just use "API" anymore, which is weird since the API matches, it's almost like the StateT SqliteBackend Handler doesn't distribute properly

You have saved me from having to commit seppuku. Nothing was making any damn sense at all with the type errors. My situation was less intense, I was just trying to wrap a dumb ReaderT onto my Handlers and it just wasn't working, for a week 👎

@t3hmrman

This comment has been minimized.

Copy link
Author

commented Mar 3, 2017

@krakrjak

This comment has been minimized.

Copy link

commented Mar 3, 2017

Well, fits and starts. I got a lot closer and now I'm not weeding through 4 pages of Type level gibberish to see the issues. I'm left with a rather mysterious problem that I think gets cleared up once I execute all the things from this post and a couple of other recent ones.

At this stage I have the handlers using the ReaderT type alias and things seemed to be looking up in the world when I ran into this final nail in my coffin.

  • Couldn't match type ‘Maybe Text -> ret’ with ‘Handler Text’
        arising from a functional dependency between:
          constraint ‘Servant.Utils.Enter.Enter
                        (Maybe Text -> ReadProgConf Text)
                        (ReaderT ProgramSettings Handler :~> Handler)
                        (Handler Text)’
            arising from a use of ‘enter’
          instance ‘Servant.Utils.Enter.Enter (a -> b) arg (a -> ret4)’
            at <no location info>

What am I still missing??? ProgramSettings is a record type and ReadProgConf is defined as
type ReadProgConf = ReaderT ProgramSettings Handler

Sincerely,
Frustrated in Monadville

@alpmestan

This comment has been minimized.

Copy link
Contributor

commented Mar 3, 2017

We need to see your code @krakrjak in order to be able to help. Especially the call to enter, the types of the things around that, etc.

@krakrjak

This comment has been minimized.

Copy link

commented Mar 3, 2017

Sure, here's an exemplary snippet. There's tons more routes and I know I'm just missing some little nugget of why this is wrong... I also know this is an auth example I'm giving, but the motivation is not for auth, these are just the most basic routes.

data ProgramSettings = ProgramSettings
                     { aField :: Text
                     , anotherField :: Text
                     }

type ReadProgConf = ReaderT ProgramSettings Handler
                     
data WithBackend a
    deriving Typeable

type WithSettings = WithBackend ProgramSettings

instance ( HasContextEntry context a, HasServer api context )
        => HasServer (WithBackend a :> api) context where
    type ServerT (WithBackend a :> api) m = a -> ServerT api m
    route :: Proxy (WithBackend a :> api) -> Context context
          -> Delayed env (Server (WithBackend a :> api))
          -> Router env
    route Proxy ctx delayed = route api ctx delayed'
      where
        api = Proxy :: Proxy api
        settings = getContextEntry ctx :: a
        delayed' = passToServer delayed $ const settings


type AnAPI = WithSettings :> "login"  :> Header "cookie" Text :> Get '[HTML] Text
        :<|> WithSettings :> "signup" :> Header "cookie" Text :> Get '[HTML] Text
        :<|> WithSettings :> "home"  :> Header "cookie" Text :> Get '[HTML] Text

anAPI :: Proxy AnAPI
anAPI = Proxy

app :: ProgramSettings -> Application
app s = serve anAPI appWithReader
  where
    addReader = runReaderTNat s
    appWithReader = enter addReader handlers

handlers = loginAPIHandler
     :<|> signupAPIHandler
     :<|> otherAPIHandler

loginAPIHandler :: ProgramSettings -> Maybe Text -> ReadProgConf Text
loginAPIHandler _        Nothing = liftIO renderLogin
loginAPIHandler settings cookie  = otherAPIHandler settings cookie

signupAPIHandler :: ProgramSettings -> Maybe Text -> ReadProgConf Text
signupAPIHandler _ _ = liftIO renderLogin

otherAPIHandler :: ProgramSettings -> Maybe Text -> ReadProgConf Text
otherAPIHandler s        Nothing       = loginAPIHandler s Nothing
otherAPIHandler settings (Just cookie) = case cookieToKey cookie of
      Nothing -> loginAPIHandler settings Nothing
      Just k  -> do
        s <- ask
        cs <- liftIO $ getActiveData s k
        liftIO $ renderPage s k cs

Edits: typo fixes and missing type alias.

@phadej

This comment has been minimized.

Copy link
Member

commented Mar 3, 2017

I don't understand, why you both define WithBackend combinator and use Reader

I removed the WithBackend and your code compiles:

{-# LANGUAGE DataKinds, TypeOperators #-}
import Control.Monad.Reader
import Data.Text (Text)
import Servant
import Servant.HTML.Lucid

data ProgramSettings = ProgramSettings
                     { aField :: Text
                     , anotherField :: Text
                     }

type ReadProgConf = ReaderT ProgramSettings Handler

type AnAPI = "login"  :> Header "cookie" Text :> Get '[HTML] Text
        :<|> "signup" :> Header "cookie" Text :> Get '[HTML] Text
        :<|> "home"  :> Header "cookie" Text :> Get '[HTML] Text

anAPI :: Proxy AnAPI
anAPI = Proxy

app :: ProgramSettings -> Application
app s = serve anAPI appWithReader
  where
    addReader = runReaderTNat s
    appWithReader = enter addReader handlers

handlers = loginAPIHandler
     :<|> signupAPIHandler
     :<|> otherAPIHandler

loginAPIHandler :: Maybe Text -> ReadProgConf Text
loginAPIHandler Nothing = liftIO renderLogin
loginAPIHandler cookie  = otherAPIHandler cookie

signupAPIHandler :: Maybe Text -> ReadProgConf Text
signupAPIHandler _ = liftIO renderLogin

otherAPIHandler :: Maybe Text -> ReadProgConf Text
otherAPIHandler Nothing       = loginAPIHandler Nothing
otherAPIHandler (Just cookie) = case cookieToKey cookie of
      Nothing -> loginAPIHandler Nothing
      Just k  -> do
        s <- ask
        cs <- liftIO $ getActiveData s k
        liftIO $ renderPage s k cs

-------------------------------------------------------------------------------
-- helpers
-------------------------------------------------------------------------------

data ActiveData
data Key

cookieToKey :: Text -> Maybe Key
cookieToKey = undefined

renderPage :: ProgramSettings ->  Key -> ActiveData -> IO Text
renderPage = undefined

renderLogin :: IO Text
renderLogin = undefined

getActiveData :: ProgramSettings -> Key -> IO ActiveData
getActiveData = undefined
@phadej

This comment has been minimized.

Copy link
Member

commented Mar 3, 2017

And a variant without using enter or custom combinators, enter just helps with little boilerplate here

{-# LANGUAGE DataKinds, TypeOperators #-}
import Control.Monad.Reader
import Data.Text (Text)
import Servant
import Servant.HTML.Lucid

data ProgramSettings = ProgramSettings
                     { aField :: Text
                     , anotherField :: Text
                     }

type AnAPI = "login"  :> Header "cookie" Text :> Get '[HTML] Text
        :<|> "signup" :> Header "cookie" Text :> Get '[HTML] Text
        :<|> "home"  :> Header "cookie" Text :> Get '[HTML] Text

anAPI :: Proxy AnAPI
anAPI = Proxy

app :: ProgramSettings -> Application
app s = serve anAPI (handlers s)

-- | Partially apply handlers with settings
handlers s = loginAPIHandler s
     :<|> signupAPIHandler s
     :<|> otherAPIHandler s

loginAPIHandler :: ProgramSettings -> Maybe Text -> Handler Text
loginAPIHandler s Nothing = liftIO renderLogin
loginAPIHandler s cookie  = otherAPIHandler s cookie 

signupAPIHandler :: ProgramSettings -> Maybe Text -> Handler Text
signupAPIHandler _ _ = liftIO renderLogin

otherAPIHandler :: ProgramSettings -> Maybe Text -> Handler Text
otherAPIHandler s Nothing       = loginAPIHandler s Nothing
otherAPIHandler s (Just cookie) = case cookieToKey cookie of
      Nothing -> loginAPIHandler s Nothing
      Just k  -> do
        cs <- liftIO $ getActiveData s k
        liftIO $ renderPage s k cs

-------------------------------------------------------------------------------
-- helpers
-------------------------------------------------------------------------------

data ActiveData
data Key

cookieToKey :: Text -> Maybe Key
cookieToKey = undefined

renderPage :: ProgramSettings ->  Key -> ActiveData -> IO Text
renderPage = undefined

renderLogin :: IO Text
renderLogin = undefined

getActiveData :: ProgramSettings -> Key -> IO ActiveData
getActiveData = undefined
@krakrjak

This comment has been minimized.

Copy link

commented Mar 3, 2017

I'm not sure why I added the WithBackend... Desperation I guess. in this smaller example everything works fine. When I extend it to the whole API it seems to fall apart... Doing the partial application and avoiding enter seems like the right thing to do here...

Does enter really serve a purpose? It's mentioned in the docs as a solution to this exact pattern, yet it seems to lead me down a rabbit hole.

One other thing... So by using the partial application method I side step the need for using the reader, what happens as this extends into more complicated bits? The intention in my exploration of this was to understand the general framework of dealing with the API as it evolves and grows with some routes needing more complexity than simply a ReaderT that I can prematurely unwrap.

@phadej

This comment has been minimized.

Copy link
Member

commented Mar 3, 2017

it works when all of the endpoints terminate with the same monad, it won't work if you have Handler and ReadProgConf in handlers.

@phadej

This comment has been minimized.

Copy link
Member

commented Mar 3, 2017

And to answer your question: enter helps with having e.g. LogT Handler, as you can runLogT on the top level, and not pass in every Handler.

@krakrjak

This comment has been minimized.

Copy link

commented Mar 3, 2017

Okay so all my endpoints end with one of Get or Post. The ReadProgConf type alias is gone. The handlers are now ProgramSettings -> ... -> Handler some are Text some are Int and a few are Response.

Looks like I can get the types to work correction without the ReaderT. I'm going to have some of that LogT and potentially many more transformers and I'd like to understand how that is supposed to work. Here it was a failed experiment on my part to get a ReaderT to work correctly with enter or without and it seemed that GHC could never quite figure out the ReaderT Foo Handler :~> Handler which seems to me like it should be a no brainer natural transformation (thus my incredulity at the difficulties).

I'm down to a few errors about content types now.... Those I think I can wrangle reasonably. I really want to hear some other experiences and see more code based around making the HasServer instances work and what is a possible way to move forward with a ReaderT or something more complicated would be nice too.

I did end up finding a niggling type mismatch in a handler through this exercise... I hope that wasn't causing too much confusion on my end. I'll try my experiments again over the weekend and see if I can report back something more definitive.

@phadej

This comment has been minimized.

Copy link
Member

commented Mar 3, 2017

I tried to summarise said comments in http://oleg.fi/gists/posts/2017-03-03-servant-and-db.html

@krakrjak

This comment has been minimized.

Copy link

commented Mar 4, 2017

This is a very good, concise write up of the situation and the solutions. Is there any way, all or some of, this write up could end up in the readthedocs docs for servant?

@phadej

This comment has been minimized.

Copy link
Member

commented Mar 4, 2017

@alpmestan what you think? I'm obviously biased :)

@alpmestan

This comment has been minimized.

Copy link
Contributor

commented Mar 4, 2017

@krakrjak @phadej This definitely should/could be in the servant docs, in my opinion. The only thing is: how/where?

Either we extend the "enter" section" with this material, which will make it pretty big, or we start some kind of "cookbook"-y section, after all the existing ones, where we can throw concrete examples of using one or more features from servant together. I'd go with the latter I think, what do you think folks?

@t3hmrman

This comment has been minimized.

Copy link
Author

commented Mar 4, 2017

@t3hmrman

This comment has been minimized.

Copy link
Author

commented Apr 5, 2018

Almost a year later -- the Cookbook section is there, and I think it's pretty clear how to do this now in Servant!

🍾

@t3hmrman t3hmrman closed this Apr 5, 2018

@alpmestan

This comment has been minimized.

Copy link
Contributor

commented Apr 5, 2018

Yes, I forgot to ping back all the issues that have been at least partially addressed by the cookbook, thanks for taking care of it for this one :-)

@t3hmrman

This comment has been minimized.

Copy link
Author

commented Apr 5, 2018

Thank you for the work on the library day in and day out! Servant's still a joy to use and I think singular in the scene, still haven't come across a library that hits the spot quite the same way :)

@chreekat

This comment has been minimized.

Copy link
Contributor

commented May 30, 2018

Just found this issue while searching for answers on how to adapt a Wai application to use Servant. @phadej's rules of thumb (#704 (comment)) were very enlightening. The Servant Haddocks make no mention of what Context is actually for, but now I would at least know what I would be looking for to learn more. :)

I might suggest that the Cookbook entries that demonstrate use of Context (such as BasicAuth) get moved into the tutorial itself. I also think that the title should reference Context directly. For instance, something like "Using Context: an example with BasicAuth". I hadn't read that section until now because I don't need or care about basic auth, but Context is quite useful indeed!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.