Docs explaining how to actually create a full combinator (ex. one to create/store a DB connection)

[t3hmrman]

I've been stuck on this for a while and would appreciate some help/someone to enshrine how to do this axiomatically in Servant. All I'm trying to do is add a DB connection that goes along with the request. In trying to avoid global state, there are only a few places such a resource can go, namely:

1. Vault (literally on the request)
2. App Context (inside servant app machinery)

Making a warp middleware to do (1) was pretty easy and intuitive. Then came the problem of storing the lookup key to the vault (which seems to require (2) ). At this point I tried to do it axiomatically within servant, and do (2) outright -- creating a combinator that supplies the a "backend".

So far what I have is:

-- Servant combinator for including the database handler in subsequent requests                                                                                                                                                                                                   
data WithBackend a
    deriving Typeable

instance (HasContextEntry context a, Backend a, HasServer api context) => HasServer (WithBackend a :> api) context where
    type ServerT (WithBackend a :> api) m = ServerT api m

    route Proxy ctx subserver = route api ctx subserver
        where
          api = (Proxy :: Proxy api)

This example is a combinator that doesn't do anything (just uses the sub server) -- however, when I change the type ServerT to a -> ServerT api m, the types stop matching up and it's unclear what to do next. I could modify the context and use the modified context, but I was hoping to be able to make my handlers of type SqliteBackend -> Handler x for example, with this combinator being the thing to add the backend.

I haven't been able to find any explanation of how to do this, everyone handwaves it and says it would be easy, but I'm finding it very difficult. I found the documentation easy to follow but an explanation of HasServer and how to roll your own NON-AUTH combinators might make it perfect.

[t3hmrman]

I think I figured it out:

-- Servant combinator for including the database handler in subsequent requests                                                                                                                                                                                                   
data WithBackend a
    deriving Typeable

instance (HasContextEntry context a, SQLBackend a, HasServer api context) => HasServer (WithBackend a :> api) context where
    type ServerT (WithBackend a :> api) m = a -> ServerT api m

    route :: Proxy (WithBackend a :> api)
          -> Context context
          -> Delayed env (Server (WithBackend a :> api))
          -> Router env
    route Proxy ctx delayed = route api ctx delayed'
        where
          api = (Proxy :: Proxy api)
          db = getContextEntry ctx :: a
          delayed' = SI.passToServer delayed (\_ -> db)

What helped me was looking at the source code for passToServer.

If I were to try and explain what's happening (maybe this will elucidate the servant beginner's mindset):

• While I thought the last argument to route could be modified easily by just returning a partial function, it's infact a Delayed object

• After looking at the delayed documentation/code in RoutingApplication.hs, I thought I needed to add a "Capture", so at first I tried to write something similar to addCapture

• The types didn't make sense to me (why would I need to provide a function that has the captured thing and returns a route/delayed?) -- I already have the thing I want, I just want to pass it along to the server.

• Looking at passToServer showed that serverD is what I needed to be trying to change. Types match what is expected, I can just ignore the request in the lambda and put in the thing I pulled from the context.

As far as improving the documentation/adding an example goes, maybe this code could be used, and a more principled explanation be given

[phadej]

Have you read thru #700, i.e. why enter with Reader or partially applying the handlers isn't working for you?

Looks like a common problem, we should add this to the tutorial more explicitly.

[t3hmrman]

Hey @phadej , thanks for the quick response. I actually didn't read about enter, I ran across enter while looking through the source, but wasn't sure what it was for/how to use it and the type annotation scared me away.

I see that I'm indeed using Contexts wrong -- what I was trying to do should have been done with by making a bigger monad stack, if I am understanding correctly. I will try to refactor my code to do so, I'm coming off a fresh read of RWH (Real World Haskell) but I'm not 100% comfortable with mtl just yet

[alpmestan]

Feel free to keep using this ticket to ask for help if needed untim your problem is solved

We do plan on improving the tutorial, I want to add a section about custom combinators bit it turns out indeed that your issue can be solved without it and it would be nice to have DB connectiond as an example/motivation for enter.

[t3hmrman]

hey @alpmestan , thanks -- I will.

I'm currently trying to wrap my head around how to use enter and will try to attach a commit to this issue to be scooped up/considered since the docs are in this repo.

I think it would be good (at least until the documentation changes in this area) to show a case study where soemone adds a db connection with context & and a custom combinator, and then explain how that's not the preferred approach, and creating a different monad stack, and using enter is preferred.

[phadej]

I'm 👎 on showing how to do things wrong. I'm 👍 on showing how to do things right, explain alternatives (and what's wrong with them).

I'm not sure where to put my rule of thumb:

• use enter to pass values to Handlers (or partial application)
• use Context machinery to implement new combinators (it's a way to solve implicit configuration problem, see e.g. http://okmij.org/ftp/Haskell/tr-15-04.pdf for an alternative)
• don't implement new combinators to pass values to handlers (which aren't originated from the request)

[lthms]

I actually like the idea of having a DB combinator for one reason: allow one to know wether a route can or cannot have an effect on the database. But one may argue that this is already the object of the POST request type…

[phadej]

@lethom in this case you'd have DB combinator also on GET routes, otherwise you wouldn't be able to fetch anything from the DB.

[alpmestan]

@lethom keep in mind that most things (if not everything) that are in the API type belong to a central description of the API that show say enough about it for anyone to interact with it in a way or another. The server interpretation checks that the provided handlers match what's expected for this API type, the various client interpretations use that description to generate clients that can interact well (by construction) with each individual endpoint, and so on. I'm not sure how a DB combinator would help there, as it's useless to everyone but the server. The breakdown by @phadej in #704 (comment) is most definitely what I would recommend to follow as well, it's the least painful path to getting things done, aka gradually reaching for fancier things as your needs grow, never when it's not required. This is of course just my opinion, but it has worked well for me and is the best approach as far as I can tell after a couple of years using servant.

[t3hmrman]

Hey all, I found a new place to get stuck at... I scrapped the combinator approach in favor of trying to use enter and here is where I am:

app :: SqliteBackend -> Application
app b = serve api appWithDB
    where
      addDB = evalStateTLNat b :: StateT SqliteBackend a :~> Server a
      appWithDB = enter addDB server

A little explanation for why this code is as it is -- when following the servant tutorial the app was just of type Application. After using the context approach, I found it necessary to change the function to instead take the relevant DB, put it in the context, and return the resulting Application (so I could then use serveWithContext with some created context.

I though that point would be a good place to transform the underlying server (that should NOT be running in ExceptT but in StateT SqliteBackend in my case) into what I want. From reading the tutorial, it makes sense that what I need is a translation function from StateT SqliteBackend a to Handler a. The code at the beginning of this comment does unfortunately error

Few questions:

• What's the difference between evalStateTLNat and evalStateTSNat?

• Should I be using liftNat instead? evalStateTLNat seems like just a specialization of liftNat for StateT

• How does one read a type signature like: Monad m => s -> (:~>) * (StateT s m) m?
  it looks like a function that takes a piece of the state you're trying to save, and produces a natural translation function which converts something of type StateT yourmonad anothermonad to anothermonad. If that interpretation is correct, then it's the right thing I need.

As I was writing this post, I realized the issue might be with the type of my server:

server :: Handler API
server = users

The users function was still of type Handler [User], which indeed doesn't make sense, if I'm doing the translations powered by enter in app down below. Assuming my app is running instead in a StateT SqliteBackend context. After cheating a little bit and removing the type annotation to find out what it should be, then changing it, I get:

server :: StateT SqliteBackend Handler [User]

This works, if I use just one endpoint of the actual app -- ironically, one that doesn't even use the backend. I ran into a problem when trying to add in the other commented-out branches of the API, but I'll try and think about that a little more on my own first before asking here.

[phadej]

the

users :: StateT SqliteBackend Handler [User}

is correct. The idea of enter is to remove (or change) the StateT SqliteBackend Handler to just Handler
MyMonad a is StateT SqliteBackend Handler a in this case.

Note the state isn't preserved across the requests.

Then, the rest of your questions:

• you have the wrong signature for addDB, the correct one is StateT SqliteBackend Handler :~> Handler.
• liftNat is not the same as evalStateTLNat, liftNat doesn't take initial state argument (lift vs evalStateT)
• the L and S in evalStateTLNat and evalStateTSNat stand for lazy and strict versions of StateT transformer

[t3hmrman]

@phadej Thank you for the explanations & answers!