Relationship with Opaleye?

[saurabhnanda]

Has Opaleye been open-sourced by Silk?

[hesselink]

No, Opaleye has been open sourced by @tomjaguarpaw. This is a repository with our layer on top of opaleye, which we open sourced so other people can use or copy it, and hopefully parts might be integrated in core opaleye at some point.

[saurabhnanda]

Can we rally some support (moral AND dev) for merging this code (or similar code which solves these problems) into opaleye? I'm not sure how can anyone use Opalaye in a serious production project without first having to rebuild this layer.

[hesselink]

I think @tomjaguarpaw used opaleye in production without this layer... I think the main reason it hasn't been merged is that it isn't clear that everyone would want to do everything this way. Also, this package does multiple things. Which ones in particular do you think are indispensable?

[saurabhnanda]

I think the main reason it hasn't been merged is that it isn't clear that everyone would want to do everything this way.

Which means that the lower-level API and the higher-level API shouldn't be tightly coupled (to allow people to build their own higher-level API if they have strong opinions). But, I strongly believe that a higher level API, which has the following parts working out-of-the-box should exist (either as a part of Opaleye itself, or a closely associated library):

• Updating records should be much easier
• Transactions
• A monad-transformer for DB operations
• Quick helper functions to query each table by the primary key (a very common use-case)
• Overridable functions for boilerplate stuff: convert between Haskell type to PG type, convert between read & write PG types
• Custom data-types for primary keys, i.e. UserId, TenantId etc
• SQL Logging for debugging purpose (is it already there?)

As I continue building out my domain API more, I'm sure I'll come across more common use-cases.

[tomjaguarpaw]

But, I strongly believe that a higher level API, which has the following parts working out-of-the-box should exist

It would be great if you could gather your ideas in one place.

(either as a part of Opaleye itself, or a closely associated library):

Perhaps you could write one :)

[saurabhnanda]

It would be great if you could gather your ideas in one place.

Want me to limit the noise to one place :)

Perhaps you could write one :)

I'm afraid I don't have enough Haskell chops to do that. However, between opalaye-sot, silk-opaleye, and opaleye-trans, don't we have everything we need?

If you can guide the project through a high level design doc, I will volunteer to write code, and docs/tutorials.

[tomjaguarpaw]

Want me to limit the noise to one place :)

I don't want anything to get lost.

If you can guide the project through a high level design doc, I will volunteer to write code, and docs/tutorials.

Sounds like it will be a great partnership :)

[hesselink]

Don't we already have everything we need then? Most of the features you describe exist in silk-opaleye, which is a library loosely coupled with opaleye itself :D

[hesselink]

More seriously: based on my experience with silk-opaleye and opaleye itself, I think that the use of native Haskell data types should be merged into opaleye itself. It brings a lot more type safety to writing your queries, which is one of the design goals of opaleye (no more mixing up user.id and post.id). As for the rest of your points:

• Updating records: we have an emptyUpdate which you can then update (hah) with the fields you want to change. Is that what you're thinking about?
• Transactions and a monad transformer: we have this, there is some design space here (carry a connection or a connection string, use a connection pool or not, retry failed queries or not) but I think our solution is reasonable, and there are other similar ones.
• Quick helpers for common queries: this goes in the direction of an ORM (or FRM in this case). Before opaleye we actually used a framework written by ourselves that tried to capture the relations in the database and generate queries based on that. It got quite hairy really quickly, so I'd be hesitant to have a 'blessed' opaleye library like this until one has been written and tested in the wild.
• SQL logging: there's a function in opaleye to print the sql to a string, which we've used. You can also use this to prepend other things (like ANALYZE) but there's nothing out of the box for that. I think an analyze function might be useful, but again there are options, since ANALYZE has a lot of knobs, and there's also the question of the output format.

[saurabhnanda]

Don't we already have everything we need then? Most of the features you describe exist in silk-opaleye, which is a library loosely coupled with opaleye itself :D

Where do you think I would've copied all the code from to submit to @tomjaguarpaw ;)

Updating records: we have an emptyUpdate which you can then update (hah) with the fields you want to change. Is that what you're thinking about?

I feel Opaleye should support both workflows - barebones type-safe updates (current) and FRM/RRM based (read a record from DB, update a field, and write it back)

With respect to logging, how do I put Opaleye in verbose mode and view all the SQL queries it's firing?

[tomjaguarpaw]

how do I put Opaleye in verbose mode and view all the SQL queries it's firing?

There is no verbose mode but you can see the queries that it sends with the functions in https://github.com/tomjaguarpaw/haskell-opaleye/blob/master/src/Opaleye/Sql.hs and https://github.com/tomjaguarpaw/haskell-opaleye/blob/master/src/Opaleye/Manipulation.hs

[hesselink]

Alternatively you can configure postgres to log all the queries that it's performing. You can use label together with ghc's call stacks to find out where the queries are coming from. We do this in our runner functions.

[saurabhnanda]

Isn't it easier to have you app-server log SQL queries along with other important things, like incoming request, handler invoked, tenant and user ID, time taken to respond, etc.

Isn't depending solely on PG logging an inferior programmer UX?

[saurabhnanda]

@hesselink you're accessing the call stack via an implicit param. Do you need to compile with profiling turned on? Performance impact?

[hesselink]

Yes, it might be easier to log in the app server. It should be easy to build on top of the sql printing function and the normal run function. Then you can also decide on your own logging library/method and format instead of opaleye choosing one for you (or having some complicated pluggable system).

For the call stacks you don't need profiling turned on. That also means that you only get the part of the call stack that you explicitly annotate with the implicit parameters. But that's ok, we mostly want the immediate calling function anyway. The other option is to label all your queries manually, I've done that as well. I've even labelled subparts of very large queries. I haven't noticed any performance impact of the call stacks, and I'd suspect it will be dwarfed by most normal computations, let alone running a database query.

[saurabhnanda]

Then you can also decide on your own logging library/method and format instead of opaleye choosing one for you (or having some complicated pluggable system).

Would a pluggable system really be that complicated? If we have a ReaderT for the DB connection, can't we just add the logging function to it as well?

[saurabhnanda]

If you can guide the project through a high level design doc, I will volunteer to write code, and docs/tutorials.

Sounds like it will be a great partnership :)

@tomjaguarpaw are you on-board with extending the Opaleye API to make common tasks easier?

[tomjaguarpaw]

How can I say "no"?!

[saurabhnanda]

How can I say no?!

You can, if you want to. You're the project-dictator :)

[saurabhnanda]

@tomjaguarpaw any update on this? Also I found https://github.com/byteally/dbrecord-opaleye which seems to be solving similar problems.

[tomjaguarpaw]

Well, I'm open to improving Opaleye, yes :)

[saurabhnanda]

Which direction would you like to take Opaleye in?

[tomjaguarpaw]

That's an extremely broad question! Suffice it to say I am happy to add functionality to Opaleye that makes it more convenient as long as it doesn't compromise the general principles (type safety, composability). Beyond that, it's completely up to other people what they want to see in the library. Suggestions are welcome. PRs are even more welcome.

[saurabhnanda]

Let me collate all the ideas in the Opaleye repo itself. I'll close this issue shortly.

[saurabhnanda]

@tomjaguarpaw did you get my ping via https://gist.github.com/saurabhnanda/c2dc9526f9f99e6c10f92120326ee8a6 -- awaiting your blessings.

[saurabhnanda]

@hesselink I believe this project is not on Hackage, right? Is there an easy way to generate the Hackage docs for the public API and host them somewhere (even it's just type signatures)?

[bergmark]

There's a RC here https://hackage.haskell.org/package/girella-1.0.0/candidate but it seems to be missing docs

[saurabhnanda]

What's girella? Public name for Silk-Opaleye, I'm guessing.

[bergmark]

same package, new name

[saurabhnanda]

@bergmark In that case, how easy is it to generate docs for girella? I'm not well versed with Hackage, yet.

[hesselink]

I don't think hackage builds docs for candidates. You can build them locally of course with cabal haddock or stack haddock, and I believe both can now also upload docs to hackage in case that's needed.

[saurabhnanda]

@hesselink @bergmark unable to fetch girella via stack. Using lts-6.12 and added the following line to stack.yaml:

extra-deps: [opaleye-0.5.1.1, girella-1.0.0]

[bergmark]

To use a candidate from stack you need to use the url to the tar.gz, but it's probably easier to use this repo instead. Also remember to use the silk branch on our opaleye fork.

[saurabhnanda]

To use a candidate from stack you need to use the url to the tar.gz, but it's probably easier to use this repo instead. Also remember to use the silk branch on our opaleye fork.

Possible to share the exact lines to add to stack?

[saurabhnanda]

So, https://github.com/silkapp/haskell-opaleye/tree/silk and https://github.com/silkapp/silk-opaleye/tree/girella go together? What's been changed in the opalaye in that branch?

[hesselink]

Our opaleye forks is opaleye 0.3 with some cherry-picked changes. I think everything we did has been contributed back to opaleye. It's a very unfortunate thing that we have this fork. The reason is the change in update behavior in tomjaguarpaw/haskell-opaleye#92.