Add support of PostgreSQL-specific `VALUES(..)` expression #284

NikitaRazmakhnin · 2021-09-22T10:44:57Z

At my company we really wanted to use VALUES Postgres expression to use it inside of JOIN
between large db data-set with some smaller user-input data-set without loading of data into app-memory.
VALUES allows us to do that and even gives performance benefit to our app.

I think it is a relevant issue #231 :)

This PR tries to bring an ability to use like:

SQL : SELECT * FROM (VALUES (a1), (a2) ...) as "vs"("v1")
Haskell: select $ from $ values [val 1, val 2, val 3]

It also works for more than one columns as well (a bit doubting about aliases/refs assembling but works good so far). :)

I implemented it as part of From and thanks to latest refactoring from
GADTs to type-classes it was not so difficult :)

Few appropriate semi-integrational tests with other sql capabilities as join, sinigle/multicolumn were added as well.

Please, let me know if I made something in a wrong way - I am ready to try to fix :)

Thanks!

Before submitting your PR, check that you've:

Bumped the version number.
Documented new APIs with Haddock markup.
Added @since declarations to the Haddock.
Ran stylish-haskell and otherwise adhered to the style guide.

After submitting your PR:

Update the Changelog.md file with a link to your PR.
Check that CI passes (or if it fails, for reasons unrelated to your change, like CI timeouts).

scalar expression of values-list for `from` targets.

parsonsmatt

This is awesome, thank you!

Only thing is that I am pretty sure we can put this directly into esqueleto proper instead of Esqueleto.Postgresql. Can you try doing that and see if the tests all work out still?

parsonsmatt · 2021-09-22T13:38:11Z

src/Database/Esqueleto/PostgreSQL.hs

@@ -363,3 +371,75 @@ filterWhere aggExpr clauseExpr = ERaw noMeta $ \_ info ->
    in ( aggBuilder <> " FILTER (WHERE " <> clauseBuilder <> ")"
       , aggValues <> clauseValues
       )
+
+newtype PgValuesExprs a = PgValuesExprs { unPgValuesExprs :: NE.NonEmpty a }


Is this a Postgres specific feature?

This article seems to indicate that it is a SQL-92 feature, and should be supported by all SQL databases.

MySQL supports VALUES, as does SQLite. So I think we can move this into esqueleto proper.

Ah sorry: my bad! Thanks for the links.

Sure, I can do that, but looks like other vendors requires different syntax, so I need to address that then :)

I will try to do that.

Oh, if they require different syntax, then let's not bother. The current solution is to keep anything that varies in these separate modules.

In some basic detail:

MySql wants keyword ROW(..) for one row of values;

SqLite requires some other placement of commas as far as I understand.. but worth to play a bit more.

I think consistency of API for vendors matters, so I am ready to try to cover other vendors as well.
But if you allow to go Postgres API earlier than other vendors - it would be awesome!

Otherwise - I could add necessary extensions for other vendors right here and ping you when it's ready :)

esqueleto doesn't currently do anything to differentiate the database provider - it's all Sqlbackend as far as we're concerned. So I'd rather not try and do anything clever here and just support PG directly. If someone else wants MySQL support or Sqlite, then we can add those in vendor-specific modules, and if we really want to unify it, we can try and do that.

Yep, understood: I also meant similar extensions in vendor-specific modules (like here) without unification (which probably also will present some code duplication but not a big deal I think).

Then I guess from Postgres perspective everything looks ok now? :)

parsonsmatt · 2021-09-22T13:41:18Z

src/Database/Esqueleto/PostgreSQL.hs

+--  pure (bound, count @Int $ user^.UserName)
+-- @
+--
+-- @since 3.5.2.3


🙌🏻 this is dope

parsonsmatt · 2021-09-22T13:41:31Z

test/PostgreSQL/Test.hs

@@ -1270,6 +1271,53 @@ testLateralQuery = do
            let _ = res :: [(Entity Lord, Value (Maybe Int))]
            asserting noExceptions

+testValuesExpression :: SpecDb
+testValuesExpression = do


blessed tests ❤️

parsonsmatt · 2021-09-22T14:11:11Z

Try merging master once #285 is pulled in

belevy · 2021-09-26T16:53:53Z

src/Database/Esqueleto/PostgreSQL.hs

+instance (ToSomeValues a, Ex.ToAliasReference a, Ex.ToAlias a) => Ex.ToFrom (PgValuesExprs a) a where
+  toFrom = fromValues . unPgValuesExprs


Is there a reason why you are going through ToFrom instead of just just making values === fromValues?

ToFrom is mostly there to support SqlSetOperation as a restricted subset and backwards compatibility with the *Join datatypes

Ah, no reasons, this is just a lack of my knowledge of code-base and how it works in day when I wrote that :)
Thanks a lot for note, this is more clear for me now!

I have fixed that :)

belevy · 2021-09-26T16:58:41Z

src/Database/Esqueleto/PostgreSQL.hs

+-- select $ do
+--  bound :& user <- from $
+--      values (   (val (10 :: Int), val ("ten" :: Text))
+--            :| [ (val 20, val "twenty")


I wish the OverloadedList extension wasn't such a foot cannon with NonEmpty and actually just gave a compilation error.

Nothing to change here, just me 😞 in haskell

of `ToFrom` typeclass for postgres `values` expression.

parsonsmatt

Adding functionality is a minor bump - otherwise this is great and looks good to go 😄

I'll make the suggested modifications myself when I release for 3.5.3.0.

parsonsmatt · 2021-09-30T16:10:08Z

changelog.md

@@ -1,3 +1,9 @@
+3.5.2.3


Adding a new feature is a minor bump, so

Suggested change

3.5.2.3

3.5.3.0

parsonsmatt · 2021-09-30T16:10:15Z

esqueleto.cabal

@@ -1,7 +1,7 @@
 cabal-version: 1.12

 name:           esqueleto
-version:        3.5.2.2
+version:        3.5.2.3


Suggested change

version: 3.5.2.3

version: 3.5.3.0

NikitaRazmakhnin added 3 commits September 22, 2021 13:28

Add PostgreSQL-specific support of VALUES(..)

1620a85

scalar expression of values-list for `from` targets.

Bump version and update changelog

469ed1b

Align identation for Postgres values func

a03307a

parsonsmatt approved these changes Sep 22, 2021

View reviewed changes

Merge branch 'master' into pg-values-expr

6373f91

belevy reviewed Sep 26, 2021

View reviewed changes

Use direct From data-type instead

52373e5

of `ToFrom` typeclass for postgres `values` expression.

parsonsmatt approved these changes Sep 30, 2021

View reviewed changes

parsonsmatt merged commit 2a44844 into bitemyapp:master Sep 30, 2021

rjmholt mentioned this pull request Nov 10, 2022

Improve JSON support in Esqueleto.Postgresql #282

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support of PostgreSQL-specific `VALUES(..)` expression #284

Add support of PostgreSQL-specific `VALUES(..)` expression #284

NikitaRazmakhnin commented Sep 22, 2021 •

edited

parsonsmatt left a comment

parsonsmatt Sep 22, 2021

NikitaRazmakhnin Sep 22, 2021

parsonsmatt Sep 22, 2021

NikitaRazmakhnin Sep 22, 2021

parsonsmatt Sep 22, 2021

NikitaRazmakhnin Sep 22, 2021 •

edited

parsonsmatt Sep 22, 2021

parsonsmatt Sep 22, 2021

parsonsmatt commented Sep 22, 2021

belevy Sep 26, 2021

NikitaRazmakhnin Sep 29, 2021

belevy Sep 26, 2021

parsonsmatt left a comment

parsonsmatt Sep 30, 2021

parsonsmatt Sep 30, 2021

		instance (ToSomeValues a, Ex.ToAliasReference a, Ex.ToAlias a) => Ex.ToFrom (PgValuesExprs a) a where
		toFrom = fromValues . unPgValuesExprs

Add support of PostgreSQL-specific VALUES(..) expression #284

Add support of PostgreSQL-specific VALUES(..) expression #284

Conversation

NikitaRazmakhnin commented Sep 22, 2021 • edited

parsonsmatt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NikitaRazmakhnin Sep 22, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

parsonsmatt commented Sep 22, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

parsonsmatt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Add support of PostgreSQL-specific `VALUES(..)` expression #284

Add support of PostgreSQL-specific `VALUES(..)` expression #284

NikitaRazmakhnin commented Sep 22, 2021 •

edited

NikitaRazmakhnin Sep 22, 2021 •

edited