Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MySQL on duplicate key update #674

Merged
merged 10 commits into from
Jun 19, 2017
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 108 additions & 0 deletions persistent-mysql/Database/Persist/MySQL.hs
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
{-# LANGUAGE ConstraintKinds #-}
{-# LANGUAGE GADTs #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE TypeFamilies #-}
{-# LANGUAGE FlexibleContexts #-}
Expand All @@ -14,6 +15,9 @@ module Database.Persist.MySQL
, MySQLBase.defaultSSLInfo
, MySQLConf(..)
, mockMigration
, insertOnDuplicateKeyUpdate
, insertManyOnDuplicateKeyUpdate
, SomeField(..)
) where

import Control.Arrow
Expand Down Expand Up @@ -1014,3 +1018,107 @@ mockMigration mig = do
result = runReaderT . runWriterT . runWriterT $ mig
resp <- result sqlbackend
mapM_ T.putStrLn $ map snd $ snd resp

-- | MySQL specific 'upsert'. This will prevent multiple queries, when one will
-- do.
insertOnDuplicateKeyUpdate
:: ( PersistEntityBackend record ~ BaseBackend backend
, PersistEntity record
, MonadIO m
, PersistStore backend
, backend ~ SqlBackend
)
=> record
-> [Update record]
-> SqlPersistT m ()
insertOnDuplicateKeyUpdate record =
insertManyOnDuplicateKeyUpdate [record] []

-- | This wraps values of an Entity's 'EntityField', making them have the same
-- type. This allows them to be put in lists.
data SomeField record where
SomeField :: EntityField record typ -> SomeField record

-- | Do a bulk insert on the given records in the first parameter. In the event
-- that a key conflicts with a record currently in the database, the second and
-- third parameters determine what will happen.
--
-- The second parameter is a list of fields to copy from the original value.
-- This allows you to specify that, when a collision occurs, you'll just update
-- the value in the database with the field values that you inserted.
--
-- The third parameter is a list of updates to perform that are independent of
-- the value that is provided. You can use this to increment a counter value.
-- These updates only occur if the original record is present in the database.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation for the third parameter is kind of confusing to me

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little difficult to explain. If you pass something like

insertManyOnDuplicateKeyUpdate _ [SomeField UserName] [UserAge +=. 1]

then it generates the following clause in the Update:

INSERT INTO ... VALUES ...
ON DUPLICATE KEY UPDATE ... 
  `user`.`name` = VALUES(`user`.`name`)
  `user`.`age` = `user`.`age` + 1

So the name gets copied from whatever record was being inserted and had a key collision, and the age gets incremented by 1.

Since the Update field requires a specific value, that means that we can't make it dependent on the record that we're updating. MySQL would support something like

ON DUPLICATE KEY UPDATE
  user.age = VALUES(user.age) + 2

which would add 2 to the record that hit the duplicate key. There's not a good way to make the update have more complicated values, unfortunately, so this feature would be difficult to incorporate as-is.

insertManyOnDuplicateKeyUpdate
:: ( PersistEntityBackend record ~ SqlBackend
, PersistEntity record
, MonadIO m
)
=> [record] -- ^ A list of the records you want to insert, or update
-> [SomeField record] -- ^ A list of the fields you want to copy over.
-> [Update record] -- ^ A list of the updates to apply that aren't dependent on the record being inserted.
-> SqlPersistT m ()
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function would be used like:

bulkInsertOnDuplicateKeyUpdate
  [ {- a big ol' list of records you want to insert/update -} ]
  [ SomeField UserName, SomeField UserEmail ] -- this copies the values that are being inserted to existing records
  [ UserModified =. now, UserEncounted +=. 1 ] -- this update is performed for any field that matches the inserted records

insertManyOnDuplicateKeyUpdate [] _ _ = return ()
insertManyOnDuplicateKeyUpdate records [] [] = insertMany_ records
insertManyOnDuplicateKeyUpdate records fieldValues updates =
uncurry rawExecute $ mkBulkInsertQuery records fieldValues updates

-- | This creates the query for 'bulkInsertOnDuplicateKeyUpdate'. It will give
-- garbage results if you don't provide a list of either fields to copy or
-- fields to update.
mkBulkInsertQuery
:: (PersistEntityBackend record ~ SqlBackend, PersistEntity record)
=> [record] -- ^ A list of the records you want to insert, or update
-> [SomeField record] -- ^ A list of the fields you want to copy over.
-> [Update record] -- ^ A list of the updates to apply that aren't dependent on the record being inserted.
-> (Text, [PersistValue])
mkBulkInsertQuery records fieldValues updates =
(q, recordValues <> updsValues)
where
fieldDefs = map (\x -> case x of SomeField rec -> persistFieldDef rec) fieldValues
updateFieldNames = map (T.pack . escapeDBName . fieldDB) fieldDefs
entityDef' = entityDef records
entityFieldNames = map (T.pack . escapeDBName . fieldDB) (entityFields entityDef')
tableName = T.pack . escapeDBName . entityDB $ entityDef'
recordValues = concatMap (map toPersistValue . toPersistFields) records
recordPlaceholders = commaSeparated $ map (parenWrapped . commaSeparated . map (const "?") . toPersistFields) records
fieldSets = map (\n -> T.concat [n, "=VALUES(", n, ")"]) updateFieldNames
upds = map mkUpdateText updates
updsValues = map (\(Update _ val _) -> toPersistValue val) updates
q = T.concat
[ "INSERT INTO "
, tableName
, " ("
, commaSeparated entityFieldNames
, ") "
, " VALUES "
, recordPlaceholders
, " ON DUPLICATE KEY UPDATE "
, commaSeparated (fieldSets <> upds)
]

-- | Vendored from @persistent@.
mkUpdateText :: PersistEntity record => Update record -> Text
mkUpdateText x =
case updateUpdate x of
Assign -> n <> "=?"
Add -> T.concat [n, "=", n, "+?"]
Subtract -> T.concat [n, "=", n, "-?"]
Multiply -> T.concat [n, "=", n, "*?"]
Divide -> T.concat [n, "=", n, "/?"]
BackendSpecificUpdate up ->
error . T.unpack $ "BackendSpecificUpdate " <> up <> " not supported"
where
n = T.pack . escapeDBName . fieldDB . updateFieldDef $ x

commaSeparated :: [Text] -> Text
commaSeparated = T.intercalate ", "

parenWrapped :: Text -> Text
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These functions make the code much more readable, nice idea

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I can use them in the other instances of these if you'd like.

parenWrapped t = T.concat ["(", t, ")"]

-- | Gets the 'FieldDef' for an 'Update'. Vendored from @persistent@.
updateFieldDef :: PersistEntity v => Update v -> FieldDef
updateFieldDef (Update f _ _) = persistFieldDef f
updateFieldDef BackendUpdate {} = error "updateFieldDef did not expect BackendUpdate"