Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.x] Add upsert to Eloquent and Base Query Builders #34698

Merged
merged 3 commits into from Oct 6, 2020
Merged

[8.x] Add upsert to Eloquent and Base Query Builders #34698

merged 3 commits into from Oct 6, 2020

Conversation

paras-malhotra
Copy link
Contributor

@paras-malhotra paras-malhotra commented Oct 6, 2020

Motivation
Laravel already supports bulk inserts, bulk insert/ignores. It's only logical to also support bulk upserts, which are already supported on frameworks such as Rails.

Bulk Inserts Already Supported

  • Bulk inserts: errors out on any row with invalid data
DB::table('users')->insert([
    ['email' => 'taylor@example.com', 'votes' => 0],
    ['email' => 'dayle@example.com', 'votes' => 0],
]);
  • Bulk insert or ignore: rows with invalid data are ignored
DB::table('users')->insertOrIgnore([
    ['id' => 1, 'email' => 'taylor@example.com'],
    ['id' => 2, 'email' => 'dayle@example.com'],
]);

Bulk Upsert (This PR)

This PR supports bulk upserts (insert and on duplicate key, update) like so:

DB::table('users')->upsert([
    ['id' => 1, 'email' => 'taylor@example.com'],
    ['id' => 2, 'email' => 'dayle@example.com'],
], 'email');

The first argument is the values to insert/update and the second argument is the column(s) that uniquely identify records. All databases except SQL Server require these columns to have a PRIMARY or UNIQUE index. This is similar to Rails upsert_all without the returning support.

Eloquent upsets are also supported like so:

User::upsert([
    ['id' => 1, 'email' => 'taylor@example.com'],
    ['id' => 2, 'email' => 'dayle@example.com'],
], 'email');

If the model uses the updated_at timestamp, upsert() will also add the updated_at timestamp columns. Inspired from https://github.com/staudenmeir/laravel-upsert with some slight modifications. Ping @staudenmeir

@rodrigopedra
Copy link
Contributor

@rodrigopedra rodrigopedra commented Oct 6, 2020

Usage of VALUES() on MySQL is deprecated starting on MySQL 8.0.20.

Reference:

https://dev.mysql.com/doc/refman/8.0/en/insert-on-duplicate.html

The use of VALUES() to refer to the new row and columns is deprecated beginning with MySQL 8.0.20, and is subject to removal in a future version of MySQL. Instead, use row and column aliases, as described in the next few paragraphs of this section.

The sample below is from the link above showcasing the new syntax to use:

INSERT INTO t1 (a,b,c) VALUES (1,2,3),(4,5,6) AS new
  ON DUPLICATE KEY UPDATE c = new.a+new.b;

@taylorotwell
Copy link
Member

@taylorotwell taylorotwell commented Oct 6, 2020

@rodrigopedra does that new syntax work in MySQL 5.7? Though I guess it doesn't really matter as this feature could require MySQL 8.0.

@rodrigopedra
Copy link
Contributor

@rodrigopedra rodrigopedra commented Oct 6, 2020

Old syntax works on both MySQL versions (5 and 8).

New syntax was introduced in MySQL 8.0.19, from the link on my previous message:

Beginning with MySQL 8.0.19, it is possible to use an alias for the row...

To support both versions I think we can keep the old syntax around as the docs says: "...and is subject to removal in a future version of MySQL". No removal timeline was set yet.

In general it takes a while to MySQL really remove deprecated features (for example the GROUP BY ... DESC was deprecated in 5.7 (https://dev.mysql.com/doc/refman/5.7/en/select.html) but was removed only in 8.0 and noted in the upgrade guide.

So we could use the old syntax for now and handle this case when it actually gets removed.

EDIT: corrected the deprecation example about GROUP BY...DESC as it was actually deprecated in MySQL 5.7

@paras-malhotra
Copy link
Contributor Author

@paras-malhotra paras-malhotra commented Oct 6, 2020

Yeah, it's still supported but deprecated beginning 8.0.20. It isn't removed yet and will be around for a while. So to support both mySQL 5 and 8, it's best to continue using values.

Besides, we're already using VALUES() in all insert operations including insert and insertOrIgnore.

@rodrigopedra
Copy link
Contributor

@rodrigopedra rodrigopedra commented Oct 6, 2020

Actually @paras-malhotra, the usage of VALUES as it is used now in insert and insertOrIgnore is not deprecated and is part of the SQL standard, it is a clause of the INSERT SQL statement.

What is deprecated is the usage of the VALUES() special function on INSERT...ON DUPLICATE KEY UPDATE statement.

https://dev.mysql.com/doc/refman/8.0/en/miscellaneous-functions.html#function_values

@paras-malhotra
Copy link
Contributor Author

@paras-malhotra paras-malhotra commented Oct 6, 2020

Ohh wow, I didn't know that 😮. Okay, I'll probably look out for the removal notice since row/column aliases don't work in 5.7. @taylorotwell if you'd like I can implement the new way (row/column aliases), but I'd recommend we go with values for now as it's still supported in both versions.

@rodrigopedra
Copy link
Contributor

@rodrigopedra rodrigopedra commented Oct 6, 2020

No problem, maybe one of the reasons to deprecate that function is the confusion with the SQL clause usage.

MariaDB took a similar step, but they renamed the VALUES() function to VALUE() (https://mariadb.com/kb/en/values-value/).

I think this feature is a great addition to the framework and that for new it is best to keep the VALUES() usage until it actually gets removed from MySQL. Maybe by then we can remove MySQL 5.x support.

@taylorotwell taylorotwell merged commit 1553d1c into laravel:8.x Oct 6, 2020
7 checks passed
@paras-malhotra paras-malhotra deleted the bulk_upsert branch Oct 6, 2020
@mpskovvang
Copy link
Contributor

@mpskovvang mpskovvang commented Oct 13, 2020

Great addition! This is the first I add to new projects myself.

However, this PR misses the option to do partial updates. You don't want to keep overwriting e.g. created_at.

@paras-malhotra do you know if partial updates is supported by other than MySQL and PostgreSQL? Thanks for contributing this to the core.

Perhaps a third argument with either an indexed array of column names or an associate array with key-value. The default is to update all columns.

@paras-malhotra
Copy link
Contributor Author

@paras-malhotra paras-malhotra commented Oct 13, 2020

@mpskovvang absolutely, that was the follow up PR: #34712 😄

@mpskovvang
Copy link
Contributor

@mpskovvang mpskovvang commented Oct 13, 2020

@mpskovvang absolutely, that was the follow up PR: #34712 😄

Did't see that. Spot on! Thanks! Much appreciated. 🙏

@webbby
Copy link

@webbby webbby commented Oct 15, 2020

Does the feature support columns combination that uniquely identifies records?

For example

App\Models\Flight::upsert([
    ['departure' => 'Oakland', 'destination' => 'San Diego', 'price' => 99],
    ['departure' => 'Chicago', 'destination' => 'New York', 'price' => 150]
], ['departure', 'destination'], ['price']);

departure - not unique
destination - not unique
(departure,destination) - unique pair

If not can it be targeted by unique index instead of columns?

LukeTowers added a commit to octobercms/library that referenced this issue Oct 19, 2020
sunaoka added a commit to sunaoka/laravel-postgres-extension that referenced this issue Oct 20, 2020
Added support for upsert in Laravel 8.10.0
laravel/framework#34698
bennothommo added a commit to octobercms/library that referenced this issue Oct 20, 2020
LukeTowers added a commit to octobercms/library that referenced this issue Oct 20, 2020
Pulls in the work by @paras-malhotra in https:ithub.com/laravel/framework/pull/34698 & laravel/framework#34712 for use in October CMS.

Co-authored-by: Ben Thomson <git@alfreido.com>
@shankhadevpadam
Copy link

@shankhadevpadam shankhadevpadam commented Nov 4, 2020

I have post meta table with fields [id, post_id, meta_key, meta_value]. i want to bulk update a meta values. But It repeatedly create a new row when using upsert method while updating form every time.

foreach ($metas as $key => $value) {
    $arr[] = [
        'post_id' => $post->id,
        'meta_key' => $key,
        'meta_value' => $value ?? '',
    ];
}

Postmeta::upsert($arr, ['post_id', 'meta_key'], ['meta_value']);

but while using updateOrCreate method it work fine.

foreach ($metas as $key => $value) {
    Postmeta::updateOrCreate(['post_id' => $post->id, 'meta_key' => $key], ['meta_value' => $value ?? '']);
}

@paras-malhotra
Copy link
Contributor Author

@paras-malhotra paras-malhotra commented Nov 4, 2020

@shankhadevpadam all databases except SQL Server require the columns in the second argument of the upsert method to have a "primary" or "unique" index. If that doesn't solve your problem, please create a new issue with steps to replicate.

@eusonlito
Copy link
Contributor

@eusonlito eusonlito commented Nov 30, 2020

Only as reference, is important understand than INSERT INTO ... ON CONFLICT will update the internal reference to increment primary key on related table because the increment has to happen before the insert is attempted. Then, if you use the upsert method as a regular insert or update action, your ID will be something like:

-[ RECORD 284 ]
id | 19039
-[ RECORD 285 ]
id | 19221
-[ RECORD 286 ]
id | 19306
-[ RECORD 287 ]
id | 19397
-[ RECORD 288 ]
id | 19784
-[ RECORD 289 ]
id | 19828
-[ RECORD 290 ]
id | 20161
-[ RECORD 291 ]
id | 20768
-[ RECORD 292 ]
id | 20799
-[ RECORD 293 ]
id | 21287
-[ RECORD 294 ]
id | 21770
-[ RECORD 295 ]
id | 22205
-[ RECORD 296 ]
id | 22401
-[ RECORD 297 ]
id | 23099
-[ RECORD 298 ]
id | 23727
-[ RECORD 299 ]
id | 23820
-[ RECORD 300 ]
id | 24190
-[ RECORD 301 ]
id | 24653
-[ RECORD 302 ]
id | 24744
-[ RECORD 303 ]
id | 25054
-[ RECORD 304 ]
id | 25295

@fractalf
Copy link

@fractalf fractalf commented Jan 11, 2021

I noticed that upsert will add values for created_at and updated_at, but insert does not do this.

Is that intended?

My preference would be that insert also has this behaviour

@ejntaylor
Copy link

@ejntaylor ejntaylor commented Mar 25, 2021

Does the feature support columns combination that uniquely identifies records?

For example

App\Models\Flight::upsert([
    ['departure' => 'Oakland', 'destination' => 'San Diego', 'price' => 99],
    ['departure' => 'Chicago', 'destination' => 'New York', 'price' => 150]
], ['departure', 'destination'], ['price']);

departure - not unique
destination - not unique
(departure,destination) - unique pair

If not can it be targeted by unique index instead of columns?

Doesn't look like this is the case - anyone had luck with this?

@rognales
Copy link

@rognales rognales commented Sep 7, 2021

When issuing a mass update or delete query via Eloquent, the saved, updated, deleting, and deleted model events will not be dispatched for the affected models. This is because the models are never actually retrieved when performing mass updates or deletes

This statements applies to UPSERT as well?

Because the created_at and updated_at did get filled by Eloquent

@mdobydullah
Copy link

@mdobydullah mdobydullah commented Dec 3, 2021

Does the feature support columns combination that uniquely identifies records?

For example

App\Models\Flight::upsert([
    ['departure' => 'Oakland', 'destination' => 'San Diego', 'price' => 99],
    ['departure' => 'Chicago', 'destination' => 'New York', 'price' => 150]
], ['departure', 'destination'], ['price']);

departure - not unique
destination - not unique
(departure,destination) - unique pair

If not can it be targeted by unique index instead of columns?

This feature is badly needed. Do you have any solutions regarding this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet